Jul 10 17:11:06.531983 ip-10-0-134-65 systemd[1]: Starting Kubernetes Kubelet... Jul 10 17:11:06.930002 ip-10-0-134-65 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 17:11:06.930002 ip-10-0-134-65 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 17:11:06.930002 ip-10-0-134-65 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 17:11:06.930002 ip-10-0-134-65 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 17:11:06.930002 ip-10-0-134-65 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 17:11:06.931809 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.931725 2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 17:11:06.937316 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937299 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 17:11:06.937316 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937315 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 17:11:06.937316 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937319 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 17:11:06.937415 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937325 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 17:11:06.937415 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937328 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 17:11:06.937415 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937331 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 17:11:06.937415 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937333 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 17:11:06.937415 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937337 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 17:11:06.937415 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937339 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 17:11:06.937415 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937342 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 17:11:06.937415 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937345 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 17:11:06.937415 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937347 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 17:11:06.937415 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937350 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 17:11:06.937415 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937352 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 17:11:06.937415 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937355 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 17:11:06.937415 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937357 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 17:11:06.937415 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937360 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 17:11:06.937415 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937362 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 17:11:06.937415 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937365 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 17:11:06.937415 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937367 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 17:11:06.937415 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937372 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 17:11:06.937415 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937375 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 17:11:06.937415 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937377 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 17:11:06.937910 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937380 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 17:11:06.937910 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937383 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 17:11:06.937910 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937386 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 17:11:06.937910 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937389 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 17:11:06.937910 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937399 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 17:11:06.937910 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937402 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 17:11:06.937910 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937405 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 17:11:06.937910 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937407 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 17:11:06.937910 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937409 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 17:11:06.937910 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937412 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 17:11:06.937910 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937414 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 17:11:06.937910 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937417 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 17:11:06.937910 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937420 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 17:11:06.937910 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937423 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 17:11:06.937910 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937426 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 17:11:06.937910 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937429 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 17:11:06.937910 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937432 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 17:11:06.937910 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937434 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 17:11:06.937910 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937436 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 17:11:06.937910 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937439 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 17:11:06.938430 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937442 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 17:11:06.938430 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937444 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 17:11:06.938430 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937447 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 17:11:06.938430 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937449 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 17:11:06.938430 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937452 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 17:11:06.938430 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937454 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 17:11:06.938430 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937457 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 17:11:06.938430 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937459 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 17:11:06.938430 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937464 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 17:11:06.938430 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937468 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 10 17:11:06.938430 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937474 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 17:11:06.938430 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937477 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 17:11:06.938430 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937481 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 17:11:06.938430 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937484 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 17:11:06.938430 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937487 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 17:11:06.938430 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937489 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 17:11:06.938430 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937492 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 17:11:06.938430 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937494 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 17:11:06.938430 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937497 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 17:11:06.938430 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937499 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 17:11:06.938928 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937502 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 17:11:06.938928 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937504 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 17:11:06.938928 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937507 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 17:11:06.938928 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937510 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 17:11:06.938928 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937513 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 17:11:06.938928 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937515 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 17:11:06.938928 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937517 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 17:11:06.938928 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937520 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 17:11:06.938928 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937523 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 17:11:06.938928 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937525 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 17:11:06.938928 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937528 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 17:11:06.938928 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937532 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 17:11:06.938928 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937535 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 17:11:06.938928 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937538 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 17:11:06.938928 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937541 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 17:11:06.938928 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937543 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 17:11:06.938928 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937546 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 17:11:06.938928 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937548 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 17:11:06.938928 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937551 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 17:11:06.939386 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937553 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 17:11:06.939386 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937555 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 17:11:06.939386 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937558 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 17:11:06.939386 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937560 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 17:11:06.939386 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937563 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 17:11:06.939386 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937565 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 17:11:06.939386 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937569 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 17:11:06.939386 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937571 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 17:11:06.939386 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937574 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 17:11:06.939386 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937576 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 17:11:06.939386 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937579 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 17:11:06.939386 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937581 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 17:11:06.939386 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937583 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 17:11:06.939386 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937587 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 17:11:06.939386 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937591 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 17:11:06.939386 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937593 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 17:11:06.939386 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.937597 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 17:11:06.939386 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938375 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 17:11:06.939386 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938382 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 17:11:06.939386 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938384 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 17:11:06.939892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938387 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 17:11:06.939892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938390 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 17:11:06.939892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938393 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 17:11:06.939892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938396 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 10 17:11:06.939892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938398 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 17:11:06.939892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938401 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 17:11:06.939892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938404 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 17:11:06.939892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938406 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 17:11:06.939892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938409 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 17:11:06.939892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938411 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 17:11:06.939892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938414 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 17:11:06.939892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938416 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 17:11:06.939892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938419 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 17:11:06.939892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938421 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 17:11:06.939892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938423 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 17:11:06.939892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938426 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 17:11:06.939892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938428 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 17:11:06.939892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938432 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 17:11:06.939892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938435 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 17:11:06.939892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938439 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 17:11:06.940357 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938443 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 17:11:06.940357 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938446 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 17:11:06.940357 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938449 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 17:11:06.940357 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938453 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 17:11:06.940357 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938457 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 17:11:06.940357 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938460 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 17:11:06.940357 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938464 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 17:11:06.940357 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938467 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 17:11:06.940357 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938470 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 17:11:06.940357 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938473 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 17:11:06.940357 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938476 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 17:11:06.940357 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938479 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 17:11:06.940357 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938482 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 17:11:06.940357 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938485 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 17:11:06.940357 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938488 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 17:11:06.940357 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938491 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 17:11:06.940357 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938494 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 17:11:06.940357 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938496 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 17:11:06.940357 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938499 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 17:11:06.940357 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938502 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 17:11:06.940854 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938504 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 17:11:06.940854 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938506 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 17:11:06.940854 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938509 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 17:11:06.940854 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938511 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 17:11:06.940854 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938514 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 17:11:06.940854 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938516 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 17:11:06.940854 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938518 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 17:11:06.940854 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938521 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 17:11:06.940854 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938523 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 17:11:06.940854 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938526 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 17:11:06.940854 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938530 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 17:11:06.940854 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938533 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 17:11:06.940854 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938535 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 17:11:06.940854 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938538 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 17:11:06.940854 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938540 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 17:11:06.940854 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938543 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 17:11:06.940854 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938545 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 17:11:06.940854 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938547 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 17:11:06.940854 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938550 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 17:11:06.940854 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938552 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 17:11:06.941333 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938555 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 17:11:06.941333 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938557 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 17:11:06.941333 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938560 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 17:11:06.941333 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938562 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 17:11:06.941333 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938564 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 17:11:06.941333 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938567 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 17:11:06.941333 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938569 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 17:11:06.941333 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938572 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 17:11:06.941333 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938575 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 17:11:06.941333 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938577 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 17:11:06.941333 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938580 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 17:11:06.941333 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938582 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 17:11:06.941333 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938584 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 17:11:06.941333 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938587 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 17:11:06.941333 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938591 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 17:11:06.941333 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938594 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 17:11:06.941333 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938596 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 17:11:06.941333 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938598 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 17:11:06.941333 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938601 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 17:11:06.941333 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938603 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 17:11:06.941862 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938606 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 17:11:06.941862 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938609 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 17:11:06.941862 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938611 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 17:11:06.941862 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938615 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 17:11:06.941862 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938617 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 17:11:06.941862 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938620 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 17:11:06.941862 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938622 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 17:11:06.941862 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938624 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 17:11:06.941862 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938627 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 17:11:06.941862 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938629 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 17:11:06.941862 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938631 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 17:11:06.941862 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938635 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 17:11:06.941862 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938639 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 17:11:06.941862 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938642 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 17:11:06.941862 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938645 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 17:11:06.941862 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.938648 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 17:11:06.941862 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939074 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 17:11:06.941862 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939080 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 10 17:11:06.941862 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939083 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 17:11:06.941862 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939086 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 17:11:06.942339 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939089 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 17:11:06.942339 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939092 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 17:11:06.942339 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939095 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 17:11:06.942339 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939097 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 17:11:06.942339 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939100 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 17:11:06.942339 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939102 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 17:11:06.942339 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939104 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 17:11:06.942339 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939107 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 17:11:06.942339 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939111 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 17:11:06.942339 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939114 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 17:11:06.942339 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939117 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 17:11:06.942339 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939120 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 17:11:06.942339 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939123 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 17:11:06.942339 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939126 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 17:11:06.942339 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939131 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 17:11:06.942339 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939134 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 17:11:06.942339 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939137 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 17:11:06.942339 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939140 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 17:11:06.942780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939143 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 17:11:06.942780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939145 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 17:11:06.942780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939150 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 17:11:06.942780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939152 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 17:11:06.942780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939155 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 17:11:06.942780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939158 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 17:11:06.942780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939160 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 17:11:06.942780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939163 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 17:11:06.942780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939165 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 17:11:06.942780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939168 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 17:11:06.942780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939170 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 17:11:06.942780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939172 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 17:11:06.942780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939175 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 17:11:06.942780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939177 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 17:11:06.942780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939179 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 17:11:06.942780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939182 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 17:11:06.942780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939185 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 17:11:06.942780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939188 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 17:11:06.942780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939190 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 17:11:06.942780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939193 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 17:11:06.943292 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939195 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 17:11:06.943292 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939198 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 17:11:06.943292 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939200 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 17:11:06.943292 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939202 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 17:11:06.943292 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939205 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 17:11:06.943292 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939207 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 17:11:06.943292 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939210 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 17:11:06.943292 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939212 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 17:11:06.943292 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939216 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 17:11:06.943292 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939219 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 17:11:06.943292 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939223 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 17:11:06.943292 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939226 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 17:11:06.943292 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939229 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 17:11:06.943292 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939232 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 17:11:06.943292 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939236 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 17:11:06.943292 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939239 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 17:11:06.943292 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939242 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 17:11:06.943292 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939245 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 17:11:06.943292 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939247 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 17:11:06.943292 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939250 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 17:11:06.943775 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939252 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 17:11:06.943775 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939254 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 17:11:06.943775 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939258 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 17:11:06.943775 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939261 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 17:11:06.943775 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939263 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 17:11:06.943775 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939266 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 17:11:06.943775 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939268 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 17:11:06.943775 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939271 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 17:11:06.943775 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939273 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 17:11:06.943775 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939276 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 17:11:06.943775 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939278 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 17:11:06.943775 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939280 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 17:11:06.943775 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939283 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 17:11:06.943775 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939285 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 17:11:06.943775 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939287 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 17:11:06.943775 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939290 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 17:11:06.943775 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939292 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 17:11:06.943775 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939294 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 17:11:06.943775 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939297 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 17:11:06.943775 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939299 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 17:11:06.944270 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939303 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 17:11:06.944270 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939306 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 17:11:06.944270 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939308 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 17:11:06.944270 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939311 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 17:11:06.944270 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939314 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 17:11:06.944270 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939316 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 17:11:06.944270 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939321 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 17:11:06.944270 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939323 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 17:11:06.944270 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939326 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 17:11:06.944270 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939328 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 17:11:06.944270 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939331 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 17:11:06.944270 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939333 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 17:11:06.944270 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939335 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 17:11:06.944270 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939337 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 17:11:06.944270 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939340 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 17:11:06.944270 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939343 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 17:11:06.944270 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.939345 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 17:11:06.944270 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939422 2583 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 17:11:06.944270 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939430 2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 17:11:06.944270 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939444 2583 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 17:11:06.944270 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939449 2583 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 17:11:06.944831 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939454 2583 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 17:11:06.944831 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939457 2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 17:11:06.944831 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939461 2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 17:11:06.944831 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939465 2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 17:11:06.944831 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939468 2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 17:11:06.944831 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939471 2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 17:11:06.944831 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939474 2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 17:11:06.944831 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939478 2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 17:11:06.944831 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939481 2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 17:11:06.944831 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939484 2583 flags.go:64] FLAG: --cgroup-root="" Jul 10 17:11:06.944831 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939487 2583 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 17:11:06.944831 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939491 2583 flags.go:64] FLAG: --client-ca-file="" Jul 10 17:11:06.944831 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939494 2583 flags.go:64] FLAG: --cloud-provider="external" Jul 10 17:11:06.944831 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939496 2583 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 17:11:06.944831 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939500 2583 flags.go:64] FLAG: --cluster-domain="" Jul 10 17:11:06.944831 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939510 2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 17:11:06.944831 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939513 2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 17:11:06.944831 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939517 2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 17:11:06.944831 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939520 2583 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 17:11:06.944831 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939523 2583 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 17:11:06.944831 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939526 2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 17:11:06.944831 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939529 2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 17:11:06.944831 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939532 2583 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 17:11:06.945379 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939535 2583 flags.go:64] FLAG: --contention-profiling="false" Jul 10 17:11:06.945379 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939538 2583 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 17:11:06.945379 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939541 2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 17:11:06.945379 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939543 2583 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 17:11:06.945379 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939546 2583 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 17:11:06.945379 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939550 2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 17:11:06.945379 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939553 2583 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 17:11:06.945379 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939556 2583 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 17:11:06.945379 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939558 2583 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 17:11:06.945379 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939561 2583 flags.go:64] FLAG: --enable-server="true" Jul 10 17:11:06.945379 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939564 2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 17:11:06.945379 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939569 2583 flags.go:64] FLAG: --event-burst="100" Jul 10 17:11:06.945379 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939572 2583 flags.go:64] FLAG: --event-qps="50" Jul 10 17:11:06.945379 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939574 2583 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 17:11:06.945379 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939577 2583 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 17:11:06.945379 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939580 2583 flags.go:64] FLAG: --eviction-hard="" Jul 10 17:11:06.945379 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939583 2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 17:11:06.945379 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939586 2583 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 17:11:06.945379 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939589 2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 17:11:06.945379 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939592 2583 flags.go:64] FLAG: --eviction-soft="" Jul 10 17:11:06.945379 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939595 2583 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 17:11:06.945379 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939597 2583 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 17:11:06.945379 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939600 2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 17:11:06.945379 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939603 2583 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 17:11:06.945379 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939606 2583 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 17:11:06.946016 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939609 2583 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 17:11:06.946016 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939613 2583 flags.go:64] FLAG: --feature-gates="" Jul 10 17:11:06.946016 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939617 2583 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 17:11:06.946016 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939620 2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 17:11:06.946016 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939623 2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 17:11:06.946016 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939626 2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 17:11:06.946016 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939629 2583 flags.go:64] FLAG: --healthz-port="10248" Jul 10 17:11:06.946016 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939632 2583 flags.go:64] FLAG: --help="false" Jul 10 17:11:06.946016 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939635 2583 flags.go:64] FLAG: --hostname-override="ip-10-0-134-65.ec2.internal" Jul 10 17:11:06.946016 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939638 2583 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 17:11:06.946016 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939641 2583 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 17:11:06.946016 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939643 2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 17:11:06.946016 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939646 2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 17:11:06.946016 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939650 2583 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 17:11:06.946016 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939653 2583 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 17:11:06.946016 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939655 2583 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 17:11:06.946016 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939678 2583 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 17:11:06.946016 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939683 2583 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 17:11:06.946016 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939686 2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 17:11:06.946016 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939690 2583 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 17:11:06.946016 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939692 2583 flags.go:64] FLAG: --kube-reserved="" Jul 10 17:11:06.946016 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939695 2583 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 17:11:06.946016 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939698 2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 17:11:06.946016 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939701 2583 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 17:11:06.946599 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939704 2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 17:11:06.946599 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939706 2583 flags.go:64] FLAG: --lock-file="" Jul 10 17:11:06.946599 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939709 2583 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 17:11:06.946599 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939712 2583 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 17:11:06.946599 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939715 2583 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 17:11:06.946599 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939721 2583 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 17:11:06.946599 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939724 2583 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 17:11:06.946599 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939727 2583 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 17:11:06.946599 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939729 2583 flags.go:64] FLAG: --logging-format="text" Jul 10 17:11:06.946599 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939733 2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 17:11:06.946599 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939736 2583 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 17:11:06.946599 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939739 2583 flags.go:64] FLAG: --manifest-url="" Jul 10 17:11:06.946599 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939742 2583 flags.go:64] FLAG: --manifest-url-header="" Jul 10 17:11:06.946599 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939746 2583 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 17:11:06.946599 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939749 2583 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 17:11:06.946599 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939752 2583 flags.go:64] FLAG: --max-pods="110" Jul 10 17:11:06.946599 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939755 2583 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 17:11:06.946599 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939758 2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 17:11:06.946599 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939761 2583 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 17:11:06.946599 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939765 2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 17:11:06.946599 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939767 2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 17:11:06.946599 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939770 2583 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 17:11:06.946599 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939773 2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 17:11:06.946599 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939780 2583 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 17:11:06.946599 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939783 2583 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 17:11:06.947214 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939786 2583 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 17:11:06.947214 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939789 2583 flags.go:64] FLAG: --pod-cidr="" Jul 10 17:11:06.947214 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939792 2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 17:11:06.947214 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939798 2583 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 17:11:06.947214 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939800 2583 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 17:11:06.947214 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939803 2583 flags.go:64] FLAG: --pods-per-core="0" Jul 10 17:11:06.947214 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939806 2583 flags.go:64] FLAG: --port="10250" Jul 10 17:11:06.947214 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939809 2583 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 17:11:06.947214 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939811 2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0549a02a136ee7946" Jul 10 17:11:06.947214 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939814 2583 flags.go:64] FLAG: --qos-reserved="" Jul 10 17:11:06.947214 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939817 2583 flags.go:64] FLAG: --read-only-port="10255" Jul 10 17:11:06.947214 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939820 2583 flags.go:64] FLAG: --register-node="true" Jul 10 17:11:06.947214 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939823 2583 flags.go:64] FLAG: --register-with-taints="" Jul 10 17:11:06.947214 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939826 2583 flags.go:64] FLAG: --registry-burst="10" Jul 10 17:11:06.947214 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939829 2583 flags.go:64] FLAG: --registry-qps="5" Jul 10 17:11:06.947214 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939832 2583 flags.go:64] FLAG: --reserved-cpus="" Jul 10 17:11:06.947214 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939835 2583 flags.go:64] FLAG: --reserved-memory="" Jul 10 17:11:06.947214 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939839 2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 17:11:06.947214 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939842 2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 17:11:06.947214 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939845 2583 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 17:11:06.947214 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939848 2583 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 17:11:06.947214 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939850 2583 flags.go:64] FLAG: --runonce="false" Jul 10 17:11:06.947214 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939853 2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 17:11:06.947214 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939856 2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 17:11:06.947214 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939859 2583 flags.go:64] FLAG: --seccomp-default="false" Jul 10 17:11:06.947916 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939862 2583 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 17:11:06.947916 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939864 2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 17:11:06.947916 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939867 2583 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 17:11:06.947916 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939870 2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 17:11:06.947916 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939873 2583 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 17:11:06.947916 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939875 2583 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 17:11:06.947916 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939878 2583 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 17:11:06.947916 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939881 2583 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 17:11:06.947916 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939884 2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 17:11:06.947916 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939886 2583 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 17:11:06.947916 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939889 2583 flags.go:64] FLAG: --system-cgroups="" Jul 10 17:11:06.947916 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939892 2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 17:11:06.947916 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939897 2583 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 17:11:06.947916 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939900 2583 flags.go:64] FLAG: --tls-cert-file="" Jul 10 17:11:06.947916 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939903 2583 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 17:11:06.947916 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939907 2583 flags.go:64] FLAG: --tls-min-version="" Jul 10 17:11:06.947916 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939909 2583 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 17:11:06.947916 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939913 2583 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 17:11:06.947916 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939916 2583 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 17:11:06.947916 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939919 2583 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 17:11:06.947916 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939922 2583 flags.go:64] FLAG: --v="2" Jul 10 17:11:06.947916 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939926 2583 flags.go:64] FLAG: --version="false" Jul 10 17:11:06.947916 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939931 2583 flags.go:64] FLAG: --vmodule="" Jul 10 17:11:06.947916 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939934 2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 17:11:06.947916 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.939937 2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 17:11:06.948561 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940077 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 10 17:11:06.948561 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940080 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 17:11:06.948561 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940083 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 17:11:06.948561 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940086 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 17:11:06.948561 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940089 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 17:11:06.948561 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940092 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 17:11:06.948561 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940095 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 17:11:06.948561 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940098 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 17:11:06.948561 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940100 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 17:11:06.948561 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940103 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 17:11:06.948561 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940106 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 17:11:06.948561 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940109 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 17:11:06.948561 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940111 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 17:11:06.948561 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940115 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 17:11:06.948561 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940117 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 17:11:06.948561 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940120 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 17:11:06.948561 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940123 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 17:11:06.948561 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940125 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 17:11:06.948561 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940129 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 17:11:06.948561 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940132 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 17:11:06.949138 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940135 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 17:11:06.949138 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940138 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 17:11:06.949138 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940141 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 17:11:06.949138 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940143 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 17:11:06.949138 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940149 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 17:11:06.949138 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940151 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 17:11:06.949138 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940154 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 17:11:06.949138 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940157 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 17:11:06.949138 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940159 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 17:11:06.949138 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940163 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 17:11:06.949138 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940166 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 17:11:06.949138 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940168 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 17:11:06.949138 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940170 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 17:11:06.949138 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940173 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 17:11:06.949138 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940176 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 17:11:06.949138 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940178 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 17:11:06.949138 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940181 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 17:11:06.949138 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940183 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 17:11:06.949138 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940185 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 17:11:06.949892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940188 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 17:11:06.949892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940190 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 17:11:06.949892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940193 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 17:11:06.949892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940195 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 17:11:06.949892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940197 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 17:11:06.949892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940200 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 17:11:06.949892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940202 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 17:11:06.949892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940205 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 17:11:06.949892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940207 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 17:11:06.949892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940209 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 17:11:06.949892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940212 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 17:11:06.949892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940214 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 17:11:06.949892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940216 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 17:11:06.949892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940219 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 17:11:06.949892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940221 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 17:11:06.949892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940224 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 17:11:06.949892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940226 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 17:11:06.949892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940231 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 17:11:06.949892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940233 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 17:11:06.949892 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940237 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 17:11:06.950500 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940241 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 17:11:06.950500 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940244 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 17:11:06.950500 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940248 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 17:11:06.950500 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940250 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 17:11:06.950500 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940253 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 17:11:06.950500 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940255 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 17:11:06.950500 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940257 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 17:11:06.950500 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940260 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 17:11:06.950500 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940263 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 17:11:06.950500 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940265 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 17:11:06.950500 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940268 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 17:11:06.950500 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940270 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 17:11:06.950500 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940273 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 17:11:06.950500 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940275 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 17:11:06.950500 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940278 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 17:11:06.950500 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940280 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 17:11:06.950500 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940283 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 17:11:06.950500 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940287 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 17:11:06.950500 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940290 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 17:11:06.950982 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940293 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 17:11:06.950982 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940295 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 17:11:06.950982 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940298 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 17:11:06.950982 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940300 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 17:11:06.950982 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940303 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 17:11:06.950982 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940305 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 17:11:06.950982 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940307 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 17:11:06.950982 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940310 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 17:11:06.950982 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940312 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 17:11:06.950982 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940314 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 17:11:06.950982 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940318 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 17:11:06.950982 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940321 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 17:11:06.950982 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940323 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 17:11:06.950982 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940326 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 17:11:06.950982 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940329 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 17:11:06.950982 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940333 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 17:11:06.950982 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940335 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 17:11:06.950982 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940338 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 17:11:06.950982 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940340 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 17:11:06.950982 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940343 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 17:11:06.951460 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.940345 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 17:11:06.951460 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.941213 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 17:11:06.951460 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.950761 2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 17:11:06.951460 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.950777 2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 17:11:06.951691 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.951680 2583 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 17:11:06.951729 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.951696 2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 17:11:06.951770 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951761 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 17:11:06.951770 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951770 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 10 17:11:06.951825 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951773 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 17:11:06.951825 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951777 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 17:11:06.951825 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951780 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 17:11:06.951825 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951783 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 17:11:06.951825 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951786 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 17:11:06.951825 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951789 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 17:11:06.951825 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951791 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 17:11:06.951825 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951794 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 17:11:06.951825 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951797 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 17:11:06.951825 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951800 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 17:11:06.951825 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951803 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 17:11:06.951825 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951805 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 17:11:06.951825 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951808 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 17:11:06.951825 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951810 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 17:11:06.951825 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951814 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 17:11:06.951825 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951817 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 17:11:06.951825 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951821 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 17:11:06.951825 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951825 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 17:11:06.951825 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951827 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 17:11:06.951825 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951830 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 17:11:06.952313 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951833 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 17:11:06.952313 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951837 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 17:11:06.952313 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951839 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 17:11:06.952313 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951842 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 17:11:06.952313 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951845 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 17:11:06.952313 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951847 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 17:11:06.952313 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951850 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 17:11:06.952313 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951852 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 17:11:06.952313 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951855 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 17:11:06.952313 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951857 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 17:11:06.952313 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951860 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 17:11:06.952313 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951863 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 17:11:06.952313 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951865 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 17:11:06.952313 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951868 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 17:11:06.952313 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951870 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 17:11:06.952313 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951872 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 17:11:06.952313 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951875 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 17:11:06.952313 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951877 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 17:11:06.952313 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951880 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 17:11:06.952780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951882 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 17:11:06.952780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951885 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 17:11:06.952780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951887 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 17:11:06.952780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951890 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 17:11:06.952780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951892 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 17:11:06.952780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951895 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 17:11:06.952780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951898 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 17:11:06.952780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951901 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 17:11:06.952780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951903 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 17:11:06.952780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951905 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 17:11:06.952780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951908 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 17:11:06.952780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951910 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 17:11:06.952780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951913 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 17:11:06.952780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951915 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 17:11:06.952780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951918 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 17:11:06.952780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951921 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 17:11:06.952780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951924 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 17:11:06.952780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951926 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 17:11:06.952780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951929 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 17:11:06.952780 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951931 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 17:11:06.953262 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951934 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 17:11:06.953262 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951938 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 17:11:06.953262 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951941 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 17:11:06.953262 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951944 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 17:11:06.953262 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951948 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 17:11:06.953262 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951951 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 17:11:06.953262 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951954 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 17:11:06.953262 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951957 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 17:11:06.953262 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951959 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 17:11:06.953262 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951962 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 17:11:06.953262 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951965 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 17:11:06.953262 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951968 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 17:11:06.953262 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951971 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 17:11:06.953262 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951974 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 17:11:06.953262 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951977 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 17:11:06.953262 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951980 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 17:11:06.953262 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951982 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 17:11:06.953262 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951985 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 17:11:06.953262 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951988 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 17:11:06.953804 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951991 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 17:11:06.953804 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951995 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 17:11:06.953804 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.951998 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 17:11:06.953804 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952001 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 17:11:06.953804 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952004 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 17:11:06.953804 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952006 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 17:11:06.953804 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952009 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 17:11:06.953804 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952012 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 17:11:06.953804 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952014 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 17:11:06.953804 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952017 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 17:11:06.953804 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952020 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 17:11:06.953804 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952022 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 17:11:06.953804 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952025 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 17:11:06.953804 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952028 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 17:11:06.953804 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952030 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 17:11:06.953804 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952032 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 17:11:06.953804 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952035 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 17:11:06.953804 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952037 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 17:11:06.953804 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952040 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 17:11:06.954256 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.952045 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 17:11:06.954256 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952167 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 17:11:06.954256 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952172 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 17:11:06.954256 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952175 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 17:11:06.954256 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952178 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 17:11:06.954256 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952181 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 17:11:06.954256 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952183 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 17:11:06.954256 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952186 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 17:11:06.954256 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952189 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 17:11:06.954256 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952192 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 17:11:06.954256 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952196 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 17:11:06.954256 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952199 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 17:11:06.954256 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952202 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 17:11:06.954256 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952205 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 17:11:06.954256 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952208 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 17:11:06.954615 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952211 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 17:11:06.954615 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952213 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 17:11:06.954615 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952216 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 17:11:06.954615 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952218 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 17:11:06.954615 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952221 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 17:11:06.954615 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952223 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 17:11:06.954615 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952226 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 17:11:06.954615 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952229 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 17:11:06.954615 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952231 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 17:11:06.954615 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952234 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 17:11:06.954615 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952236 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 17:11:06.954615 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952239 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 17:11:06.954615 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952241 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 17:11:06.954615 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952244 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 17:11:06.954615 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952246 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 17:11:06.954615 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952249 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 17:11:06.954615 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952252 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 17:11:06.954615 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952254 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 17:11:06.954615 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952257 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 17:11:06.954615 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952259 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 17:11:06.955106 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952262 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 17:11:06.955106 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952264 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 17:11:06.955106 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952267 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 17:11:06.955106 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952269 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 17:11:06.955106 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952272 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 17:11:06.955106 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952274 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 17:11:06.955106 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952277 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 17:11:06.955106 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952280 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 17:11:06.955106 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952283 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 17:11:06.955106 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952286 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 17:11:06.955106 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952288 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 17:11:06.955106 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952292 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 17:11:06.955106 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952294 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 17:11:06.955106 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952297 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 17:11:06.955106 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952300 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 17:11:06.955106 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952302 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 17:11:06.955106 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952305 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 17:11:06.955106 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952307 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 17:11:06.955106 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952310 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 17:11:06.955106 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952313 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 17:11:06.955585 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952316 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 10 17:11:06.955585 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952318 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 17:11:06.955585 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952321 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 17:11:06.955585 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952323 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 17:11:06.955585 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952326 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 17:11:06.955585 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952328 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 17:11:06.955585 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952331 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 17:11:06.955585 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952333 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 17:11:06.955585 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952336 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 17:11:06.955585 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952339 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 17:11:06.955585 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952343 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 17:11:06.955585 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952346 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 17:11:06.955585 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952349 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 17:11:06.955585 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952351 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 17:11:06.955585 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952354 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 17:11:06.955585 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952356 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 17:11:06.955585 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952359 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 17:11:06.955585 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952361 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 17:11:06.955585 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952363 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 17:11:06.955585 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952366 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 17:11:06.956068 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952368 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 17:11:06.956068 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952371 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 17:11:06.956068 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952373 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 17:11:06.956068 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952375 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 17:11:06.956068 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952378 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 17:11:06.956068 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952380 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 17:11:06.956068 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952384 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 17:11:06.956068 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952386 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 17:11:06.956068 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952389 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 17:11:06.956068 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952391 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 17:11:06.956068 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952394 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 17:11:06.956068 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952396 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 17:11:06.956068 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952399 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 17:11:06.956068 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952401 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 17:11:06.956068 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952403 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 17:11:06.956068 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952406 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 17:11:06.956068 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952408 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 17:11:06.956068 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952410 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 17:11:06.956068 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952413 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 17:11:06.956518 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952415 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 17:11:06.956518 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952418 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 17:11:06.956518 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952420 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 17:11:06.956518 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952423 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 17:11:06.956518 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952425 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 17:11:06.956518 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:06.952428 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 17:11:06.956518 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.952433 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 17:11:06.956518 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.952556 2583 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 17:11:06.956518 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.956217 2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 17:11:06.957884 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.957871 2583 server.go:1019] "Starting client certificate rotation" Jul 10 17:11:06.957996 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.957976 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 17:11:06.958030 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.958022 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 17:11:06.960595 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.960582 2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 17:11:06.963453 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.963435 2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 17:11:06.977039 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.977015 2583 log.go:25] "Validated CRI v1 runtime API" Jul 10 17:11:06.982180 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.982159 2583 log.go:25] "Validated CRI v1 image API" Jul 10 17:11:06.983088 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.983074 2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 17:11:06.986561 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.986540 2583 fs.go:135] Filesystem UUIDs: map[50eb1eb6-4a22-4cd6-a866-31944ca3a589:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 adfe1223-1ac8-4107-b8a1-f95e71b4def3:/dev/nvme0n1p4] Jul 10 17:11:06.986641 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.986560 2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 17:11:06.987934 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.987918 2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 17:11:06.993328 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.993209 2583 manager.go:216] Machine: {Timestamp:2026-07-10 17:11:06.990981008 +0000 UTC m=+0.334406629 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099279 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ac11682bcffe7447ae84dd006c97f SystemUUID:ec2ac116-82bc-ffe7-447a-e84dd006c97f BootID:01f8b578-2422-4cb7-aeb0-289ec3c1e7e0 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f1:1e:23:e6:b1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f1:1e:23:e6:b1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:bd:04:ca:d6:64 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 17:11:06.993328 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.993319 2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 17:11:06.993455 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.993442 2583 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 17:11:06.996718 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.996691 2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 17:11:06.996867 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.996720 2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-65.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 17:11:06.998170 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.998159 2583 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 17:11:06.998211 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.998173 2583 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 17:11:06.998211 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.998180 2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 17:11:06.998211 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.998193 2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 17:11:06.999709 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:06.999698 2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 17:11:07.003493 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.003481 2583 state_mem.go:36] "Initialized new in-memory state store" Jul 10 17:11:07.003630 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.003620 2583 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 17:11:07.003683 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.003653 2583 kubelet.go:486] "Attempting to sync node with API server" Jul 10 17:11:07.003712 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.003692 2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 17:11:07.003768 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.003713 2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 17:11:07.003768 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.003722 2583 kubelet.go:398] "Adding apiserver pod source" Jul 10 17:11:07.003768 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.003735 2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 17:11:07.004377 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.004363 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 17:11:07.004419 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.004390 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 17:11:07.008259 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.008237 2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 17:11:07.010589 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.010570 2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 17:11:07.010692 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.010620 2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 17:11:07.010896 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.010883 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 17:11:07.010946 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.010905 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 17:11:07.010946 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.010915 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 17:11:07.010946 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.010924 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 17:11:07.010946 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.010936 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 17:11:07.010946 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.010943 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 17:11:07.011500 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.011384 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 17:11:07.011500 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.011397 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 17:11:07.011500 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.011406 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 17:11:07.011500 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.011416 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 17:11:07.011500 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.011428 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 17:11:07.011500 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.011439 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 17:11:07.012121 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.012108 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 17:11:07.012121 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.012121 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 17:11:07.015273 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.015257 2583 server.go:1268] "Started kubelet" Jul 10 17:11:07.015388 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.015341 2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 17:11:07.015445 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.015416 2583 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 17:11:07.015795 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.015694 2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 17:11:07.016165 ip-10-0-134-65 systemd[1]: Started Kubernetes Kubelet. Jul 10 17:11:07.016806 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.016628 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z5h4f" Jul 10 17:11:07.017219 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.017128 2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 17:11:07.017551 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.017532 2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-65.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 17:11:07.017710 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.017698 2583 server.go:310] "Adding debug handlers to kubelet server" Jul 10 17:11:07.017933 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:07.017908 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 17:11:07.017933 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:07.017905 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-65.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 17:11:07.021958 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:07.021939 2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 17:11:07.023731 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.023716 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z5h4f" Jul 10 17:11:07.024321 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:07.023302 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-65.ec2.internal.18c0fcda50135745 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-65.ec2.internal,UID:ip-10-0-134-65.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-65.ec2.internal,},FirstTimestamp:2026-07-10 17:11:07.015231301 +0000 UTC m=+0.358656922,LastTimestamp:2026-07-10 17:11:07.015231301 +0000 UTC m=+0.358656922,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-65.ec2.internal,}" Jul 10 17:11:07.025216 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.025204 2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 17:11:07.025264 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.025250 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 17:11:07.025307 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:07.025302 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-65.ec2.internal\" not found" Jul 10 17:11:07.025988 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.025965 2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 17:11:07.025988 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.025982 2583 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 17:11:07.026130 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.026000 2583 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 17:11:07.026273 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.026254 2583 factory.go:55] Registering systemd factory Jul 10 17:11:07.026369 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.026277 2583 factory.go:223] Registration of the systemd container factory successfully Jul 10 17:11:07.026426 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.026378 2583 reconstruct.go:97] "Volume reconstruction finished" Jul 10 17:11:07.026426 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.026387 2583 reconciler.go:29] "Reconciler: start to sync state" Jul 10 17:11:07.026524 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.026509 2583 factory.go:153] Registering CRI-O factory Jul 10 17:11:07.026574 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.026527 2583 factory.go:223] Registration of the crio container factory successfully Jul 10 17:11:07.026614 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.026584 2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 17:11:07.026614 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.026604 2583 factory.go:103] Registering Raw factory Jul 10 17:11:07.026693 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.026616 2583 manager.go:1196] Started watching for new ooms in manager Jul 10 17:11:07.027629 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.027613 2583 manager.go:349] Starting recovery of all containers Jul 10 17:11:07.029857 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:07.029820 2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-65.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 10 17:11:07.030040 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:07.030013 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 10 17:11:07.042315 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.042297 2583 manager.go:354] Recovery completed Jul 10 17:11:07.046920 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.046901 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:11:07.049021 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.048994 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-65.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:11:07.049107 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.049027 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:11:07.049107 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.049037 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-65.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:11:07.049606 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.049591 2583 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 17:11:07.049684 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.049609 2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 17:11:07.049684 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.049594 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:11:07.049684 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.049626 2583 state_mem.go:36] "Initialized new in-memory state store" Jul 10 17:11:07.050481 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.050466 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-65.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:11:07.050554 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.050495 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:11:07.050554 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.050510 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-65.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:11:07.051213 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.051200 2583 policy_none.go:49] "None policy: Start" Jul 10 17:11:07.051264 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.051216 2583 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 17:11:07.051264 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.051227 2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 17:11:07.052465 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.052455 2583 policy_none.go:47] "Start" Jul 10 17:11:07.093580 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.093360 2583 manager.go:337] "Starting Device Plugin manager" Jul 10 17:11:07.093580 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:07.093568 2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 17:11:07.093580 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.093582 2583 server.go:87] "Starting device plugin registration server" Jul 10 17:11:07.111909 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.093841 2583 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 17:11:07.111909 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.093851 2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 17:11:07.111909 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.093961 2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 17:11:07.111909 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.094050 2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 17:11:07.111909 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.094061 2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 17:11:07.111909 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:07.094787 2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 17:11:07.111909 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:07.094846 2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-65.ec2.internal\" not found" Jul 10 17:11:07.182501 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.182393 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 17:11:07.183945 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.183926 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 17:11:07.184044 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.183965 2583 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 17:11:07.184044 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.183991 2583 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 17:11:07.184044 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:07.184025 2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 17:11:07.186649 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.186627 2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 17:11:07.194679 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.194649 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:11:07.195496 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.195478 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-65.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:11:07.195582 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.195510 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:11:07.195582 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.195523 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-65.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:11:07.195582 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.195549 2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-65.ec2.internal" Jul 10 17:11:07.204436 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.204420 2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-65.ec2.internal" Jul 10 17:11:07.204501 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:07.204443 2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-65.ec2.internal\": node \"ip-10-0-134-65.ec2.internal\" not found" Jul 10 17:11:07.232357 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:07.232332 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-65.ec2.internal\" not found" Jul 10 17:11:07.284155 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.284122 2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-65.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-65.ec2.internal"] Jul 10 17:11:07.284295 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.284196 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:11:07.285244 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.285226 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-65.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:11:07.285308 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.285261 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:11:07.285308 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.285271 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-65.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:11:07.286672 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.286647 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:11:07.286784 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.286763 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-65.ec2.internal" Jul 10 17:11:07.286880 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.286797 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:11:07.287406 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.287390 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-65.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:11:07.287488 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.287420 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:11:07.287488 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.287430 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-65.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:11:07.287488 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.287429 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-65.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:11:07.287488 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.287454 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:11:07.287488 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.287469 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-65.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:11:07.288738 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.288722 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-65.ec2.internal" Jul 10 17:11:07.288815 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.288753 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:11:07.289428 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.289402 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-65.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:11:07.289481 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.289430 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:11:07.289481 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.289443 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-65.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:11:07.303232 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:07.303211 2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-65.ec2.internal\" not found" node="ip-10-0-134-65.ec2.internal" Jul 10 17:11:07.306527 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:07.306510 2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-65.ec2.internal\" not found" node="ip-10-0-134-65.ec2.internal" Jul 10 17:11:07.328523 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.328490 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/84e6cfddc1f961a111c9509968c60461-config\") pod \"kube-apiserver-proxy-ip-10-0-134-65.ec2.internal\" (UID: \"84e6cfddc1f961a111c9509968c60461\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-65.ec2.internal" Jul 10 17:11:07.328523 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.328532 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9437cafed03781a61d9e1ba47f46277b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-65.ec2.internal\" (UID: \"9437cafed03781a61d9e1ba47f46277b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-65.ec2.internal" Jul 10 17:11:07.328714 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.328551 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9437cafed03781a61d9e1ba47f46277b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-65.ec2.internal\" (UID: \"9437cafed03781a61d9e1ba47f46277b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-65.ec2.internal" Jul 10 17:11:07.332606 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:07.332586 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-65.ec2.internal\" not found" Jul 10 17:11:07.429443 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.429409 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/84e6cfddc1f961a111c9509968c60461-config\") pod \"kube-apiserver-proxy-ip-10-0-134-65.ec2.internal\" (UID: \"84e6cfddc1f961a111c9509968c60461\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-65.ec2.internal" Jul 10 17:11:07.429594 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.429447 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/84e6cfddc1f961a111c9509968c60461-config\") pod \"kube-apiserver-proxy-ip-10-0-134-65.ec2.internal\" (UID: \"84e6cfddc1f961a111c9509968c60461\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-65.ec2.internal" Jul 10 17:11:07.429594 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.429459 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9437cafed03781a61d9e1ba47f46277b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-65.ec2.internal\" (UID: \"9437cafed03781a61d9e1ba47f46277b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-65.ec2.internal" Jul 10 17:11:07.429594 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.429478 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9437cafed03781a61d9e1ba47f46277b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-65.ec2.internal\" (UID: \"9437cafed03781a61d9e1ba47f46277b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-65.ec2.internal" Jul 10 17:11:07.429594 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.429504 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9437cafed03781a61d9e1ba47f46277b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-65.ec2.internal\" (UID: \"9437cafed03781a61d9e1ba47f46277b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-65.ec2.internal" Jul 10 17:11:07.429594 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.429510 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9437cafed03781a61d9e1ba47f46277b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-65.ec2.internal\" (UID: \"9437cafed03781a61d9e1ba47f46277b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-65.ec2.internal" Jul 10 17:11:07.433537 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:07.433492 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-65.ec2.internal\" not found" Jul 10 17:11:07.534425 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:07.534388 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-65.ec2.internal\" not found" Jul 10 17:11:07.605629 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.605595 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-65.ec2.internal" Jul 10 17:11:07.611730 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.611710 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-65.ec2.internal" Jul 10 17:11:07.635363 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:07.635334 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-65.ec2.internal\" not found" Jul 10 17:11:07.735940 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:07.735848 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-65.ec2.internal\" not found" Jul 10 17:11:07.836341 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:07.836308 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-65.ec2.internal\" not found" Jul 10 17:11:07.936824 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:07.936787 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-65.ec2.internal\" not found" Jul 10 17:11:07.956980 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.956942 2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 17:11:07.957131 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:07.957114 2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 17:11:08.036505 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:08.036436 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 17:06:08 +0000 UTC" deadline="2028-03-02 19:41:11.229868345 +0000 UTC" Jul 10 17:11:08.036505 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:08.036465 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14426h30m3.193406778s" Jul 10 17:11:08.037523 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:08.037507 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-65.ec2.internal\" not found" Jul 10 17:11:08.138368 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:08.138344 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-65.ec2.internal\" not found" Jul 10 17:11:08.147334 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:08.147290 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e6cfddc1f961a111c9509968c60461.slice/crio-57c3555174ca63f1b217aada859b60df6c7745d35bf0603f626180e8fb07cd98 WatchSource:0}: Error finding container 57c3555174ca63f1b217aada859b60df6c7745d35bf0603f626180e8fb07cd98: Status 404 returned error can't find the container with id 57c3555174ca63f1b217aada859b60df6c7745d35bf0603f626180e8fb07cd98 Jul 10 17:11:08.147533 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:08.147514 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9437cafed03781a61d9e1ba47f46277b.slice/crio-c8f0b7259529f3bf0403c84423965c44475d9235230d67a4a0ef299b29285052 WatchSource:0}: Error finding container c8f0b7259529f3bf0403c84423965c44475d9235230d67a4a0ef299b29285052: Status 404 returned error can't find the container with id c8f0b7259529f3bf0403c84423965c44475d9235230d67a4a0ef299b29285052 Jul 10 17:11:08.154165 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:08.154148 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 17:11:08.188836 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:08.188779 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-65.ec2.internal" event={"ID":"9437cafed03781a61d9e1ba47f46277b","Type":"ContainerStarted","Data":"c8f0b7259529f3bf0403c84423965c44475d9235230d67a4a0ef299b29285052"} Jul 10 17:11:08.189788 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:08.189757 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-65.ec2.internal" event={"ID":"84e6cfddc1f961a111c9509968c60461","Type":"ContainerStarted","Data":"57c3555174ca63f1b217aada859b60df6c7745d35bf0603f626180e8fb07cd98"} Jul 10 17:11:08.201392 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:08.201372 2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 17:11:08.239243 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:08.239205 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-65.ec2.internal\" not found" Jul 10 17:11:08.264228 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:08.264196 2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 17:11:08.340017 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:08.339920 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-65.ec2.internal\" not found" Jul 10 17:11:09.456221 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:09.456177 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-65.ec2.internal\" not found" Jul 10 17:11:09.557038 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:09.557005 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-65.ec2.internal\" not found" Jul 10 17:11:09.633808 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:09.633727 2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 17:11:09.641711 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:09.641690 2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-65.ec2.internal" Jul 10 17:11:09.654032 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:09.653982 2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 17:11:09.654968 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:09.654944 2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-65.ec2.internal" Jul 10 17:11:09.667006 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:09.666979 2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 17:11:10.020956 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.020879 2583 apiserver.go:52] "Watching apiserver" Jul 10 17:11:10.028252 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.028222 2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 17:11:10.028738 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.028706 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-js69w","kube-system/global-pull-secret-syncer-smz4h","kube-system/konnectivity-agent-lql7p","openshift-dns/node-resolver-bq9tx","openshift-image-registry/node-ca-5krz5","openshift-multus/multus-52frq","openshift-multus/multus-additional-cni-plugins-vk6rc","openshift-multus/network-metrics-daemon-h9jx7","openshift-network-diagnostics/network-check-target-7mnkc","kube-system/kube-apiserver-proxy-ip-10-0-134-65.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zmbx","openshift-cluster-node-tuning-operator/tuned-9f4lz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-65.ec2.internal","openshift-network-operator/iptables-alerter-rbwdx"] Jul 10 17:11:10.030480 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.030454 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vk6rc" Jul 10 17:11:10.031877 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.031854 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zmbx" Jul 10 17:11:10.033150 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.033130 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lql7p" Jul 10 17:11:10.034715 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.034456 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 17:11:10.034715 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.034471 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bq9tx" Jul 10 17:11:10.034715 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.034488 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rlcnb\"" Jul 10 17:11:10.034926 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.034807 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 17:11:10.034926 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.034849 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 17:11:10.035023 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.034978 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 17:11:10.035023 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.035018 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 17:11:10.035899 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.035880 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5krz5" Jul 10 17:11:10.037837 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.037295 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-52frq" Jul 10 17:11:10.037837 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.037679 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 17:11:10.040117 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.040098 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 17:11:10.041105 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.040340 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 17:11:10.041105 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.040906 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 17:11:10.041105 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.040923 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ld7xh\"" Jul 10 17:11:10.041105 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.041103 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 17:11:10.041729 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.041711 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-c7d8t\"" Jul 10 17:11:10.041819 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.041775 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 17:11:10.041944 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.041930 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 17:11:10.042000 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.041956 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 17:11:10.042105 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.042090 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 17:11:10.042257 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.042243 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 17:11:10.042531 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.042516 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 17:11:10.042716 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.042701 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-79jhq\"" Jul 10 17:11:10.042900 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.042885 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 17:11:10.043004 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.042989 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7mnkc" Jul 10 17:11:10.043101 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:10.043082 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7mnkc" podUID="7000764e-d442-4c90-aa00-97b83a65e4b6" Jul 10 17:11:10.044685 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.044402 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rbwdx" Jul 10 17:11:10.045332 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.045315 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9gld9\"" Jul 10 17:11:10.045430 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.045372 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xl2n8\"" Jul 10 17:11:10.045811 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.045794 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:11:10.045885 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:10.045869 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smz4h" podUID="ea2e6b67-bf39-4196-a657-7859b8b73199" Jul 10 17:11:10.047252 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.047233 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9jx7" Jul 10 17:11:10.047343 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:10.047304 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9jx7" podUID="d177790b-fe2d-469e-9dff-c653997fe096" Jul 10 17:11:10.048766 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.048748 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.050779 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.050757 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 17:11:10.050859 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.050833 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.051139 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.051121 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 17:11:10.055633 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.051479 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 17:11:10.055633 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.054312 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 17:11:10.055633 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.054489 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 17:11:10.055633 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.054543 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 17:11:10.055633 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.054774 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5jp2b\"" Jul 10 17:11:10.055633 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.054952 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 17:11:10.055633 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.054973 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 17:11:10.055633 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.055111 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mjgrt\"" Jul 10 17:11:10.055633 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.055127 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 17:11:10.055633 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.055361 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgrcr\" (UniqueName: \"kubernetes.io/projected/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-kube-api-access-bgrcr\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.055633 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.055455 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0bd5b01a-f45d-41bf-aa6e-3fb6151cd008-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7zmbx\" (UID: \"0bd5b01a-f45d-41bf-aa6e-3fb6151cd008\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zmbx" Jul 10 17:11:10.055633 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.055520 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crgqw\" (UniqueName: \"kubernetes.io/projected/9369f652-8e06-4d0a-b00e-07ad43e2bec1-kube-api-access-crgqw\") pod \"node-resolver-bq9tx\" (UID: \"9369f652-8e06-4d0a-b00e-07ad43e2bec1\") " pod="openshift-dns/node-resolver-bq9tx" Jul 10 17:11:10.055633 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.055557 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e13ba88-7d4c-42b8-a992-95b3c447b222-cni-binary-copy\") pod \"multus-additional-cni-plugins-vk6rc\" (UID: \"1e13ba88-7d4c-42b8-a992-95b3c447b222\") " pod="openshift-multus/multus-additional-cni-plugins-vk6rc" Jul 10 17:11:10.055633 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.055602 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e13ba88-7d4c-42b8-a992-95b3c447b222-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vk6rc\" (UID: \"1e13ba88-7d4c-42b8-a992-95b3c447b222\") " pod="openshift-multus/multus-additional-cni-plugins-vk6rc" Jul 10 17:11:10.056489 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.055653 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1e13ba88-7d4c-42b8-a992-95b3c447b222-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vk6rc\" (UID: \"1e13ba88-7d4c-42b8-a992-95b3c447b222\") " pod="openshift-multus/multus-additional-cni-plugins-vk6rc" Jul 10 17:11:10.056489 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.055708 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/df9374c1-e474-4ceb-b6d6-a8879a52e376-serviceca\") pod \"node-ca-5krz5\" (UID: \"df9374c1-e474-4ceb-b6d6-a8879a52e376\") " pod="openshift-image-registry/node-ca-5krz5" Jul 10 17:11:10.056489 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.055801 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0bd5b01a-f45d-41bf-aa6e-3fb6151cd008-etc-selinux\") pod \"aws-ebs-csi-driver-node-7zmbx\" (UID: \"0bd5b01a-f45d-41bf-aa6e-3fb6151cd008\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zmbx" Jul 10 17:11:10.056489 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.055843 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0bd5b01a-f45d-41bf-aa6e-3fb6151cd008-sys-fs\") pod \"aws-ebs-csi-driver-node-7zmbx\" (UID: \"0bd5b01a-f45d-41bf-aa6e-3fb6151cd008\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zmbx" Jul 10 17:11:10.056489 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.055878 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjldt\" (UniqueName: \"kubernetes.io/projected/0bd5b01a-f45d-41bf-aa6e-3fb6151cd008-kube-api-access-jjldt\") pod \"aws-ebs-csi-driver-node-7zmbx\" (UID: \"0bd5b01a-f45d-41bf-aa6e-3fb6151cd008\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zmbx" Jul 10 17:11:10.056489 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.056231 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9369f652-8e06-4d0a-b00e-07ad43e2bec1-tmp-dir\") pod \"node-resolver-bq9tx\" (UID: \"9369f652-8e06-4d0a-b00e-07ad43e2bec1\") " pod="openshift-dns/node-resolver-bq9tx" Jul 10 17:11:10.056489 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.056262 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-multus-cni-dir\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.056489 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.056279 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-cni-binary-copy\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.056489 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.056315 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-multus-daemon-config\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.056489 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.056359 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0bd5b01a-f45d-41bf-aa6e-3fb6151cd008-device-dir\") pod \"aws-ebs-csi-driver-node-7zmbx\" (UID: \"0bd5b01a-f45d-41bf-aa6e-3fb6151cd008\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zmbx" Jul 10 17:11:10.056489 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.056407 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3756f101-49b1-4754-ad4a-13c572eec69c-konnectivity-ca\") pod \"konnectivity-agent-lql7p\" (UID: \"3756f101-49b1-4754-ad4a-13c572eec69c\") " pod="kube-system/konnectivity-agent-lql7p" Jul 10 17:11:10.056489 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.056432 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9369f652-8e06-4d0a-b00e-07ad43e2bec1-hosts-file\") pod \"node-resolver-bq9tx\" (UID: \"9369f652-8e06-4d0a-b00e-07ad43e2bec1\") " pod="openshift-dns/node-resolver-bq9tx" Jul 10 17:11:10.056489 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.056439 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cnqnd\"" Jul 10 17:11:10.056489 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.056456 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgdsg\" (UniqueName: \"kubernetes.io/projected/1e13ba88-7d4c-42b8-a992-95b3c447b222-kube-api-access-fgdsg\") pod \"multus-additional-cni-plugins-vk6rc\" (UID: \"1e13ba88-7d4c-42b8-a992-95b3c447b222\") " pod="openshift-multus/multus-additional-cni-plugins-vk6rc" Jul 10 17:11:10.056489 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.056479 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ngr6\" (UniqueName: \"kubernetes.io/projected/df9374c1-e474-4ceb-b6d6-a8879a52e376-kube-api-access-9ngr6\") pod \"node-ca-5krz5\" (UID: \"df9374c1-e474-4ceb-b6d6-a8879a52e376\") " pod="openshift-image-registry/node-ca-5krz5" Jul 10 17:11:10.057176 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.056501 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-host-run-netns\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.057176 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.056524 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-host-var-lib-cni-bin\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.057176 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.056546 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-hostroot\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.057176 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.056569 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0bd5b01a-f45d-41bf-aa6e-3fb6151cd008-socket-dir\") pod \"aws-ebs-csi-driver-node-7zmbx\" (UID: \"0bd5b01a-f45d-41bf-aa6e-3fb6151cd008\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zmbx" Jul 10 17:11:10.057176 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.056592 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e13ba88-7d4c-42b8-a992-95b3c447b222-system-cni-dir\") pod \"multus-additional-cni-plugins-vk6rc\" (UID: \"1e13ba88-7d4c-42b8-a992-95b3c447b222\") " pod="openshift-multus/multus-additional-cni-plugins-vk6rc" Jul 10 17:11:10.057176 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.056615 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-cnibin\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.057176 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.056649 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-host-var-lib-cni-multus\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.057176 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.056671 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0bd5b01a-f45d-41bf-aa6e-3fb6151cd008-registration-dir\") pod \"aws-ebs-csi-driver-node-7zmbx\" (UID: \"0bd5b01a-f45d-41bf-aa6e-3fb6151cd008\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zmbx" Jul 10 17:11:10.057176 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.056692 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df9374c1-e474-4ceb-b6d6-a8879a52e376-host\") pod \"node-ca-5krz5\" (UID: \"df9374c1-e474-4ceb-b6d6-a8879a52e376\") " pod="openshift-image-registry/node-ca-5krz5" Jul 10 17:11:10.057176 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.056716 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-os-release\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.057176 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.056750 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-multus-socket-dir-parent\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.057176 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.056776 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-host-run-k8s-cni-cncf-io\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.057176 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.056798 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-etc-kubernetes\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.057176 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.056837 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsl5d\" (UniqueName: \"kubernetes.io/projected/7000764e-d442-4c90-aa00-97b83a65e4b6-kube-api-access-rsl5d\") pod \"network-check-target-7mnkc\" (UID: \"7000764e-d442-4c90-aa00-97b83a65e4b6\") " pod="openshift-network-diagnostics/network-check-target-7mnkc" Jul 10 17:11:10.057176 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.056862 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e13ba88-7d4c-42b8-a992-95b3c447b222-cnibin\") pod \"multus-additional-cni-plugins-vk6rc\" (UID: \"1e13ba88-7d4c-42b8-a992-95b3c447b222\") " pod="openshift-multus/multus-additional-cni-plugins-vk6rc" Jul 10 17:11:10.057176 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.056885 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1e13ba88-7d4c-42b8-a992-95b3c447b222-os-release\") pod \"multus-additional-cni-plugins-vk6rc\" (UID: \"1e13ba88-7d4c-42b8-a992-95b3c447b222\") " pod="openshift-multus/multus-additional-cni-plugins-vk6rc" Jul 10 17:11:10.057841 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.056902 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1e13ba88-7d4c-42b8-a992-95b3c447b222-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vk6rc\" (UID: \"1e13ba88-7d4c-42b8-a992-95b3c447b222\") " pod="openshift-multus/multus-additional-cni-plugins-vk6rc" Jul 10 17:11:10.057841 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.056916 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-host-var-lib-kubelet\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.057841 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.056931 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-host-run-multus-certs\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.057841 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.056953 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3756f101-49b1-4754-ad4a-13c572eec69c-agent-certs\") pod \"konnectivity-agent-lql7p\" (UID: \"3756f101-49b1-4754-ad4a-13c572eec69c\") " pod="kube-system/konnectivity-agent-lql7p" Jul 10 17:11:10.057841 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.056981 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-system-cni-dir\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.057841 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.057003 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-multus-conf-dir\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.059022 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.058842 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 17:11:10.059022 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.058944 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 17:11:10.065553 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.065528 2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 17:11:10.093294 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.092828 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-pptxs" Jul 10 17:11:10.108681 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.108658 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pptxs" Jul 10 17:11:10.143029 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.143000 2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 17:11:10.157618 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.157585 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/df9374c1-e474-4ceb-b6d6-a8879a52e376-serviceca\") pod \"node-ca-5krz5\" (UID: \"df9374c1-e474-4ceb-b6d6-a8879a52e376\") " pod="openshift-image-registry/node-ca-5krz5" Jul 10 17:11:10.157618 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.157618 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-host-kubelet\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.157846 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.157647 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-systemd-units\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.157846 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.157670 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-etc-openvswitch\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.157846 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.157753 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-host-run-ovn-kubernetes\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.157846 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.157797 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0bd5b01a-f45d-41bf-aa6e-3fb6151cd008-etc-selinux\") pod \"aws-ebs-csi-driver-node-7zmbx\" (UID: \"0bd5b01a-f45d-41bf-aa6e-3fb6151cd008\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zmbx" Jul 10 17:11:10.157846 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.157834 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9369f652-8e06-4d0a-b00e-07ad43e2bec1-tmp-dir\") pod \"node-resolver-bq9tx\" (UID: \"9369f652-8e06-4d0a-b00e-07ad43e2bec1\") " pod="openshift-dns/node-resolver-bq9tx" Jul 10 17:11:10.157846 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.157850 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-multus-cni-dir\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.158122 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.157871 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-cni-binary-copy\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.158122 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.157892 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/73049e7b-08d4-4772-8dfc-531b8449d42b-etc-tuned\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.158122 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.157923 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5fln\" (UniqueName: \"kubernetes.io/projected/d177790b-fe2d-469e-9dff-c653997fe096-kube-api-access-g5fln\") pod \"network-metrics-daemon-h9jx7\" (UID: \"d177790b-fe2d-469e-9dff-c653997fe096\") " pod="openshift-multus/network-metrics-daemon-h9jx7" Jul 10 17:11:10.158122 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.157930 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0bd5b01a-f45d-41bf-aa6e-3fb6151cd008-etc-selinux\") pod \"aws-ebs-csi-driver-node-7zmbx\" (UID: \"0bd5b01a-f45d-41bf-aa6e-3fb6151cd008\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zmbx" Jul 10 17:11:10.158122 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.157949 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-log-socket\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.158122 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.157982 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0bd5b01a-f45d-41bf-aa6e-3fb6151cd008-device-dir\") pod \"aws-ebs-csi-driver-node-7zmbx\" (UID: \"0bd5b01a-f45d-41bf-aa6e-3fb6151cd008\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zmbx" Jul 10 17:11:10.158122 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158006 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9369f652-8e06-4d0a-b00e-07ad43e2bec1-hosts-file\") pod \"node-resolver-bq9tx\" (UID: \"9369f652-8e06-4d0a-b00e-07ad43e2bec1\") " pod="openshift-dns/node-resolver-bq9tx" Jul 10 17:11:10.158122 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158025 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgdsg\" (UniqueName: \"kubernetes.io/projected/1e13ba88-7d4c-42b8-a992-95b3c447b222-kube-api-access-fgdsg\") pod \"multus-additional-cni-plugins-vk6rc\" (UID: \"1e13ba88-7d4c-42b8-a992-95b3c447b222\") " pod="openshift-multus/multus-additional-cni-plugins-vk6rc" Jul 10 17:11:10.158122 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158049 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-host-var-lib-cni-bin\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.158122 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158066 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-hostroot\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.158122 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158091 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d177790b-fe2d-469e-9dff-c653997fe096-metrics-certs\") pod \"network-metrics-daemon-h9jx7\" (UID: \"d177790b-fe2d-469e-9dff-c653997fe096\") " pod="openshift-multus/network-metrics-daemon-h9jx7" Jul 10 17:11:10.158122 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158096 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-multus-cni-dir\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.158122 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158108 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-var-lib-openvswitch\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.158724 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158144 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-host-cni-bin\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.158724 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158159 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e13ba88-7d4c-42b8-a992-95b3c447b222-system-cni-dir\") pod \"multus-additional-cni-plugins-vk6rc\" (UID: \"1e13ba88-7d4c-42b8-a992-95b3c447b222\") " pod="openshift-multus/multus-additional-cni-plugins-vk6rc" Jul 10 17:11:10.158724 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158160 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9369f652-8e06-4d0a-b00e-07ad43e2bec1-hosts-file\") pod \"node-resolver-bq9tx\" (UID: \"9369f652-8e06-4d0a-b00e-07ad43e2bec1\") " pod="openshift-dns/node-resolver-bq9tx" Jul 10 17:11:10.158724 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158175 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-host-var-lib-cni-multus\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.158724 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158191 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c7288bbc-320f-4eb8-ac48-d28666cebcdf-iptables-alerter-script\") pod \"iptables-alerter-rbwdx\" (UID: \"c7288bbc-320f-4eb8-ac48-d28666cebcdf\") " pod="openshift-network-operator/iptables-alerter-rbwdx" Jul 10 17:11:10.158724 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158211 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-node-log\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.158724 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158230 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0bd5b01a-f45d-41bf-aa6e-3fb6151cd008-device-dir\") pod \"aws-ebs-csi-driver-node-7zmbx\" (UID: \"0bd5b01a-f45d-41bf-aa6e-3fb6151cd008\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zmbx" Jul 10 17:11:10.158724 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158240 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df9374c1-e474-4ceb-b6d6-a8879a52e376-host\") pod \"node-ca-5krz5\" (UID: \"df9374c1-e474-4ceb-b6d6-a8879a52e376\") " pod="openshift-image-registry/node-ca-5krz5" Jul 10 17:11:10.158724 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158255 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-os-release\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.158724 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158300 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-host-run-k8s-cni-cncf-io\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.158724 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158313 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9369f652-8e06-4d0a-b00e-07ad43e2bec1-tmp-dir\") pod \"node-resolver-bq9tx\" (UID: \"9369f652-8e06-4d0a-b00e-07ad43e2bec1\") " pod="openshift-dns/node-resolver-bq9tx" Jul 10 17:11:10.158724 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158318 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/73049e7b-08d4-4772-8dfc-531b8449d42b-etc-modprobe-d\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.158724 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158333 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/73049e7b-08d4-4772-8dfc-531b8449d42b-sys\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.158724 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158319 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/df9374c1-e474-4ceb-b6d6-a8879a52e376-serviceca\") pod \"node-ca-5krz5\" (UID: \"df9374c1-e474-4ceb-b6d6-a8879a52e376\") " pod="openshift-image-registry/node-ca-5krz5" Jul 10 17:11:10.158724 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158355 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73049e7b-08d4-4772-8dfc-531b8449d42b-var-lib-kubelet\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.158724 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158378 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-host-cni-netd\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.158724 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158414 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-hostroot\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.159371 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158427 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e13ba88-7d4c-42b8-a992-95b3c447b222-cnibin\") pod \"multus-additional-cni-plugins-vk6rc\" (UID: \"1e13ba88-7d4c-42b8-a992-95b3c447b222\") " pod="openshift-multus/multus-additional-cni-plugins-vk6rc" Jul 10 17:11:10.159371 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158443 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-host-var-lib-cni-bin\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.159371 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158454 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df9374c1-e474-4ceb-b6d6-a8879a52e376-host\") pod \"node-ca-5krz5\" (UID: \"df9374c1-e474-4ceb-b6d6-a8879a52e376\") " pod="openshift-image-registry/node-ca-5krz5" Jul 10 17:11:10.159371 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158465 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-host-var-lib-cni-multus\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.159371 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158498 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e13ba88-7d4c-42b8-a992-95b3c447b222-system-cni-dir\") pod \"multus-additional-cni-plugins-vk6rc\" (UID: \"1e13ba88-7d4c-42b8-a992-95b3c447b222\") " pod="openshift-multus/multus-additional-cni-plugins-vk6rc" Jul 10 17:11:10.159371 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158530 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e13ba88-7d4c-42b8-a992-95b3c447b222-cnibin\") pod \"multus-additional-cni-plugins-vk6rc\" (UID: \"1e13ba88-7d4c-42b8-a992-95b3c447b222\") " pod="openshift-multus/multus-additional-cni-plugins-vk6rc" Jul 10 17:11:10.159371 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158530 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-cni-binary-copy\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.159371 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158545 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-os-release\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.159371 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158547 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-host-run-k8s-cni-cncf-io\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.159371 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158551 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/73049e7b-08d4-4772-8dfc-531b8449d42b-etc-sysconfig\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.159371 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158599 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.159371 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158636 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3756f101-49b1-4754-ad4a-13c572eec69c-agent-certs\") pod \"konnectivity-agent-lql7p\" (UID: \"3756f101-49b1-4754-ad4a-13c572eec69c\") " pod="kube-system/konnectivity-agent-lql7p" Jul 10 17:11:10.159371 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158663 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-system-cni-dir\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.159371 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158752 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/73049e7b-08d4-4772-8dfc-531b8449d42b-etc-sysctl-d\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.159371 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158794 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-system-cni-dir\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.159371 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158838 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lck8\" (UniqueName: \"kubernetes.io/projected/c7288bbc-320f-4eb8-ac48-d28666cebcdf-kube-api-access-9lck8\") pod \"iptables-alerter-rbwdx\" (UID: \"c7288bbc-320f-4eb8-ac48-d28666cebcdf\") " pod="openshift-network-operator/iptables-alerter-rbwdx" Jul 10 17:11:10.159371 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158868 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e41baaf7-4171-4df1-8bdd-d0697a61fdce-env-overrides\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.160140 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158895 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0bd5b01a-f45d-41bf-aa6e-3fb6151cd008-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7zmbx\" (UID: \"0bd5b01a-f45d-41bf-aa6e-3fb6151cd008\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zmbx" Jul 10 17:11:10.160140 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158919 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ea2e6b67-bf39-4196-a657-7859b8b73199-kubelet-config\") pod \"global-pull-secret-syncer-smz4h\" (UID: \"ea2e6b67-bf39-4196-a657-7859b8b73199\") " pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:11:10.160140 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158925 2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 17:11:10.160140 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158942 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/73049e7b-08d4-4772-8dfc-531b8449d42b-etc-sysctl-conf\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.160140 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158949 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0bd5b01a-f45d-41bf-aa6e-3fb6151cd008-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7zmbx\" (UID: \"0bd5b01a-f45d-41bf-aa6e-3fb6151cd008\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zmbx" Jul 10 17:11:10.160140 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158961 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/73049e7b-08d4-4772-8dfc-531b8449d42b-lib-modules\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.160140 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158980 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/73049e7b-08d4-4772-8dfc-531b8449d42b-host\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.160140 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.158997 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73049e7b-08d4-4772-8dfc-531b8449d42b-tmp\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.160140 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159028 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5vw9\" (UniqueName: \"kubernetes.io/projected/73049e7b-08d4-4772-8dfc-531b8449d42b-kube-api-access-m5vw9\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.160140 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159055 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1e13ba88-7d4c-42b8-a992-95b3c447b222-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vk6rc\" (UID: \"1e13ba88-7d4c-42b8-a992-95b3c447b222\") " pod="openshift-multus/multus-additional-cni-plugins-vk6rc" Jul 10 17:11:10.160140 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159076 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ea2e6b67-bf39-4196-a657-7859b8b73199-dbus\") pod \"global-pull-secret-syncer-smz4h\" (UID: \"ea2e6b67-bf39-4196-a657-7859b8b73199\") " pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:11:10.160140 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159097 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0bd5b01a-f45d-41bf-aa6e-3fb6151cd008-sys-fs\") pod \"aws-ebs-csi-driver-node-7zmbx\" (UID: \"0bd5b01a-f45d-41bf-aa6e-3fb6151cd008\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zmbx" Jul 10 17:11:10.160140 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159129 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjldt\" (UniqueName: \"kubernetes.io/projected/0bd5b01a-f45d-41bf-aa6e-3fb6151cd008-kube-api-access-jjldt\") pod \"aws-ebs-csi-driver-node-7zmbx\" (UID: \"0bd5b01a-f45d-41bf-aa6e-3fb6151cd008\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zmbx" Jul 10 17:11:10.160140 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159181 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-multus-daemon-config\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.160140 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159191 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0bd5b01a-f45d-41bf-aa6e-3fb6151cd008-sys-fs\") pod \"aws-ebs-csi-driver-node-7zmbx\" (UID: \"0bd5b01a-f45d-41bf-aa6e-3fb6151cd008\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zmbx" Jul 10 17:11:10.160140 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159208 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-etc-kubernetes\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.160140 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159242 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73049e7b-08d4-4772-8dfc-531b8449d42b-etc-kubernetes\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.160851 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159266 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-run-openvswitch\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.160851 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159244 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-etc-kubernetes\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.160851 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159301 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3756f101-49b1-4754-ad4a-13c572eec69c-konnectivity-ca\") pod \"konnectivity-agent-lql7p\" (UID: \"3756f101-49b1-4754-ad4a-13c572eec69c\") " pod="kube-system/konnectivity-agent-lql7p" Jul 10 17:11:10.160851 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159407 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ngr6\" (UniqueName: \"kubernetes.io/projected/df9374c1-e474-4ceb-b6d6-a8879a52e376-kube-api-access-9ngr6\") pod \"node-ca-5krz5\" (UID: \"df9374c1-e474-4ceb-b6d6-a8879a52e376\") " pod="openshift-image-registry/node-ca-5krz5" Jul 10 17:11:10.160851 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159435 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-host-run-netns\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.160851 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159460 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-run-ovn\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.160851 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159482 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e41baaf7-4171-4df1-8bdd-d0697a61fdce-ovnkube-config\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.160851 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159510 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-host-run-netns\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.160851 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159528 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0bd5b01a-f45d-41bf-aa6e-3fb6151cd008-socket-dir\") pod \"aws-ebs-csi-driver-node-7zmbx\" (UID: \"0bd5b01a-f45d-41bf-aa6e-3fb6151cd008\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zmbx" Jul 10 17:11:10.160851 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159571 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-cnibin\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.160851 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159623 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0bd5b01a-f45d-41bf-aa6e-3fb6151cd008-registration-dir\") pod \"aws-ebs-csi-driver-node-7zmbx\" (UID: \"0bd5b01a-f45d-41bf-aa6e-3fb6151cd008\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zmbx" Jul 10 17:11:10.160851 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159646 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-multus-socket-dir-parent\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.160851 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159665 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsl5d\" (UniqueName: \"kubernetes.io/projected/7000764e-d442-4c90-aa00-97b83a65e4b6-kube-api-access-rsl5d\") pod \"network-check-target-7mnkc\" (UID: \"7000764e-d442-4c90-aa00-97b83a65e4b6\") " pod="openshift-network-diagnostics/network-check-target-7mnkc" Jul 10 17:11:10.160851 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159693 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1e13ba88-7d4c-42b8-a992-95b3c447b222-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vk6rc\" (UID: \"1e13ba88-7d4c-42b8-a992-95b3c447b222\") " pod="openshift-multus/multus-additional-cni-plugins-vk6rc" Jul 10 17:11:10.160851 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159689 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea2e6b67-bf39-4196-a657-7859b8b73199-original-pull-secret\") pod \"global-pull-secret-syncer-smz4h\" (UID: \"ea2e6b67-bf39-4196-a657-7859b8b73199\") " pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:11:10.160851 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159730 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0bd5b01a-f45d-41bf-aa6e-3fb6151cd008-registration-dir\") pod \"aws-ebs-csi-driver-node-7zmbx\" (UID: \"0bd5b01a-f45d-41bf-aa6e-3fb6151cd008\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zmbx" Jul 10 17:11:10.160851 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159739 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-multus-daemon-config\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.161417 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159705 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0bd5b01a-f45d-41bf-aa6e-3fb6151cd008-socket-dir\") pod \"aws-ebs-csi-driver-node-7zmbx\" (UID: \"0bd5b01a-f45d-41bf-aa6e-3fb6151cd008\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zmbx" Jul 10 17:11:10.161417 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159750 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-cnibin\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.161417 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159731 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-host-slash\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.161417 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159785 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-multus-socket-dir-parent\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.161417 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159826 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e41baaf7-4171-4df1-8bdd-d0697a61fdce-ovn-node-metrics-cert\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.161417 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159874 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1e13ba88-7d4c-42b8-a992-95b3c447b222-os-release\") pod \"multus-additional-cni-plugins-vk6rc\" (UID: \"1e13ba88-7d4c-42b8-a992-95b3c447b222\") " pod="openshift-multus/multus-additional-cni-plugins-vk6rc" Jul 10 17:11:10.161417 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159892 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3756f101-49b1-4754-ad4a-13c572eec69c-konnectivity-ca\") pod \"konnectivity-agent-lql7p\" (UID: \"3756f101-49b1-4754-ad4a-13c572eec69c\") " pod="kube-system/konnectivity-agent-lql7p" Jul 10 17:11:10.161417 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159911 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1e13ba88-7d4c-42b8-a992-95b3c447b222-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vk6rc\" (UID: \"1e13ba88-7d4c-42b8-a992-95b3c447b222\") " pod="openshift-multus/multus-additional-cni-plugins-vk6rc" Jul 10 17:11:10.161417 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159935 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-host-var-lib-kubelet\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.161417 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.159969 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-host-var-lib-kubelet\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.161417 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.160008 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-host-run-multus-certs\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.161417 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.160047 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/73049e7b-08d4-4772-8dfc-531b8449d42b-run\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.161417 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.160059 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1e13ba88-7d4c-42b8-a992-95b3c447b222-os-release\") pod \"multus-additional-cni-plugins-vk6rc\" (UID: \"1e13ba88-7d4c-42b8-a992-95b3c447b222\") " pod="openshift-multus/multus-additional-cni-plugins-vk6rc" Jul 10 17:11:10.161417 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.160072 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-host-run-multus-certs\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.161417 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.160130 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-host-run-netns\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.161417 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.160151 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6tj5\" (UniqueName: \"kubernetes.io/projected/e41baaf7-4171-4df1-8bdd-d0697a61fdce-kube-api-access-g6tj5\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.161417 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.160167 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-multus-conf-dir\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.161930 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.160190 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgrcr\" (UniqueName: \"kubernetes.io/projected/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-kube-api-access-bgrcr\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.161930 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.160204 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-run-systemd\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.161930 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.160213 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-multus-conf-dir\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.161930 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.160218 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e41baaf7-4171-4df1-8bdd-d0697a61fdce-ovnkube-script-lib\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.161930 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.160260 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crgqw\" (UniqueName: \"kubernetes.io/projected/9369f652-8e06-4d0a-b00e-07ad43e2bec1-kube-api-access-crgqw\") pod \"node-resolver-bq9tx\" (UID: \"9369f652-8e06-4d0a-b00e-07ad43e2bec1\") " pod="openshift-dns/node-resolver-bq9tx" Jul 10 17:11:10.161930 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.160276 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e13ba88-7d4c-42b8-a992-95b3c447b222-cni-binary-copy\") pod \"multus-additional-cni-plugins-vk6rc\" (UID: \"1e13ba88-7d4c-42b8-a992-95b3c447b222\") " pod="openshift-multus/multus-additional-cni-plugins-vk6rc" Jul 10 17:11:10.161930 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.160291 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e13ba88-7d4c-42b8-a992-95b3c447b222-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vk6rc\" (UID: \"1e13ba88-7d4c-42b8-a992-95b3c447b222\") " pod="openshift-multus/multus-additional-cni-plugins-vk6rc" Jul 10 17:11:10.161930 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.160315 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/73049e7b-08d4-4772-8dfc-531b8449d42b-etc-systemd\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.161930 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.160338 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7288bbc-320f-4eb8-ac48-d28666cebcdf-host-slash\") pod \"iptables-alerter-rbwdx\" (UID: \"c7288bbc-320f-4eb8-ac48-d28666cebcdf\") " pod="openshift-network-operator/iptables-alerter-rbwdx" Jul 10 17:11:10.161930 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.160445 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1e13ba88-7d4c-42b8-a992-95b3c447b222-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vk6rc\" (UID: \"1e13ba88-7d4c-42b8-a992-95b3c447b222\") " pod="openshift-multus/multus-additional-cni-plugins-vk6rc" Jul 10 17:11:10.161930 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.160541 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e13ba88-7d4c-42b8-a992-95b3c447b222-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vk6rc\" (UID: \"1e13ba88-7d4c-42b8-a992-95b3c447b222\") " pod="openshift-multus/multus-additional-cni-plugins-vk6rc" Jul 10 17:11:10.161930 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.160689 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e13ba88-7d4c-42b8-a992-95b3c447b222-cni-binary-copy\") pod \"multus-additional-cni-plugins-vk6rc\" (UID: \"1e13ba88-7d4c-42b8-a992-95b3c447b222\") " pod="openshift-multus/multus-additional-cni-plugins-vk6rc" Jul 10 17:11:10.163759 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.163736 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3756f101-49b1-4754-ad4a-13c572eec69c-agent-certs\") pod \"konnectivity-agent-lql7p\" (UID: \"3756f101-49b1-4754-ad4a-13c572eec69c\") " pod="kube-system/konnectivity-agent-lql7p" Jul 10 17:11:10.169911 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:10.169887 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 17:11:10.169911 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:10.169913 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 17:11:10.169911 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:10.169927 2583 projected.go:196] Error preparing data for projected volume kube-api-access-rsl5d for pod openshift-network-diagnostics/network-check-target-7mnkc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:11:10.170443 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:10.170014 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7000764e-d442-4c90-aa00-97b83a65e4b6-kube-api-access-rsl5d podName:7000764e-d442-4c90-aa00-97b83a65e4b6 nodeName:}" failed. No retries permitted until 2026-07-10 17:11:10.669977044 +0000 UTC m=+2.997674593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rsl5d" (UniqueName: "kubernetes.io/projected/7000764e-d442-4c90-aa00-97b83a65e4b6-kube-api-access-rsl5d") pod "network-check-target-7mnkc" (UID: "7000764e-d442-4c90-aa00-97b83a65e4b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:11:10.177964 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.172144 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgdsg\" (UniqueName: \"kubernetes.io/projected/1e13ba88-7d4c-42b8-a992-95b3c447b222-kube-api-access-fgdsg\") pod \"multus-additional-cni-plugins-vk6rc\" (UID: \"1e13ba88-7d4c-42b8-a992-95b3c447b222\") " pod="openshift-multus/multus-additional-cni-plugins-vk6rc" Jul 10 17:11:10.177964 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.172593 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crgqw\" (UniqueName: \"kubernetes.io/projected/9369f652-8e06-4d0a-b00e-07ad43e2bec1-kube-api-access-crgqw\") pod \"node-resolver-bq9tx\" (UID: \"9369f652-8e06-4d0a-b00e-07ad43e2bec1\") " pod="openshift-dns/node-resolver-bq9tx" Jul 10 17:11:10.177964 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.172687 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ngr6\" (UniqueName: \"kubernetes.io/projected/df9374c1-e474-4ceb-b6d6-a8879a52e376-kube-api-access-9ngr6\") pod \"node-ca-5krz5\" (UID: \"df9374c1-e474-4ceb-b6d6-a8879a52e376\") " pod="openshift-image-registry/node-ca-5krz5" Jul 10 17:11:10.177964 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.173685 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgrcr\" (UniqueName: \"kubernetes.io/projected/7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7-kube-api-access-bgrcr\") pod \"multus-52frq\" (UID: \"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7\") " pod="openshift-multus/multus-52frq" Jul 10 17:11:10.177964 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.173900 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjldt\" (UniqueName: \"kubernetes.io/projected/0bd5b01a-f45d-41bf-aa6e-3fb6151cd008-kube-api-access-jjldt\") pod \"aws-ebs-csi-driver-node-7zmbx\" (UID: \"0bd5b01a-f45d-41bf-aa6e-3fb6151cd008\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zmbx" Jul 10 17:11:10.261002 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.260958 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d177790b-fe2d-469e-9dff-c653997fe096-metrics-certs\") pod \"network-metrics-daemon-h9jx7\" (UID: \"d177790b-fe2d-469e-9dff-c653997fe096\") " pod="openshift-multus/network-metrics-daemon-h9jx7" Jul 10 17:11:10.261002 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.260993 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-var-lib-openvswitch\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.261002 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261011 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-host-cni-bin\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.261298 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261069 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-host-cni-bin\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.261298 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261084 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-var-lib-openvswitch\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.261298 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:10.261145 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:11:10.261298 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261145 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c7288bbc-320f-4eb8-ac48-d28666cebcdf-iptables-alerter-script\") pod \"iptables-alerter-rbwdx\" (UID: \"c7288bbc-320f-4eb8-ac48-d28666cebcdf\") " pod="openshift-network-operator/iptables-alerter-rbwdx" Jul 10 17:11:10.261298 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:10.261221 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d177790b-fe2d-469e-9dff-c653997fe096-metrics-certs podName:d177790b-fe2d-469e-9dff-c653997fe096 nodeName:}" failed. No retries permitted until 2026-07-10 17:11:10.761199311 +0000 UTC m=+3.088896848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d177790b-fe2d-469e-9dff-c653997fe096-metrics-certs") pod "network-metrics-daemon-h9jx7" (UID: "d177790b-fe2d-469e-9dff-c653997fe096") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:11:10.261298 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261241 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-node-log\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.261298 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261274 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/73049e7b-08d4-4772-8dfc-531b8449d42b-etc-modprobe-d\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.261298 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261299 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/73049e7b-08d4-4772-8dfc-531b8449d42b-sys\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.261714 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261322 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73049e7b-08d4-4772-8dfc-531b8449d42b-var-lib-kubelet\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.261714 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261343 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-host-cni-netd\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.261714 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261366 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/73049e7b-08d4-4772-8dfc-531b8449d42b-etc-sysconfig\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.261714 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261413 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.261714 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261435 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/73049e7b-08d4-4772-8dfc-531b8449d42b-sys\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.261714 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261442 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/73049e7b-08d4-4772-8dfc-531b8449d42b-etc-sysctl-d\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.261714 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261469 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lck8\" (UniqueName: \"kubernetes.io/projected/c7288bbc-320f-4eb8-ac48-d28666cebcdf-kube-api-access-9lck8\") pod \"iptables-alerter-rbwdx\" (UID: \"c7288bbc-320f-4eb8-ac48-d28666cebcdf\") " pod="openshift-network-operator/iptables-alerter-rbwdx" Jul 10 17:11:10.261714 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261489 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e41baaf7-4171-4df1-8bdd-d0697a61fdce-env-overrides\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.261714 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261496 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.261714 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261520 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ea2e6b67-bf39-4196-a657-7859b8b73199-kubelet-config\") pod \"global-pull-secret-syncer-smz4h\" (UID: \"ea2e6b67-bf39-4196-a657-7859b8b73199\") " pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:11:10.261714 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261541 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/73049e7b-08d4-4772-8dfc-531b8449d42b-etc-sysctl-conf\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.261714 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261555 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/73049e7b-08d4-4772-8dfc-531b8449d42b-etc-sysconfig\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.261714 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261561 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/73049e7b-08d4-4772-8dfc-531b8449d42b-etc-modprobe-d\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.261714 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261564 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/73049e7b-08d4-4772-8dfc-531b8449d42b-lib-modules\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.261714 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261607 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-host-cni-netd\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.261714 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261637 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-node-log\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.261714 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261648 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/73049e7b-08d4-4772-8dfc-531b8449d42b-etc-sysctl-d\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.262539 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261656 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/73049e7b-08d4-4772-8dfc-531b8449d42b-lib-modules\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.262539 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261680 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ea2e6b67-bf39-4196-a657-7859b8b73199-kubelet-config\") pod \"global-pull-secret-syncer-smz4h\" (UID: \"ea2e6b67-bf39-4196-a657-7859b8b73199\") " pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:11:10.262539 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261698 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/73049e7b-08d4-4772-8dfc-531b8449d42b-host\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.262539 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261698 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73049e7b-08d4-4772-8dfc-531b8449d42b-var-lib-kubelet\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.262539 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261721 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73049e7b-08d4-4772-8dfc-531b8449d42b-tmp\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.262539 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261738 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5vw9\" (UniqueName: \"kubernetes.io/projected/73049e7b-08d4-4772-8dfc-531b8449d42b-kube-api-access-m5vw9\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.262539 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261752 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ea2e6b67-bf39-4196-a657-7859b8b73199-dbus\") pod \"global-pull-secret-syncer-smz4h\" (UID: \"ea2e6b67-bf39-4196-a657-7859b8b73199\") " pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:11:10.262539 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261759 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c7288bbc-320f-4eb8-ac48-d28666cebcdf-iptables-alerter-script\") pod \"iptables-alerter-rbwdx\" (UID: \"c7288bbc-320f-4eb8-ac48-d28666cebcdf\") " pod="openshift-network-operator/iptables-alerter-rbwdx" Jul 10 17:11:10.262539 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261770 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73049e7b-08d4-4772-8dfc-531b8449d42b-etc-kubernetes\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.262539 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261786 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-run-openvswitch\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.262539 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261808 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-run-ovn\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.262539 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261817 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/73049e7b-08d4-4772-8dfc-531b8449d42b-etc-sysctl-conf\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.262539 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261850 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/73049e7b-08d4-4772-8dfc-531b8449d42b-host\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.262539 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261823 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e41baaf7-4171-4df1-8bdd-d0697a61fdce-ovnkube-config\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.262539 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261938 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea2e6b67-bf39-4196-a657-7859b8b73199-original-pull-secret\") pod \"global-pull-secret-syncer-smz4h\" (UID: \"ea2e6b67-bf39-4196-a657-7859b8b73199\") " pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:11:10.262539 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261968 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-host-slash\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.262539 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.261993 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e41baaf7-4171-4df1-8bdd-d0697a61fdce-ovn-node-metrics-cert\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.262539 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.262022 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/73049e7b-08d4-4772-8dfc-531b8449d42b-run\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.263254 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.262025 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-run-openvswitch\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.263254 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.262036 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ea2e6b67-bf39-4196-a657-7859b8b73199-dbus\") pod \"global-pull-secret-syncer-smz4h\" (UID: \"ea2e6b67-bf39-4196-a657-7859b8b73199\") " pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:11:10.263254 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.262083 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-host-run-netns\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.263254 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.262136 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e41baaf7-4171-4df1-8bdd-d0697a61fdce-env-overrides\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.263254 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.262044 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-host-run-netns\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.263254 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:10.262169 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 17:11:10.263254 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.262187 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6tj5\" (UniqueName: \"kubernetes.io/projected/e41baaf7-4171-4df1-8bdd-d0697a61fdce-kube-api-access-g6tj5\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.263254 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.262197 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-run-ovn\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.263254 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:10.262230 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea2e6b67-bf39-4196-a657-7859b8b73199-original-pull-secret podName:ea2e6b67-bf39-4196-a657-7859b8b73199 nodeName:}" failed. No retries permitted until 2026-07-10 17:11:10.762213424 +0000 UTC m=+3.089910957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea2e6b67-bf39-4196-a657-7859b8b73199-original-pull-secret") pod "global-pull-secret-syncer-smz4h" (UID: "ea2e6b67-bf39-4196-a657-7859b8b73199") : object "kube-system"/"original-pull-secret" not registered Jul 10 17:11:10.263254 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.262255 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/73049e7b-08d4-4772-8dfc-531b8449d42b-run\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.263254 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.262245 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-host-slash\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.263254 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.262270 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73049e7b-08d4-4772-8dfc-531b8449d42b-etc-kubernetes\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.263254 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.262280 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-run-systemd\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.263254 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.262312 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e41baaf7-4171-4df1-8bdd-d0697a61fdce-ovnkube-script-lib\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.263254 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.262315 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-run-systemd\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.263254 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.262348 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/73049e7b-08d4-4772-8dfc-531b8449d42b-etc-systemd\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.263254 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.262376 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7288bbc-320f-4eb8-ac48-d28666cebcdf-host-slash\") pod \"iptables-alerter-rbwdx\" (UID: \"c7288bbc-320f-4eb8-ac48-d28666cebcdf\") " pod="openshift-network-operator/iptables-alerter-rbwdx" Jul 10 17:11:10.264094 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.262419 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-host-kubelet\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.264094 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.262440 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/73049e7b-08d4-4772-8dfc-531b8449d42b-etc-systemd\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.264094 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.262473 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-host-kubelet\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.264094 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.262452 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-systemd-units\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.264094 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.262506 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-systemd-units\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.264094 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.262486 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7288bbc-320f-4eb8-ac48-d28666cebcdf-host-slash\") pod \"iptables-alerter-rbwdx\" (UID: \"c7288bbc-320f-4eb8-ac48-d28666cebcdf\") " pod="openshift-network-operator/iptables-alerter-rbwdx" Jul 10 17:11:10.264094 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.262644 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-etc-openvswitch\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.264094 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.262667 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-host-run-ovn-kubernetes\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.264094 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.262675 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-etc-openvswitch\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.264094 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.262715 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/73049e7b-08d4-4772-8dfc-531b8449d42b-etc-tuned\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.264094 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.262745 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5fln\" (UniqueName: \"kubernetes.io/projected/d177790b-fe2d-469e-9dff-c653997fe096-kube-api-access-g5fln\") pod \"network-metrics-daemon-h9jx7\" (UID: \"d177790b-fe2d-469e-9dff-c653997fe096\") " pod="openshift-multus/network-metrics-daemon-h9jx7" Jul 10 17:11:10.264094 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.262768 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-log-socket\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.264094 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.262715 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-host-run-ovn-kubernetes\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.264094 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.262835 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e41baaf7-4171-4df1-8bdd-d0697a61fdce-log-socket\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.264094 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.262985 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e41baaf7-4171-4df1-8bdd-d0697a61fdce-ovnkube-script-lib\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.264094 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.263010 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e41baaf7-4171-4df1-8bdd-d0697a61fdce-ovnkube-config\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.264698 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.264467 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73049e7b-08d4-4772-8dfc-531b8449d42b-tmp\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.265031 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.265012 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e41baaf7-4171-4df1-8bdd-d0697a61fdce-ovn-node-metrics-cert\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.265098 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.265035 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/73049e7b-08d4-4772-8dfc-531b8449d42b-etc-tuned\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.273888 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.273813 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5fln\" (UniqueName: \"kubernetes.io/projected/d177790b-fe2d-469e-9dff-c653997fe096-kube-api-access-g5fln\") pod \"network-metrics-daemon-h9jx7\" (UID: \"d177790b-fe2d-469e-9dff-c653997fe096\") " pod="openshift-multus/network-metrics-daemon-h9jx7" Jul 10 17:11:10.274194 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.274168 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6tj5\" (UniqueName: \"kubernetes.io/projected/e41baaf7-4171-4df1-8bdd-d0697a61fdce-kube-api-access-g6tj5\") pod \"ovnkube-node-js69w\" (UID: \"e41baaf7-4171-4df1-8bdd-d0697a61fdce\") " pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.274595 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.274574 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lck8\" (UniqueName: \"kubernetes.io/projected/c7288bbc-320f-4eb8-ac48-d28666cebcdf-kube-api-access-9lck8\") pod \"iptables-alerter-rbwdx\" (UID: \"c7288bbc-320f-4eb8-ac48-d28666cebcdf\") " pod="openshift-network-operator/iptables-alerter-rbwdx" Jul 10 17:11:10.274688 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.274574 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5vw9\" (UniqueName: \"kubernetes.io/projected/73049e7b-08d4-4772-8dfc-531b8449d42b-kube-api-access-m5vw9\") pod \"tuned-9f4lz\" (UID: \"73049e7b-08d4-4772-8dfc-531b8449d42b\") " pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.344100 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.344059 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vk6rc" Jul 10 17:11:10.360109 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.359878 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zmbx" Jul 10 17:11:10.368085 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.368058 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lql7p" Jul 10 17:11:10.373081 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.373057 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bq9tx" Jul 10 17:11:10.381854 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.381828 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5krz5" Jul 10 17:11:10.387839 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.387818 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-52frq" Jul 10 17:11:10.396704 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.396682 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rbwdx" Jul 10 17:11:10.405590 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.405556 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:10.409697 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.409678 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" Jul 10 17:11:10.538586 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.538498 2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 17:11:10.764850 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.764817 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsl5d\" (UniqueName: \"kubernetes.io/projected/7000764e-d442-4c90-aa00-97b83a65e4b6-kube-api-access-rsl5d\") pod \"network-check-target-7mnkc\" (UID: \"7000764e-d442-4c90-aa00-97b83a65e4b6\") " pod="openshift-network-diagnostics/network-check-target-7mnkc" Jul 10 17:11:10.764850 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.764860 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea2e6b67-bf39-4196-a657-7859b8b73199-original-pull-secret\") pod \"global-pull-secret-syncer-smz4h\" (UID: \"ea2e6b67-bf39-4196-a657-7859b8b73199\") " pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:11:10.765095 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:10.764905 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d177790b-fe2d-469e-9dff-c653997fe096-metrics-certs\") pod \"network-metrics-daemon-h9jx7\" (UID: \"d177790b-fe2d-469e-9dff-c653997fe096\") " pod="openshift-multus/network-metrics-daemon-h9jx7" Jul 10 17:11:10.765095 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:10.765033 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 17:11:10.765188 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:10.765102 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea2e6b67-bf39-4196-a657-7859b8b73199-original-pull-secret podName:ea2e6b67-bf39-4196-a657-7859b8b73199 nodeName:}" failed. No retries permitted until 2026-07-10 17:11:11.765087161 +0000 UTC m=+4.092784679 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea2e6b67-bf39-4196-a657-7859b8b73199-original-pull-secret") pod "global-pull-secret-syncer-smz4h" (UID: "ea2e6b67-bf39-4196-a657-7859b8b73199") : object "kube-system"/"original-pull-secret" not registered Jul 10 17:11:10.765188 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:10.765035 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 17:11:10.765188 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:10.765170 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 17:11:10.765188 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:10.765185 2583 projected.go:196] Error preparing data for projected volume kube-api-access-rsl5d for pod openshift-network-diagnostics/network-check-target-7mnkc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:11:10.765353 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:10.765037 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:11:10.765353 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:10.765239 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7000764e-d442-4c90-aa00-97b83a65e4b6-kube-api-access-rsl5d podName:7000764e-d442-4c90-aa00-97b83a65e4b6 nodeName:}" failed. No retries permitted until 2026-07-10 17:11:11.765222479 +0000 UTC m=+4.092920015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rsl5d" (UniqueName: "kubernetes.io/projected/7000764e-d442-4c90-aa00-97b83a65e4b6-kube-api-access-rsl5d") pod "network-check-target-7mnkc" (UID: "7000764e-d442-4c90-aa00-97b83a65e4b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:11:10.765353 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:10.765274 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d177790b-fe2d-469e-9dff-c653997fe096-metrics-certs podName:d177790b-fe2d-469e-9dff-c653997fe096 nodeName:}" failed. No retries permitted until 2026-07-10 17:11:11.765259069 +0000 UTC m=+4.092956586 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d177790b-fe2d-469e-9dff-c653997fe096-metrics-certs") pod "network-metrics-daemon-h9jx7" (UID: "d177790b-fe2d-469e-9dff-c653997fe096") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:11:11.111794 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:11.111752 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 17:06:10 +0000 UTC" deadline="2028-04-09 22:56:20.890862028 +0000 UTC" Jul 10 17:11:11.111794 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:11.111788 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15341h45m9.779078568s" Jul 10 17:11:11.269054 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:11.268941 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3756f101_49b1_4754_ad4a_13c572eec69c.slice/crio-75910acba75e12440c095c654853eba56676272486a8d0d424cd41b8b390f4b1 WatchSource:0}: Error finding container 75910acba75e12440c095c654853eba56676272486a8d0d424cd41b8b390f4b1: Status 404 returned error can't find the container with id 75910acba75e12440c095c654853eba56676272486a8d0d424cd41b8b390f4b1 Jul 10 17:11:11.771795 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:11.771539 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d177790b-fe2d-469e-9dff-c653997fe096-metrics-certs\") pod \"network-metrics-daemon-h9jx7\" (UID: \"d177790b-fe2d-469e-9dff-c653997fe096\") " pod="openshift-multus/network-metrics-daemon-h9jx7" Jul 10 17:11:11.771950 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:11.771802 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsl5d\" (UniqueName: \"kubernetes.io/projected/7000764e-d442-4c90-aa00-97b83a65e4b6-kube-api-access-rsl5d\") pod \"network-check-target-7mnkc\" (UID: \"7000764e-d442-4c90-aa00-97b83a65e4b6\") " pod="openshift-network-diagnostics/network-check-target-7mnkc" Jul 10 17:11:11.771950 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:11.771826 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea2e6b67-bf39-4196-a657-7859b8b73199-original-pull-secret\") pod \"global-pull-secret-syncer-smz4h\" (UID: \"ea2e6b67-bf39-4196-a657-7859b8b73199\") " pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:11:11.771950 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:11.771704 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:11:11.772105 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:11.771972 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d177790b-fe2d-469e-9dff-c653997fe096-metrics-certs podName:d177790b-fe2d-469e-9dff-c653997fe096 nodeName:}" failed. No retries permitted until 2026-07-10 17:11:13.771957127 +0000 UTC m=+6.099654643 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d177790b-fe2d-469e-9dff-c653997fe096-metrics-certs") pod "network-metrics-daemon-h9jx7" (UID: "d177790b-fe2d-469e-9dff-c653997fe096") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:11:11.772105 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:11.772039 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 17:11:11.772105 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:11.772050 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 17:11:11.772105 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:11.772058 2583 projected.go:196] Error preparing data for projected volume kube-api-access-rsl5d for pod openshift-network-diagnostics/network-check-target-7mnkc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:11:11.772105 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:11.772081 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7000764e-d442-4c90-aa00-97b83a65e4b6-kube-api-access-rsl5d podName:7000764e-d442-4c90-aa00-97b83a65e4b6 nodeName:}" failed. No retries permitted until 2026-07-10 17:11:13.772073463 +0000 UTC m=+6.099770981 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rsl5d" (UniqueName: "kubernetes.io/projected/7000764e-d442-4c90-aa00-97b83a65e4b6-kube-api-access-rsl5d") pod "network-check-target-7mnkc" (UID: "7000764e-d442-4c90-aa00-97b83a65e4b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:11:11.772105 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:11.771913 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 17:11:11.772105 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:11.772104 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea2e6b67-bf39-4196-a657-7859b8b73199-original-pull-secret podName:ea2e6b67-bf39-4196-a657-7859b8b73199 nodeName:}" failed. No retries permitted until 2026-07-10 17:11:13.772098376 +0000 UTC m=+6.099795892 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea2e6b67-bf39-4196-a657-7859b8b73199-original-pull-secret") pod "global-pull-secret-syncer-smz4h" (UID: "ea2e6b67-bf39-4196-a657-7859b8b73199") : object "kube-system"/"original-pull-secret" not registered Jul 10 17:11:12.112400 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:12.112273 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 17:06:10 +0000 UTC" deadline="2027-12-05 18:58:58.385363607 +0000 UTC" Jul 10 17:11:12.112400 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:12.112332 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12313h47m46.27303598s" Jul 10 17:11:12.202978 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:12.202945 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:11:12.203133 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:12.203078 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smz4h" podUID="ea2e6b67-bf39-4196-a657-7859b8b73199" Jul 10 17:11:12.203491 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:12.203245 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7mnkc" Jul 10 17:11:12.203491 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:12.203326 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7mnkc" podUID="7000764e-d442-4c90-aa00-97b83a65e4b6" Jul 10 17:11:12.203491 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:12.203351 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9jx7" Jul 10 17:11:12.203491 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:12.203459 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9jx7" podUID="d177790b-fe2d-469e-9dff-c653997fe096" Jul 10 17:11:12.236807 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:12.236761 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-65.ec2.internal" event={"ID":"84e6cfddc1f961a111c9509968c60461","Type":"ContainerStarted","Data":"2aba61c15d0a76c79c98480d8d7a9adb31cee73ebcdb70333d260fb23c30d846"} Jul 10 17:11:12.243079 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:12.243046 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" event={"ID":"73049e7b-08d4-4772-8dfc-531b8449d42b","Type":"ContainerStarted","Data":"6adbba39fac1b125e0a8278b818abaf317f8df8ba03ca11f1030a245796aa72b"} Jul 10 17:11:12.250907 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:12.250874 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vk6rc" event={"ID":"1e13ba88-7d4c-42b8-a992-95b3c447b222","Type":"ContainerStarted","Data":"848405f626ebd597d620e0692435f2086fec0da8bf152b0163081d0ca12d35bd"} Jul 10 17:11:12.256196 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:12.255511 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-52frq" event={"ID":"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7","Type":"ContainerStarted","Data":"757d27ff5dd5b16d223237dd68f1df6976dad2682299adc79aa42d50f3615167"} Jul 10 17:11:12.271218 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:12.271172 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rbwdx" event={"ID":"c7288bbc-320f-4eb8-ac48-d28666cebcdf","Type":"ContainerStarted","Data":"6f99327d02db66dce5c51f1a41f3f5175bfaf22e374e5555dcbfd6ddc8060edf"} Jul 10 17:11:12.274900 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:12.274787 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-js69w" event={"ID":"e41baaf7-4171-4df1-8bdd-d0697a61fdce","Type":"ContainerStarted","Data":"0158eef9be569a4780bcd423010378c0014a6f3449ea54dbdb378464d6f4004f"} Jul 10 17:11:12.278749 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:12.278667 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zmbx" event={"ID":"0bd5b01a-f45d-41bf-aa6e-3fb6151cd008","Type":"ContainerStarted","Data":"fb249d59722deb8307d8848600f5ca07dc1aa56bd90bd0faf89f436a596dd3d2"} Jul 10 17:11:12.283632 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:12.283570 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5krz5" event={"ID":"df9374c1-e474-4ceb-b6d6-a8879a52e376","Type":"ContainerStarted","Data":"78dca9fefed297264bf1ad192072af20b4b3c6f899e393efba8a7cff827ac232"} Jul 10 17:11:12.287703 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:12.287665 2583 generic.go:356] "Generic (PLEG): container finished" podID="9437cafed03781a61d9e1ba47f46277b" containerID="7330178f6390e49fdc2153716fe97e00ea0248874c8677a08c080432ceb4a023" exitCode=0 Jul 10 17:11:12.287827 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:12.287757 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-65.ec2.internal" event={"ID":"9437cafed03781a61d9e1ba47f46277b","Type":"ContainerDied","Data":"7330178f6390e49fdc2153716fe97e00ea0248874c8677a08c080432ceb4a023"} Jul 10 17:11:12.302843 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:12.302810 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bq9tx" event={"ID":"9369f652-8e06-4d0a-b00e-07ad43e2bec1","Type":"ContainerStarted","Data":"bf11893941d4c240378dc6011140588a714667eea821fdfba6ef80d47b53b9c7"} Jul 10 17:11:12.307184 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:12.307152 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lql7p" event={"ID":"3756f101-49b1-4754-ad4a-13c572eec69c","Type":"ContainerStarted","Data":"75910acba75e12440c095c654853eba56676272486a8d0d424cd41b8b390f4b1"} Jul 10 17:11:12.307913 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:12.307865 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-65.ec2.internal" podStartSLOduration=3.307846246 podStartE2EDuration="3.307846246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:11:12.255671745 +0000 UTC m=+4.583369285" watchObservedRunningTime="2026-07-10 17:11:12.307846246 +0000 UTC m=+4.635543786" Jul 10 17:11:13.341667 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:13.341627 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-65.ec2.internal" event={"ID":"9437cafed03781a61d9e1ba47f46277b","Type":"ContainerStarted","Data":"78af12ea5d23574e882a64ea8b56d10ecc6010a4bc42b0c1d60c1cb27e44119b"} Jul 10 17:11:13.358117 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:13.358056 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-65.ec2.internal" podStartSLOduration=4.35803499 podStartE2EDuration="4.35803499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:11:13.357743455 +0000 UTC m=+5.685440996" watchObservedRunningTime="2026-07-10 17:11:13.35803499 +0000 UTC m=+5.685732530" Jul 10 17:11:13.787405 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:13.787295 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d177790b-fe2d-469e-9dff-c653997fe096-metrics-certs\") pod \"network-metrics-daemon-h9jx7\" (UID: \"d177790b-fe2d-469e-9dff-c653997fe096\") " pod="openshift-multus/network-metrics-daemon-h9jx7" Jul 10 17:11:13.787405 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:13.787378 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsl5d\" (UniqueName: \"kubernetes.io/projected/7000764e-d442-4c90-aa00-97b83a65e4b6-kube-api-access-rsl5d\") pod \"network-check-target-7mnkc\" (UID: \"7000764e-d442-4c90-aa00-97b83a65e4b6\") " pod="openshift-network-diagnostics/network-check-target-7mnkc" Jul 10 17:11:13.787663 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:13.787421 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea2e6b67-bf39-4196-a657-7859b8b73199-original-pull-secret\") pod \"global-pull-secret-syncer-smz4h\" (UID: \"ea2e6b67-bf39-4196-a657-7859b8b73199\") " pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:11:13.787663 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:13.787558 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 17:11:13.787663 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:13.787620 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea2e6b67-bf39-4196-a657-7859b8b73199-original-pull-secret podName:ea2e6b67-bf39-4196-a657-7859b8b73199 nodeName:}" failed. No retries permitted until 2026-07-10 17:11:17.78760201 +0000 UTC m=+10.115299545 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea2e6b67-bf39-4196-a657-7859b8b73199-original-pull-secret") pod "global-pull-secret-syncer-smz4h" (UID: "ea2e6b67-bf39-4196-a657-7859b8b73199") : object "kube-system"/"original-pull-secret" not registered Jul 10 17:11:13.787810 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:13.787697 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:11:13.787810 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:13.787729 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d177790b-fe2d-469e-9dff-c653997fe096-metrics-certs podName:d177790b-fe2d-469e-9dff-c653997fe096 nodeName:}" failed. No retries permitted until 2026-07-10 17:11:17.787718546 +0000 UTC m=+10.115416073 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d177790b-fe2d-469e-9dff-c653997fe096-metrics-certs") pod "network-metrics-daemon-h9jx7" (UID: "d177790b-fe2d-469e-9dff-c653997fe096") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:11:13.787910 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:13.787810 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 17:11:13.787910 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:13.787830 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 17:11:13.787910 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:13.787842 2583 projected.go:196] Error preparing data for projected volume kube-api-access-rsl5d for pod openshift-network-diagnostics/network-check-target-7mnkc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:11:13.787910 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:13.787881 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7000764e-d442-4c90-aa00-97b83a65e4b6-kube-api-access-rsl5d podName:7000764e-d442-4c90-aa00-97b83a65e4b6 nodeName:}" failed. No retries permitted until 2026-07-10 17:11:17.787869329 +0000 UTC m=+10.115566852 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rsl5d" (UniqueName: "kubernetes.io/projected/7000764e-d442-4c90-aa00-97b83a65e4b6-kube-api-access-rsl5d") pod "network-check-target-7mnkc" (UID: "7000764e-d442-4c90-aa00-97b83a65e4b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:11:14.202850 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:14.202760 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9jx7" Jul 10 17:11:14.203111 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:14.202903 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9jx7" podUID="d177790b-fe2d-469e-9dff-c653997fe096" Jul 10 17:11:14.203324 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:14.203302 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:11:14.203612 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:14.203579 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smz4h" podUID="ea2e6b67-bf39-4196-a657-7859b8b73199" Jul 10 17:11:14.203686 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:14.203668 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7mnkc" Jul 10 17:11:14.203802 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:14.203780 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7mnkc" podUID="7000764e-d442-4c90-aa00-97b83a65e4b6" Jul 10 17:11:16.201525 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:16.201486 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7mnkc" Jul 10 17:11:16.201950 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:16.201624 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7mnkc" podUID="7000764e-d442-4c90-aa00-97b83a65e4b6" Jul 10 17:11:16.202028 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:16.202008 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9jx7" Jul 10 17:11:16.202134 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:16.202114 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9jx7" podUID="d177790b-fe2d-469e-9dff-c653997fe096" Jul 10 17:11:16.202191 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:16.202129 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:11:16.202294 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:16.202252 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smz4h" podUID="ea2e6b67-bf39-4196-a657-7859b8b73199" Jul 10 17:11:17.825184 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:17.825146 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsl5d\" (UniqueName: \"kubernetes.io/projected/7000764e-d442-4c90-aa00-97b83a65e4b6-kube-api-access-rsl5d\") pod \"network-check-target-7mnkc\" (UID: \"7000764e-d442-4c90-aa00-97b83a65e4b6\") " pod="openshift-network-diagnostics/network-check-target-7mnkc" Jul 10 17:11:17.825184 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:17.825190 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea2e6b67-bf39-4196-a657-7859b8b73199-original-pull-secret\") pod \"global-pull-secret-syncer-smz4h\" (UID: \"ea2e6b67-bf39-4196-a657-7859b8b73199\") " pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:11:17.825954 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:17.825272 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d177790b-fe2d-469e-9dff-c653997fe096-metrics-certs\") pod \"network-metrics-daemon-h9jx7\" (UID: \"d177790b-fe2d-469e-9dff-c653997fe096\") " pod="openshift-multus/network-metrics-daemon-h9jx7" Jul 10 17:11:17.825954 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:17.825318 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 17:11:17.825954 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:17.825377 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea2e6b67-bf39-4196-a657-7859b8b73199-original-pull-secret podName:ea2e6b67-bf39-4196-a657-7859b8b73199 nodeName:}" failed. No retries permitted until 2026-07-10 17:11:25.825359243 +0000 UTC m=+18.153056759 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea2e6b67-bf39-4196-a657-7859b8b73199-original-pull-secret") pod "global-pull-secret-syncer-smz4h" (UID: "ea2e6b67-bf39-4196-a657-7859b8b73199") : object "kube-system"/"original-pull-secret" not registered Jul 10 17:11:17.825954 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:17.825449 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:11:17.825954 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:17.825467 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 17:11:17.825954 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:17.825483 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 17:11:17.825954 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:17.825497 2583 projected.go:196] Error preparing data for projected volume kube-api-access-rsl5d for pod openshift-network-diagnostics/network-check-target-7mnkc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:11:17.825954 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:17.825515 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d177790b-fe2d-469e-9dff-c653997fe096-metrics-certs podName:d177790b-fe2d-469e-9dff-c653997fe096 nodeName:}" failed. No retries permitted until 2026-07-10 17:11:25.825491429 +0000 UTC m=+18.153188952 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d177790b-fe2d-469e-9dff-c653997fe096-metrics-certs") pod "network-metrics-daemon-h9jx7" (UID: "d177790b-fe2d-469e-9dff-c653997fe096") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:11:17.825954 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:17.825541 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7000764e-d442-4c90-aa00-97b83a65e4b6-kube-api-access-rsl5d podName:7000764e-d442-4c90-aa00-97b83a65e4b6 nodeName:}" failed. No retries permitted until 2026-07-10 17:11:25.825531278 +0000 UTC m=+18.153228799 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rsl5d" (UniqueName: "kubernetes.io/projected/7000764e-d442-4c90-aa00-97b83a65e4b6-kube-api-access-rsl5d") pod "network-check-target-7mnkc" (UID: "7000764e-d442-4c90-aa00-97b83a65e4b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:11:18.201339 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:18.201263 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:11:18.201518 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:18.201376 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smz4h" podUID="ea2e6b67-bf39-4196-a657-7859b8b73199" Jul 10 17:11:18.201826 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:18.201806 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7mnkc" Jul 10 17:11:18.201918 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:18.201901 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7mnkc" podUID="7000764e-d442-4c90-aa00-97b83a65e4b6" Jul 10 17:11:18.202061 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:18.202043 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9jx7" Jul 10 17:11:18.202162 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:18.202142 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9jx7" podUID="d177790b-fe2d-469e-9dff-c653997fe096" Jul 10 17:11:20.200533 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:20.200492 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7mnkc" Jul 10 17:11:20.200533 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:20.200513 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9jx7" Jul 10 17:11:20.201048 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:20.200627 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7mnkc" podUID="7000764e-d442-4c90-aa00-97b83a65e4b6" Jul 10 17:11:20.201048 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:20.201015 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9jx7" podUID="d177790b-fe2d-469e-9dff-c653997fe096" Jul 10 17:11:20.201154 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:20.201053 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:11:20.201154 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:20.201100 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smz4h" podUID="ea2e6b67-bf39-4196-a657-7859b8b73199" Jul 10 17:11:22.200516 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:22.200484 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7mnkc" Jul 10 17:11:22.200516 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:22.200522 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:11:22.201021 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:22.200488 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9jx7" Jul 10 17:11:22.201021 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:22.200631 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7mnkc" podUID="7000764e-d442-4c90-aa00-97b83a65e4b6" Jul 10 17:11:22.201021 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:22.200725 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9jx7" podUID="d177790b-fe2d-469e-9dff-c653997fe096" Jul 10 17:11:22.201021 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:22.200826 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smz4h" podUID="ea2e6b67-bf39-4196-a657-7859b8b73199" Jul 10 17:11:24.200384 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:24.200347 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7mnkc" Jul 10 17:11:24.200384 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:24.200369 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:11:24.200891 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:24.200489 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7mnkc" podUID="7000764e-d442-4c90-aa00-97b83a65e4b6" Jul 10 17:11:24.200891 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:24.200586 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smz4h" podUID="ea2e6b67-bf39-4196-a657-7859b8b73199" Jul 10 17:11:24.200891 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:24.200650 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9jx7" Jul 10 17:11:24.200891 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:24.200749 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9jx7" podUID="d177790b-fe2d-469e-9dff-c653997fe096" Jul 10 17:11:25.888491 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:25.888461 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d177790b-fe2d-469e-9dff-c653997fe096-metrics-certs\") pod \"network-metrics-daemon-h9jx7\" (UID: \"d177790b-fe2d-469e-9dff-c653997fe096\") " pod="openshift-multus/network-metrics-daemon-h9jx7" Jul 10 17:11:25.888881 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:25.888507 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsl5d\" (UniqueName: \"kubernetes.io/projected/7000764e-d442-4c90-aa00-97b83a65e4b6-kube-api-access-rsl5d\") pod \"network-check-target-7mnkc\" (UID: \"7000764e-d442-4c90-aa00-97b83a65e4b6\") " pod="openshift-network-diagnostics/network-check-target-7mnkc" Jul 10 17:11:25.888881 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:25.888536 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea2e6b67-bf39-4196-a657-7859b8b73199-original-pull-secret\") pod \"global-pull-secret-syncer-smz4h\" (UID: \"ea2e6b67-bf39-4196-a657-7859b8b73199\") " pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:11:25.888881 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:25.888606 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:11:25.888881 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:25.888617 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 17:11:25.888881 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:25.888638 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 17:11:25.888881 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:25.888648 2583 projected.go:196] Error preparing data for projected volume kube-api-access-rsl5d for pod openshift-network-diagnostics/network-check-target-7mnkc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:11:25.888881 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:25.888652 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 17:11:25.888881 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:25.888656 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d177790b-fe2d-469e-9dff-c653997fe096-metrics-certs podName:d177790b-fe2d-469e-9dff-c653997fe096 nodeName:}" failed. No retries permitted until 2026-07-10 17:11:41.888642809 +0000 UTC m=+34.216340326 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d177790b-fe2d-469e-9dff-c653997fe096-metrics-certs") pod "network-metrics-daemon-h9jx7" (UID: "d177790b-fe2d-469e-9dff-c653997fe096") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:11:25.888881 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:25.888723 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7000764e-d442-4c90-aa00-97b83a65e4b6-kube-api-access-rsl5d podName:7000764e-d442-4c90-aa00-97b83a65e4b6 nodeName:}" failed. No retries permitted until 2026-07-10 17:11:41.888703176 +0000 UTC m=+34.216400697 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rsl5d" (UniqueName: "kubernetes.io/projected/7000764e-d442-4c90-aa00-97b83a65e4b6-kube-api-access-rsl5d") pod "network-check-target-7mnkc" (UID: "7000764e-d442-4c90-aa00-97b83a65e4b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:11:25.888881 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:25.888741 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea2e6b67-bf39-4196-a657-7859b8b73199-original-pull-secret podName:ea2e6b67-bf39-4196-a657-7859b8b73199 nodeName:}" failed. No retries permitted until 2026-07-10 17:11:41.888730475 +0000 UTC m=+34.216427994 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea2e6b67-bf39-4196-a657-7859b8b73199-original-pull-secret") pod "global-pull-secret-syncer-smz4h" (UID: "ea2e6b67-bf39-4196-a657-7859b8b73199") : object "kube-system"/"original-pull-secret" not registered Jul 10 17:11:26.200537 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:26.200455 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9jx7" Jul 10 17:11:26.200537 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:26.200480 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:11:26.200758 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:26.200606 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9jx7" podUID="d177790b-fe2d-469e-9dff-c653997fe096" Jul 10 17:11:26.200758 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:26.200634 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7mnkc" Jul 10 17:11:26.200859 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:26.200758 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smz4h" podUID="ea2e6b67-bf39-4196-a657-7859b8b73199" Jul 10 17:11:26.200859 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:26.200836 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7mnkc" podUID="7000764e-d442-4c90-aa00-97b83a65e4b6" Jul 10 17:11:28.201345 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:28.201305 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7mnkc" Jul 10 17:11:28.201845 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:28.201435 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9jx7" Jul 10 17:11:28.201845 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:28.201454 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:11:28.201845 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:28.201517 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7mnkc" podUID="7000764e-d442-4c90-aa00-97b83a65e4b6" Jul 10 17:11:28.201845 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:28.201548 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9jx7" podUID="d177790b-fe2d-469e-9dff-c653997fe096" Jul 10 17:11:28.201845 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:28.201622 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smz4h" podUID="ea2e6b67-bf39-4196-a657-7859b8b73199" Jul 10 17:11:30.201506 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:30.200755 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7mnkc" Jul 10 17:11:30.201506 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:30.200869 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7mnkc" podUID="7000764e-d442-4c90-aa00-97b83a65e4b6" Jul 10 17:11:30.201506 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:30.201234 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9jx7" Jul 10 17:11:30.201506 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:30.201337 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9jx7" podUID="d177790b-fe2d-469e-9dff-c653997fe096" Jul 10 17:11:30.202018 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:30.201892 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:11:30.202018 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:30.201983 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smz4h" podUID="ea2e6b67-bf39-4196-a657-7859b8b73199" Jul 10 17:11:30.374600 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:30.374354 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" event={"ID":"73049e7b-08d4-4772-8dfc-531b8449d42b","Type":"ContainerStarted","Data":"b5fbc1e8f29ff4c23814f19e0c73286943a95ea313fc2fae798f83f3dd6aec5a"} Jul 10 17:11:30.375953 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:30.375924 2583 generic.go:356] "Generic (PLEG): container finished" podID="1e13ba88-7d4c-42b8-a992-95b3c447b222" containerID="29ee6695edcad04e448ef6d7f1da4666b086051008906bea870a8938378c1097" exitCode=0 Jul 10 17:11:30.376058 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:30.376037 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vk6rc" event={"ID":"1e13ba88-7d4c-42b8-a992-95b3c447b222","Type":"ContainerDied","Data":"29ee6695edcad04e448ef6d7f1da4666b086051008906bea870a8938378c1097"} Jul 10 17:11:30.377556 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:30.377532 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-52frq" event={"ID":"7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7","Type":"ContainerStarted","Data":"6d05cf4c957c30e3d656460a5ce3a8a59cde12cb3cfee8ef0e231be4e9cb34be"} Jul 10 17:11:30.379358 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:30.379334 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-js69w" event={"ID":"e41baaf7-4171-4df1-8bdd-d0697a61fdce","Type":"ContainerStarted","Data":"77b75056ee30f2afa5ba30a19fc1a9cb680885579c95123ddf108003f53f68d6"} Jul 10 17:11:30.380718 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:30.380693 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zmbx" event={"ID":"0bd5b01a-f45d-41bf-aa6e-3fb6151cd008","Type":"ContainerStarted","Data":"985b344c2bb93a1377c06b4a2c3d8f2bfa45f323a20c4ca6d14402ec8190a5bd"} Jul 10 17:11:30.382162 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:30.382140 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5krz5" event={"ID":"df9374c1-e474-4ceb-b6d6-a8879a52e376","Type":"ContainerStarted","Data":"0e96bc2bd10d7971543e66cb01a2187788a41f8ebac29ebb627df9a673257847"} Jul 10 17:11:30.383895 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:30.383870 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bq9tx" event={"ID":"9369f652-8e06-4d0a-b00e-07ad43e2bec1","Type":"ContainerStarted","Data":"4399e497e02a3449ba3da6b60c7d050beb7fb20bb0383b72d6f5c09bd91a64ef"} Jul 10 17:11:30.385205 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:30.385187 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lql7p" event={"ID":"3756f101-49b1-4754-ad4a-13c572eec69c","Type":"ContainerStarted","Data":"e4daf89872fc9e39aa3ef993a92f060874a508203b058cc5ef53d5e86b31eb71"} Jul 10 17:11:30.396125 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:30.395985 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9f4lz" podStartSLOduration=3.586018034 podStartE2EDuration="22.395967106s" totalImagesPullingTime="18.809949072s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:11:30.39451399 +0000 UTC m=+22.722211533" watchObservedRunningTime="2026-07-10 17:11:30.395967106 +0000 UTC m=+22.723664648" Jul 10 17:11:30.409597 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:30.409556 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lql7p" podStartSLOduration=7.893836986 podStartE2EDuration="22.409538583s" totalImagesPullingTime="14.515701597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:11:30.409525289 +0000 UTC m=+22.737222812" watchObservedRunningTime="2026-07-10 17:11:30.409538583 +0000 UTC m=+22.737236123" Jul 10 17:11:30.425877 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:30.425839 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5krz5" podStartSLOduration=3.607249496 podStartE2EDuration="22.425824853s" totalImagesPullingTime="18.818575357s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:11:30.425441791 +0000 UTC m=+22.753139330" watchObservedRunningTime="2026-07-10 17:11:30.425824853 +0000 UTC m=+22.753522392" Jul 10 17:11:30.473062 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:30.473026 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-52frq" podStartSLOduration=3.600619171 podStartE2EDuration="22.473010847s" totalImagesPullingTime="18.872391676s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:11:30.446469754 +0000 UTC m=+22.774167293" watchObservedRunningTime="2026-07-10 17:11:30.473010847 +0000 UTC m=+22.800708403" Jul 10 17:11:30.491309 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:30.491269 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bq9tx" podStartSLOduration=7.985412431 podStartE2EDuration="22.491253143s" totalImagesPullingTime="14.505840712s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:11:30.49113716 +0000 UTC m=+22.818834699" watchObservedRunningTime="2026-07-10 17:11:30.491253143 +0000 UTC m=+22.818950682" Jul 10 17:11:31.389867 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:31.389669 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js69w_e41baaf7-4171-4df1-8bdd-d0697a61fdce/ovn-acl-logging/0.log" Jul 10 17:11:31.390652 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:31.390210 2583 generic.go:356] "Generic (PLEG): container finished" podID="e41baaf7-4171-4df1-8bdd-d0697a61fdce" containerID="f0778bb4cd405d5a8755a029cfe3c1878721c73ad25c8cacad024b5384c5ed13" exitCode=1 Jul 10 17:11:31.390652 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:31.390328 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-js69w" event={"ID":"e41baaf7-4171-4df1-8bdd-d0697a61fdce","Type":"ContainerStarted","Data":"7e02a720c9e7b382c5305ff2f05b4ae4c59a445ed99fd909dbed6d126c450d85"} Jul 10 17:11:31.390652 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:31.390491 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-js69w" event={"ID":"e41baaf7-4171-4df1-8bdd-d0697a61fdce","Type":"ContainerStarted","Data":"c95181b9d35881d3b1594194a5d27bd95957818447da8fc49bd3b1d2be641c90"} Jul 10 17:11:31.390652 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:31.390552 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-js69w" event={"ID":"e41baaf7-4171-4df1-8bdd-d0697a61fdce","Type":"ContainerStarted","Data":"a05ec3fec279cab77b0b3f53eda8d31c872b9ceadd018e0272d09c294bd040f8"} Jul 10 17:11:31.390652 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:31.390602 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-js69w" event={"ID":"e41baaf7-4171-4df1-8bdd-d0697a61fdce","Type":"ContainerStarted","Data":"d0d23a445d1eb8d4abda0de1b573dc35625d31c68673fb4c1945575f14832314"} Jul 10 17:11:31.390652 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:31.390646 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-js69w" event={"ID":"e41baaf7-4171-4df1-8bdd-d0697a61fdce","Type":"ContainerDied","Data":"f0778bb4cd405d5a8755a029cfe3c1878721c73ad25c8cacad024b5384c5ed13"} Jul 10 17:11:32.043801 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:32.043771 2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 17:11:32.127500 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:32.127357 2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T17:11:32.043794437Z","UUID":"6a574a1b-8c51-4f29-9c09-29b3460c2f02","Handler":null,"Name":"","Endpoint":""} Jul 10 17:11:32.129067 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:32.129042 2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 17:11:32.129186 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:32.129086 2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 17:11:32.200341 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:32.200269 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7mnkc" Jul 10 17:11:32.200341 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:32.200284 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9jx7" Jul 10 17:11:32.200584 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:32.200409 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:11:32.200584 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:32.200412 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7mnkc" podUID="7000764e-d442-4c90-aa00-97b83a65e4b6" Jul 10 17:11:32.200584 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:32.200510 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9jx7" podUID="d177790b-fe2d-469e-9dff-c653997fe096" Jul 10 17:11:32.200707 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:32.200602 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smz4h" podUID="ea2e6b67-bf39-4196-a657-7859b8b73199" Jul 10 17:11:32.394376 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:32.394337 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rbwdx" event={"ID":"c7288bbc-320f-4eb8-ac48-d28666cebcdf","Type":"ContainerStarted","Data":"c5c65031062ac23c02ae6148162e954e8690f2e997c5556b0fa1d0e893b5dadd"} Jul 10 17:11:32.396249 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:32.396228 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zmbx" event={"ID":"0bd5b01a-f45d-41bf-aa6e-3fb6151cd008","Type":"ContainerStarted","Data":"ea09150af4abf28b302276dca3f4702bb9b498b5432b59726918fbfab3a636fd"} Jul 10 17:11:32.581940 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:32.581864 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lql7p_3756f101-49b1-4754-ad4a-13c572eec69c/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:32.707818 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:32.707790 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lql7p_3756f101-49b1-4754-ad4a-13c572eec69c/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:32.810294 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:32.810267 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lql7p_3756f101-49b1-4754-ad4a-13c572eec69c/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:33.400993 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:33.400971 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js69w_e41baaf7-4171-4df1-8bdd-d0697a61fdce/ovn-acl-logging/0.log" Jul 10 17:11:33.401430 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:33.401358 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-js69w" event={"ID":"e41baaf7-4171-4df1-8bdd-d0697a61fdce","Type":"ContainerStarted","Data":"a47c4b5fd4b6e8ac90258db6b447ac8974290bdf00310f207519a9a9b2c2d186"} Jul 10 17:11:33.997711 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:33.997501 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lql7p_3756f101-49b1-4754-ad4a-13c572eec69c/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:34.100520 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:34.100485 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lql7p_3756f101-49b1-4754-ad4a-13c572eec69c/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:34.200651 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:34.200620 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7mnkc" Jul 10 17:11:34.200826 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:34.200619 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:11:34.200826 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:34.200758 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7mnkc" podUID="7000764e-d442-4c90-aa00-97b83a65e4b6" Jul 10 17:11:34.200826 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:34.200631 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9jx7" Jul 10 17:11:34.200826 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:34.200805 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smz4h" podUID="ea2e6b67-bf39-4196-a657-7859b8b73199" Jul 10 17:11:34.201015 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:34.200886 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9jx7" podUID="d177790b-fe2d-469e-9dff-c653997fe096" Jul 10 17:11:34.405720 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:34.405638 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zmbx" event={"ID":"0bd5b01a-f45d-41bf-aa6e-3fb6151cd008","Type":"ContainerStarted","Data":"62a13ff4cf5bb9fef1cffb12d579b5418d08c7874f66e526421d43a7d2b7f853"} Jul 10 17:11:34.429781 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:34.429719 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rbwdx" podStartSLOduration=7.616813744 podStartE2EDuration="26.429699516s" totalImagesPullingTime="18.812885772s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:11:32.413753469 +0000 UTC m=+24.741451008" watchObservedRunningTime="2026-07-10 17:11:34.429699516 +0000 UTC m=+26.757397056" Jul 10 17:11:35.366914 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:35.366838 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lql7p" Jul 10 17:11:35.367636 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:35.367608 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lql7p" Jul 10 17:11:35.383361 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:35.383317 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zmbx" podStartSLOduration=5.437714397 podStartE2EDuration="27.383301591s" totalImagesPullingTime="21.945587194s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:11:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:11:34.433804756 +0000 UTC m=+26.761502295" watchObservedRunningTime="2026-07-10 17:11:35.383301591 +0000 UTC m=+27.710999130" Jul 10 17:11:35.412120 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:35.412085 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js69w_e41baaf7-4171-4df1-8bdd-d0697a61fdce/ovn-acl-logging/0.log" Jul 10 17:11:35.412834 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:35.412494 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-js69w" event={"ID":"e41baaf7-4171-4df1-8bdd-d0697a61fdce","Type":"ContainerStarted","Data":"61baa9513e1baf64c4de4490e37bb312891cefaf16b6320fa0844c0364f67a92"} Jul 10 17:11:35.412929 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:35.412911 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lql7p" Jul 10 17:11:35.413259 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:35.413237 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lql7p" Jul 10 17:11:35.413336 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:35.413325 2583 scope.go:117] "RemoveContainer" containerID="f0778bb4cd405d5a8755a029cfe3c1878721c73ad25c8cacad024b5384c5ed13" Jul 10 17:11:35.513787 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:35.513716 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lql7p_3756f101-49b1-4754-ad4a-13c572eec69c/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:36.200185 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:36.200146 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:11:36.200348 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:36.200147 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7mnkc" Jul 10 17:11:36.200348 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:36.200274 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smz4h" podUID="ea2e6b67-bf39-4196-a657-7859b8b73199" Jul 10 17:11:36.200348 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:36.200291 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9jx7" Jul 10 17:11:36.200500 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:36.200405 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7mnkc" podUID="7000764e-d442-4c90-aa00-97b83a65e4b6" Jul 10 17:11:36.200588 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:36.200514 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9jx7" podUID="d177790b-fe2d-469e-9dff-c653997fe096" Jul 10 17:11:36.532799 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:36.532711 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lql7p_3756f101-49b1-4754-ad4a-13c572eec69c/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:36.626326 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:36.626285 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lql7p_3756f101-49b1-4754-ad4a-13c572eec69c/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:36.712050 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:36.712012 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lql7p_3756f101-49b1-4754-ad4a-13c572eec69c/konnectivity-agent/0.log" limit=1024 Jul 10 17:11:37.419673 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:37.419646 2583 generic.go:356] "Generic (PLEG): container finished" podID="1e13ba88-7d4c-42b8-a992-95b3c447b222" containerID="e59b25d1ad28e77c29e364e6bc01f6ac37718b2eca81b14b9bdb2d17e6d00760" exitCode=0 Jul 10 17:11:37.419885 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:37.419709 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vk6rc" event={"ID":"1e13ba88-7d4c-42b8-a992-95b3c447b222","Type":"ContainerDied","Data":"e59b25d1ad28e77c29e364e6bc01f6ac37718b2eca81b14b9bdb2d17e6d00760"} Jul 10 17:11:37.423677 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:37.423655 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js69w_e41baaf7-4171-4df1-8bdd-d0697a61fdce/ovn-acl-logging/0.log" Jul 10 17:11:37.424050 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:37.424020 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-js69w" event={"ID":"e41baaf7-4171-4df1-8bdd-d0697a61fdce","Type":"ContainerStarted","Data":"4dd772d416984f81ebb041a361e76754a2b1393acda2a87d0831cb4647b6ec53"} Jul 10 17:11:37.424123 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:37.424092 2583 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 10 17:11:37.424406 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:37.424366 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:37.424501 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:37.424454 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:37.442312 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:37.442289 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:37.442509 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:37.442447 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:37.521590 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:37.521483 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-js69w" podStartSLOduration=10.627433989 podStartE2EDuration="29.521465826s" totalImagesPullingTime="18.894031837s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:11:37.48171018 +0000 UTC m=+29.809407738" watchObservedRunningTime="2026-07-10 17:11:37.521465826 +0000 UTC m=+29.849163364" Jul 10 17:11:37.522105 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:37.522078 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7mnkc"] Jul 10 17:11:37.522189 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:37.522178 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7mnkc" Jul 10 17:11:37.522268 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:37.522253 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7mnkc" podUID="7000764e-d442-4c90-aa00-97b83a65e4b6" Jul 10 17:11:37.531242 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:37.531215 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h9jx7"] Jul 10 17:11:37.531408 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:37.531342 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9jx7" Jul 10 17:11:37.531503 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:37.531469 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9jx7" podUID="d177790b-fe2d-469e-9dff-c653997fe096" Jul 10 17:11:37.531870 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:37.531849 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-smz4h"] Jul 10 17:11:37.531951 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:37.531936 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:11:37.532033 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:37.532018 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smz4h" podUID="ea2e6b67-bf39-4196-a657-7859b8b73199" Jul 10 17:11:37.914113 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:37.913935 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:11:39.200282 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:39.200236 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9jx7" Jul 10 17:11:39.200282 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:39.200260 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7mnkc" Jul 10 17:11:39.200820 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:39.200236 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:11:39.200820 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:39.200372 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9jx7" podUID="d177790b-fe2d-469e-9dff-c653997fe096" Jul 10 17:11:39.200820 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:39.200422 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smz4h" podUID="ea2e6b67-bf39-4196-a657-7859b8b73199" Jul 10 17:11:39.200820 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:39.200494 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7mnkc" podUID="7000764e-d442-4c90-aa00-97b83a65e4b6" Jul 10 17:11:39.430526 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:39.430493 2583 generic.go:356] "Generic (PLEG): container finished" podID="1e13ba88-7d4c-42b8-a992-95b3c447b222" containerID="4250b2bb60b4ff32758831d3607369162c630434a006c5e45c9c241e60c1c0ef" exitCode=0 Jul 10 17:11:39.430678 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:39.430583 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vk6rc" event={"ID":"1e13ba88-7d4c-42b8-a992-95b3c447b222","Type":"ContainerDied","Data":"4250b2bb60b4ff32758831d3607369162c630434a006c5e45c9c241e60c1c0ef"} Jul 10 17:11:40.437812 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:40.437780 2583 generic.go:356] "Generic (PLEG): container finished" podID="1e13ba88-7d4c-42b8-a992-95b3c447b222" containerID="71b3c89dde6f6c00660b5d8a97d68c936270f177560f81cafae70b867a2cb531" exitCode=0 Jul 10 17:11:40.438500 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:40.437846 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vk6rc" event={"ID":"1e13ba88-7d4c-42b8-a992-95b3c447b222","Type":"ContainerDied","Data":"71b3c89dde6f6c00660b5d8a97d68c936270f177560f81cafae70b867a2cb531"} Jul 10 17:11:41.200059 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:41.200024 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7mnkc" Jul 10 17:11:41.200257 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:41.200143 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9jx7" Jul 10 17:11:41.200257 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:41.200149 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7mnkc" podUID="7000764e-d442-4c90-aa00-97b83a65e4b6" Jul 10 17:11:41.200257 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:41.200157 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:11:41.200429 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:41.200264 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9jx7" podUID="d177790b-fe2d-469e-9dff-c653997fe096" Jul 10 17:11:41.200429 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:41.200354 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smz4h" podUID="ea2e6b67-bf39-4196-a657-7859b8b73199" Jul 10 17:11:41.919432 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:41.919381 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsl5d\" (UniqueName: \"kubernetes.io/projected/7000764e-d442-4c90-aa00-97b83a65e4b6-kube-api-access-rsl5d\") pod \"network-check-target-7mnkc\" (UID: \"7000764e-d442-4c90-aa00-97b83a65e4b6\") " pod="openshift-network-diagnostics/network-check-target-7mnkc" Jul 10 17:11:41.919432 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:41.919436 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea2e6b67-bf39-4196-a657-7859b8b73199-original-pull-secret\") pod \"global-pull-secret-syncer-smz4h\" (UID: \"ea2e6b67-bf39-4196-a657-7859b8b73199\") " pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:11:41.919852 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:41.919476 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d177790b-fe2d-469e-9dff-c653997fe096-metrics-certs\") pod \"network-metrics-daemon-h9jx7\" (UID: \"d177790b-fe2d-469e-9dff-c653997fe096\") " pod="openshift-multus/network-metrics-daemon-h9jx7" Jul 10 17:11:41.919852 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:41.919550 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 17:11:41.919852 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:41.919570 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 17:11:41.919852 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:41.919581 2583 projected.go:196] Error preparing data for projected volume kube-api-access-rsl5d for pod openshift-network-diagnostics/network-check-target-7mnkc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:11:41.919852 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:41.919587 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 17:11:41.919852 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:41.919586 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:11:41.919852 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:41.919626 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7000764e-d442-4c90-aa00-97b83a65e4b6-kube-api-access-rsl5d podName:7000764e-d442-4c90-aa00-97b83a65e4b6 nodeName:}" failed. No retries permitted until 2026-07-10 17:12:13.919611193 +0000 UTC m=+66.247308710 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rsl5d" (UniqueName: "kubernetes.io/projected/7000764e-d442-4c90-aa00-97b83a65e4b6-kube-api-access-rsl5d") pod "network-check-target-7mnkc" (UID: "7000764e-d442-4c90-aa00-97b83a65e4b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:11:41.919852 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:41.919662 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea2e6b67-bf39-4196-a657-7859b8b73199-original-pull-secret podName:ea2e6b67-bf39-4196-a657-7859b8b73199 nodeName:}" failed. No retries permitted until 2026-07-10 17:12:13.91964628 +0000 UTC m=+66.247343799 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea2e6b67-bf39-4196-a657-7859b8b73199-original-pull-secret") pod "global-pull-secret-syncer-smz4h" (UID: "ea2e6b67-bf39-4196-a657-7859b8b73199") : object "kube-system"/"original-pull-secret" not registered Jul 10 17:11:41.919852 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:41.919678 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d177790b-fe2d-469e-9dff-c653997fe096-metrics-certs podName:d177790b-fe2d-469e-9dff-c653997fe096 nodeName:}" failed. No retries permitted until 2026-07-10 17:12:13.919668238 +0000 UTC m=+66.247365755 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d177790b-fe2d-469e-9dff-c653997fe096-metrics-certs") pod "network-metrics-daemon-h9jx7" (UID: "d177790b-fe2d-469e-9dff-c653997fe096") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:11:43.021290 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.021256 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-65.ec2.internal" event="NodeReady" Jul 10 17:11:43.021795 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.021443 2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 17:11:43.078375 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.078341 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wr97q"] Jul 10 17:11:43.080231 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.080216 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wr97q" Jul 10 17:11:43.084009 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.083985 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 17:11:43.084138 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.084003 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 17:11:43.084138 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.083984 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4zvk4\"" Jul 10 17:11:43.085134 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.085113 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-995c6f9fc-hzwlg"] Jul 10 17:11:43.087018 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.086868 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 17:11:43.087246 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.087229 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-r9l2z"] Jul 10 17:11:43.087402 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.087366 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:11:43.088997 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.088981 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r9l2z" Jul 10 17:11:43.090293 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.090271 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 10 17:11:43.092094 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.092077 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 10 17:11:43.092183 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.092161 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-czcbt\"" Jul 10 17:11:43.093799 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.093779 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 17:11:43.093799 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.093798 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 17:11:43.093917 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.093902 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rv7kz\"" Jul 10 17:11:43.095080 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.095052 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 10 17:11:43.101262 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.101240 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wr97q"] Jul 10 17:11:43.103263 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.103240 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 10 17:11:43.114994 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.114972 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r9l2z"] Jul 10 17:11:43.127858 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.127834 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-995c6f9fc-hzwlg"] Jul 10 17:11:43.200686 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.200655 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9jx7" Jul 10 17:11:43.200861 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.200712 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7mnkc" Jul 10 17:11:43.200861 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.200680 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:11:43.204000 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.203979 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 17:11:43.208832 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.208238 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 17:11:43.208832 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.208248 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 17:11:43.208832 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.208701 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rwncl\"" Jul 10 17:11:43.209099 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.209052 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-r2699\"" Jul 10 17:11:43.209295 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.209279 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 17:11:43.227662 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.227633 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4ebb446-d269-4e98-b1e4-258948073db1-metrics-tls\") pod \"dns-default-r9l2z\" (UID: \"d4ebb446-d269-4e98-b1e4-258948073db1\") " pod="openshift-dns/dns-default-r9l2z" Jul 10 17:11:43.227759 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.227674 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb7j4\" (UniqueName: \"kubernetes.io/projected/e973aa85-c8e0-457a-bb77-543b9e5180d8-kube-api-access-bb7j4\") pod \"image-registry-995c6f9fc-hzwlg\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:11:43.227759 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.227693 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e973aa85-c8e0-457a-bb77-543b9e5180d8-registry-tls\") pod \"image-registry-995c6f9fc-hzwlg\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:11:43.227759 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.227743 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8edbb95-6b40-46ed-a7ad-0de010dbf2dd-cert\") pod \"ingress-canary-wr97q\" (UID: \"b8edbb95-6b40-46ed-a7ad-0de010dbf2dd\") " pod="openshift-ingress-canary/ingress-canary-wr97q" Jul 10 17:11:43.227869 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.227782 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e973aa85-c8e0-457a-bb77-543b9e5180d8-registry-certificates\") pod \"image-registry-995c6f9fc-hzwlg\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:11:43.227869 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.227842 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e973aa85-c8e0-457a-bb77-543b9e5180d8-image-registry-private-configuration\") pod \"image-registry-995c6f9fc-hzwlg\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:11:43.227940 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.227870 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p76n4\" (UniqueName: \"kubernetes.io/projected/b8edbb95-6b40-46ed-a7ad-0de010dbf2dd-kube-api-access-p76n4\") pod \"ingress-canary-wr97q\" (UID: \"b8edbb95-6b40-46ed-a7ad-0de010dbf2dd\") " pod="openshift-ingress-canary/ingress-canary-wr97q" Jul 10 17:11:43.227940 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.227901 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4ebb446-d269-4e98-b1e4-258948073db1-tmp-dir\") pod \"dns-default-r9l2z\" (UID: \"d4ebb446-d269-4e98-b1e4-258948073db1\") " pod="openshift-dns/dns-default-r9l2z" Jul 10 17:11:43.227940 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.227928 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e973aa85-c8e0-457a-bb77-543b9e5180d8-trusted-ca\") pod \"image-registry-995c6f9fc-hzwlg\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:11:43.228059 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.227950 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d4ebb446-d269-4e98-b1e4-258948073db1-config-volume\") pod \"dns-default-r9l2z\" (UID: \"d4ebb446-d269-4e98-b1e4-258948073db1\") " pod="openshift-dns/dns-default-r9l2z" Jul 10 17:11:43.228059 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.227992 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e973aa85-c8e0-457a-bb77-543b9e5180d8-installation-pull-secrets\") pod \"image-registry-995c6f9fc-hzwlg\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:11:43.228059 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.228040 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t65rk\" (UniqueName: \"kubernetes.io/projected/d4ebb446-d269-4e98-b1e4-258948073db1-kube-api-access-t65rk\") pod \"dns-default-r9l2z\" (UID: \"d4ebb446-d269-4e98-b1e4-258948073db1\") " pod="openshift-dns/dns-default-r9l2z" Jul 10 17:11:43.228167 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.228107 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e973aa85-c8e0-457a-bb77-543b9e5180d8-bound-sa-token\") pod \"image-registry-995c6f9fc-hzwlg\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:11:43.228167 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.228137 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e973aa85-c8e0-457a-bb77-543b9e5180d8-ca-trust-extracted\") pod \"image-registry-995c6f9fc-hzwlg\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:11:43.328650 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.328566 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e973aa85-c8e0-457a-bb77-543b9e5180d8-bound-sa-token\") pod \"image-registry-995c6f9fc-hzwlg\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:11:43.328650 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.328606 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e973aa85-c8e0-457a-bb77-543b9e5180d8-ca-trust-extracted\") pod \"image-registry-995c6f9fc-hzwlg\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:11:43.328650 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.328635 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4ebb446-d269-4e98-b1e4-258948073db1-metrics-tls\") pod \"dns-default-r9l2z\" (UID: \"d4ebb446-d269-4e98-b1e4-258948073db1\") " pod="openshift-dns/dns-default-r9l2z" Jul 10 17:11:43.328911 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.328672 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb7j4\" (UniqueName: \"kubernetes.io/projected/e973aa85-c8e0-457a-bb77-543b9e5180d8-kube-api-access-bb7j4\") pod \"image-registry-995c6f9fc-hzwlg\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:11:43.328911 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.328698 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e973aa85-c8e0-457a-bb77-543b9e5180d8-registry-tls\") pod \"image-registry-995c6f9fc-hzwlg\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:11:43.328911 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.328716 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8edbb95-6b40-46ed-a7ad-0de010dbf2dd-cert\") pod \"ingress-canary-wr97q\" (UID: \"b8edbb95-6b40-46ed-a7ad-0de010dbf2dd\") " pod="openshift-ingress-canary/ingress-canary-wr97q" Jul 10 17:11:43.328911 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.328738 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e973aa85-c8e0-457a-bb77-543b9e5180d8-registry-certificates\") pod \"image-registry-995c6f9fc-hzwlg\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:11:43.328911 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.328786 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e973aa85-c8e0-457a-bb77-543b9e5180d8-image-registry-private-configuration\") pod \"image-registry-995c6f9fc-hzwlg\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:11:43.328911 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.328809 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p76n4\" (UniqueName: \"kubernetes.io/projected/b8edbb95-6b40-46ed-a7ad-0de010dbf2dd-kube-api-access-p76n4\") pod \"ingress-canary-wr97q\" (UID: \"b8edbb95-6b40-46ed-a7ad-0de010dbf2dd\") " pod="openshift-ingress-canary/ingress-canary-wr97q" Jul 10 17:11:43.328911 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:43.328815 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 17:11:43.328911 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.328838 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4ebb446-d269-4e98-b1e4-258948073db1-tmp-dir\") pod \"dns-default-r9l2z\" (UID: \"d4ebb446-d269-4e98-b1e4-258948073db1\") " pod="openshift-dns/dns-default-r9l2z" Jul 10 17:11:43.328911 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.328866 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e973aa85-c8e0-457a-bb77-543b9e5180d8-trusted-ca\") pod \"image-registry-995c6f9fc-hzwlg\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:11:43.328911 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:43.328882 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4ebb446-d269-4e98-b1e4-258948073db1-metrics-tls podName:d4ebb446-d269-4e98-b1e4-258948073db1 nodeName:}" failed. No retries permitted until 2026-07-10 17:11:43.828862863 +0000 UTC m=+36.156560379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4ebb446-d269-4e98-b1e4-258948073db1-metrics-tls") pod "dns-default-r9l2z" (UID: "d4ebb446-d269-4e98-b1e4-258948073db1") : secret "dns-default-metrics-tls" not found Jul 10 17:11:43.329423 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.328915 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d4ebb446-d269-4e98-b1e4-258948073db1-config-volume\") pod \"dns-default-r9l2z\" (UID: \"d4ebb446-d269-4e98-b1e4-258948073db1\") " pod="openshift-dns/dns-default-r9l2z" Jul 10 17:11:43.329423 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.328947 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e973aa85-c8e0-457a-bb77-543b9e5180d8-installation-pull-secrets\") pod \"image-registry-995c6f9fc-hzwlg\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:11:43.329423 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.328989 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t65rk\" (UniqueName: \"kubernetes.io/projected/d4ebb446-d269-4e98-b1e4-258948073db1-kube-api-access-t65rk\") pod \"dns-default-r9l2z\" (UID: \"d4ebb446-d269-4e98-b1e4-258948073db1\") " pod="openshift-dns/dns-default-r9l2z" Jul 10 17:11:43.329423 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.328991 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e973aa85-c8e0-457a-bb77-543b9e5180d8-ca-trust-extracted\") pod \"image-registry-995c6f9fc-hzwlg\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:11:43.329639 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:43.329525 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 17:11:43.329639 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:43.329544 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-995c6f9fc-hzwlg: secret "image-registry-tls" not found Jul 10 17:11:43.329639 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:43.329614 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e973aa85-c8e0-457a-bb77-543b9e5180d8-registry-tls podName:e973aa85-c8e0-457a-bb77-543b9e5180d8 nodeName:}" failed. No retries permitted until 2026-07-10 17:11:43.829594645 +0000 UTC m=+36.157292177 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e973aa85-c8e0-457a-bb77-543b9e5180d8-registry-tls") pod "image-registry-995c6f9fc-hzwlg" (UID: "e973aa85-c8e0-457a-bb77-543b9e5180d8") : secret "image-registry-tls" not found Jul 10 17:11:43.329787 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.329630 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d4ebb446-d269-4e98-b1e4-258948073db1-config-volume\") pod \"dns-default-r9l2z\" (UID: \"d4ebb446-d269-4e98-b1e4-258948073db1\") " pod="openshift-dns/dns-default-r9l2z" Jul 10 17:11:43.329844 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.329827 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e973aa85-c8e0-457a-bb77-543b9e5180d8-registry-certificates\") pod \"image-registry-995c6f9fc-hzwlg\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:11:43.329909 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.329891 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4ebb446-d269-4e98-b1e4-258948073db1-tmp-dir\") pod \"dns-default-r9l2z\" (UID: \"d4ebb446-d269-4e98-b1e4-258948073db1\") " pod="openshift-dns/dns-default-r9l2z" Jul 10 17:11:43.330471 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.330445 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e973aa85-c8e0-457a-bb77-543b9e5180d8-trusted-ca\") pod \"image-registry-995c6f9fc-hzwlg\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:11:43.333813 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.333784 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e973aa85-c8e0-457a-bb77-543b9e5180d8-image-registry-private-configuration\") pod \"image-registry-995c6f9fc-hzwlg\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:11:43.333927 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.333836 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e973aa85-c8e0-457a-bb77-543b9e5180d8-installation-pull-secrets\") pod \"image-registry-995c6f9fc-hzwlg\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:11:43.333927 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.333871 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8edbb95-6b40-46ed-a7ad-0de010dbf2dd-cert\") pod \"ingress-canary-wr97q\" (UID: \"b8edbb95-6b40-46ed-a7ad-0de010dbf2dd\") " pod="openshift-ingress-canary/ingress-canary-wr97q" Jul 10 17:11:43.341419 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.341374 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t65rk\" (UniqueName: \"kubernetes.io/projected/d4ebb446-d269-4e98-b1e4-258948073db1-kube-api-access-t65rk\") pod \"dns-default-r9l2z\" (UID: \"d4ebb446-d269-4e98-b1e4-258948073db1\") " pod="openshift-dns/dns-default-r9l2z" Jul 10 17:11:43.342027 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.342003 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p76n4\" (UniqueName: \"kubernetes.io/projected/b8edbb95-6b40-46ed-a7ad-0de010dbf2dd-kube-api-access-p76n4\") pod \"ingress-canary-wr97q\" (UID: \"b8edbb95-6b40-46ed-a7ad-0de010dbf2dd\") " pod="openshift-ingress-canary/ingress-canary-wr97q" Jul 10 17:11:43.342110 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.342030 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e973aa85-c8e0-457a-bb77-543b9e5180d8-bound-sa-token\") pod \"image-registry-995c6f9fc-hzwlg\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:11:43.342646 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.342614 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb7j4\" (UniqueName: \"kubernetes.io/projected/e973aa85-c8e0-457a-bb77-543b9e5180d8-kube-api-access-bb7j4\") pod \"image-registry-995c6f9fc-hzwlg\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:11:43.389185 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.389157 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wr97q" Jul 10 17:11:43.538190 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.537976 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wr97q"] Jul 10 17:11:43.541942 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:43.541919 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8edbb95_6b40_46ed_a7ad_0de010dbf2dd.slice/crio-7af239b70a1b5adf0fa53cbe88076c6a6227d565462957ddf31f06bd005f5a08 WatchSource:0}: Error finding container 7af239b70a1b5adf0fa53cbe88076c6a6227d565462957ddf31f06bd005f5a08: Status 404 returned error can't find the container with id 7af239b70a1b5adf0fa53cbe88076c6a6227d565462957ddf31f06bd005f5a08 Jul 10 17:11:43.832807 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.832771 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4ebb446-d269-4e98-b1e4-258948073db1-metrics-tls\") pod \"dns-default-r9l2z\" (UID: \"d4ebb446-d269-4e98-b1e4-258948073db1\") " pod="openshift-dns/dns-default-r9l2z" Jul 10 17:11:43.833005 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:43.832837 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e973aa85-c8e0-457a-bb77-543b9e5180d8-registry-tls\") pod \"image-registry-995c6f9fc-hzwlg\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:11:43.833005 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:43.832939 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 17:11:43.833005 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:43.832949 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 17:11:43.833005 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:43.832972 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-995c6f9fc-hzwlg: secret "image-registry-tls" not found Jul 10 17:11:43.833216 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:43.833014 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4ebb446-d269-4e98-b1e4-258948073db1-metrics-tls podName:d4ebb446-d269-4e98-b1e4-258948073db1 nodeName:}" failed. No retries permitted until 2026-07-10 17:11:44.832993429 +0000 UTC m=+37.160690964 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4ebb446-d269-4e98-b1e4-258948073db1-metrics-tls") pod "dns-default-r9l2z" (UID: "d4ebb446-d269-4e98-b1e4-258948073db1") : secret "dns-default-metrics-tls" not found Jul 10 17:11:43.833216 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:43.833036 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e973aa85-c8e0-457a-bb77-543b9e5180d8-registry-tls podName:e973aa85-c8e0-457a-bb77-543b9e5180d8 nodeName:}" failed. No retries permitted until 2026-07-10 17:11:44.833022719 +0000 UTC m=+37.160720240 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e973aa85-c8e0-457a-bb77-543b9e5180d8-registry-tls") pod "image-registry-995c6f9fc-hzwlg" (UID: "e973aa85-c8e0-457a-bb77-543b9e5180d8") : secret "image-registry-tls" not found Jul 10 17:11:44.036111 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:44.036073 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fvjjb"] Jul 10 17:11:44.039336 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:44.039317 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fvjjb" Jul 10 17:11:44.043738 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:44.043714 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 17:11:44.043871 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:44.043853 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 17:11:44.043944 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:44.043853 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 17:11:44.043944 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:44.043861 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-strpp\"" Jul 10 17:11:44.043944 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:44.043926 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 17:11:44.058186 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:44.058163 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fvjjb"] Jul 10 17:11:44.136263 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:44.136187 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/29817af8-c839-459c-a69d-733e4926b16b-crio-socket\") pod \"insights-runtime-extractor-fvjjb\" (UID: \"29817af8-c839-459c-a69d-733e4926b16b\") " pod="openshift-insights/insights-runtime-extractor-fvjjb" Jul 10 17:11:44.136379 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:44.136302 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29817af8-c839-459c-a69d-733e4926b16b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fvjjb\" (UID: \"29817af8-c839-459c-a69d-733e4926b16b\") " pod="openshift-insights/insights-runtime-extractor-fvjjb" Jul 10 17:11:44.136379 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:44.136332 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29817af8-c839-459c-a69d-733e4926b16b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fvjjb\" (UID: \"29817af8-c839-459c-a69d-733e4926b16b\") " pod="openshift-insights/insights-runtime-extractor-fvjjb" Jul 10 17:11:44.136379 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:44.136357 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz22x\" (UniqueName: \"kubernetes.io/projected/29817af8-c839-459c-a69d-733e4926b16b-kube-api-access-cz22x\") pod \"insights-runtime-extractor-fvjjb\" (UID: \"29817af8-c839-459c-a69d-733e4926b16b\") " pod="openshift-insights/insights-runtime-extractor-fvjjb" Jul 10 17:11:44.136546 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:44.136451 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/29817af8-c839-459c-a69d-733e4926b16b-data-volume\") pod \"insights-runtime-extractor-fvjjb\" (UID: \"29817af8-c839-459c-a69d-733e4926b16b\") " pod="openshift-insights/insights-runtime-extractor-fvjjb" Jul 10 17:11:44.237330 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:44.237291 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29817af8-c839-459c-a69d-733e4926b16b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fvjjb\" (UID: \"29817af8-c839-459c-a69d-733e4926b16b\") " pod="openshift-insights/insights-runtime-extractor-fvjjb" Jul 10 17:11:44.237520 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:44.237338 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29817af8-c839-459c-a69d-733e4926b16b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fvjjb\" (UID: \"29817af8-c839-459c-a69d-733e4926b16b\") " pod="openshift-insights/insights-runtime-extractor-fvjjb" Jul 10 17:11:44.237520 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:44.237367 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz22x\" (UniqueName: \"kubernetes.io/projected/29817af8-c839-459c-a69d-733e4926b16b-kube-api-access-cz22x\") pod \"insights-runtime-extractor-fvjjb\" (UID: \"29817af8-c839-459c-a69d-733e4926b16b\") " pod="openshift-insights/insights-runtime-extractor-fvjjb" Jul 10 17:11:44.237520 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:44.237436 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/29817af8-c839-459c-a69d-733e4926b16b-data-volume\") pod \"insights-runtime-extractor-fvjjb\" (UID: \"29817af8-c839-459c-a69d-733e4926b16b\") " pod="openshift-insights/insights-runtime-extractor-fvjjb" Jul 10 17:11:44.237520 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:44.237469 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/29817af8-c839-459c-a69d-733e4926b16b-crio-socket\") pod \"insights-runtime-extractor-fvjjb\" (UID: \"29817af8-c839-459c-a69d-733e4926b16b\") " pod="openshift-insights/insights-runtime-extractor-fvjjb" Jul 10 17:11:44.237666 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:44.237622 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/29817af8-c839-459c-a69d-733e4926b16b-crio-socket\") pod \"insights-runtime-extractor-fvjjb\" (UID: \"29817af8-c839-459c-a69d-733e4926b16b\") " pod="openshift-insights/insights-runtime-extractor-fvjjb" Jul 10 17:11:44.237702 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:44.237675 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 17:11:44.237745 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:44.237734 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29817af8-c839-459c-a69d-733e4926b16b-kube-rbac-proxy-cm podName:29817af8-c839-459c-a69d-733e4926b16b nodeName:}" failed. No retries permitted until 2026-07-10 17:11:44.73771934 +0000 UTC m=+37.065416857 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/29817af8-c839-459c-a69d-733e4926b16b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fvjjb" (UID: "29817af8-c839-459c-a69d-733e4926b16b") : configmap "kube-rbac-proxy" not found Jul 10 17:11:44.238056 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:44.238033 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 17:11:44.238155 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:44.238089 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29817af8-c839-459c-a69d-733e4926b16b-insights-runtime-extractor-tls podName:29817af8-c839-459c-a69d-733e4926b16b nodeName:}" failed. No retries permitted until 2026-07-10 17:11:44.738073238 +0000 UTC m=+37.065770770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29817af8-c839-459c-a69d-733e4926b16b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fvjjb" (UID: "29817af8-c839-459c-a69d-733e4926b16b") : secret "insights-runtime-extractor-tls" not found Jul 10 17:11:44.238568 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:44.238543 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/29817af8-c839-459c-a69d-733e4926b16b-data-volume\") pod \"insights-runtime-extractor-fvjjb\" (UID: \"29817af8-c839-459c-a69d-733e4926b16b\") " pod="openshift-insights/insights-runtime-extractor-fvjjb" Jul 10 17:11:44.253652 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:44.253632 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz22x\" (UniqueName: \"kubernetes.io/projected/29817af8-c839-459c-a69d-733e4926b16b-kube-api-access-cz22x\") pod \"insights-runtime-extractor-fvjjb\" (UID: \"29817af8-c839-459c-a69d-733e4926b16b\") " pod="openshift-insights/insights-runtime-extractor-fvjjb" Jul 10 17:11:44.450497 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:44.450457 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wr97q" event={"ID":"b8edbb95-6b40-46ed-a7ad-0de010dbf2dd","Type":"ContainerStarted","Data":"7af239b70a1b5adf0fa53cbe88076c6a6227d565462957ddf31f06bd005f5a08"} Jul 10 17:11:44.741240 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:44.741142 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29817af8-c839-459c-a69d-733e4926b16b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fvjjb\" (UID: \"29817af8-c839-459c-a69d-733e4926b16b\") " pod="openshift-insights/insights-runtime-extractor-fvjjb" Jul 10 17:11:44.741240 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:44.741195 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29817af8-c839-459c-a69d-733e4926b16b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fvjjb\" (UID: \"29817af8-c839-459c-a69d-733e4926b16b\") " pod="openshift-insights/insights-runtime-extractor-fvjjb" Jul 10 17:11:44.741479 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:44.741330 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 17:11:44.741479 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:44.741425 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29817af8-c839-459c-a69d-733e4926b16b-insights-runtime-extractor-tls podName:29817af8-c839-459c-a69d-733e4926b16b nodeName:}" failed. No retries permitted until 2026-07-10 17:11:45.741405769 +0000 UTC m=+38.069103300 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29817af8-c839-459c-a69d-733e4926b16b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fvjjb" (UID: "29817af8-c839-459c-a69d-733e4926b16b") : secret "insights-runtime-extractor-tls" not found Jul 10 17:11:44.741647 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:44.741493 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 17:11:44.741647 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:44.741566 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29817af8-c839-459c-a69d-733e4926b16b-kube-rbac-proxy-cm podName:29817af8-c839-459c-a69d-733e4926b16b nodeName:}" failed. No retries permitted until 2026-07-10 17:11:45.741547841 +0000 UTC m=+38.069245376 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/29817af8-c839-459c-a69d-733e4926b16b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fvjjb" (UID: "29817af8-c839-459c-a69d-733e4926b16b") : configmap "kube-rbac-proxy" not found Jul 10 17:11:44.842234 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:44.842196 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4ebb446-d269-4e98-b1e4-258948073db1-metrics-tls\") pod \"dns-default-r9l2z\" (UID: \"d4ebb446-d269-4e98-b1e4-258948073db1\") " pod="openshift-dns/dns-default-r9l2z" Jul 10 17:11:44.842422 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:44.842257 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e973aa85-c8e0-457a-bb77-543b9e5180d8-registry-tls\") pod \"image-registry-995c6f9fc-hzwlg\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:11:44.842507 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:44.842428 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 17:11:44.842507 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:44.842443 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-995c6f9fc-hzwlg: secret "image-registry-tls" not found Jul 10 17:11:44.842507 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:44.842505 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e973aa85-c8e0-457a-bb77-543b9e5180d8-registry-tls podName:e973aa85-c8e0-457a-bb77-543b9e5180d8 nodeName:}" failed. No retries permitted until 2026-07-10 17:11:46.842488266 +0000 UTC m=+39.170185800 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e973aa85-c8e0-457a-bb77-543b9e5180d8-registry-tls") pod "image-registry-995c6f9fc-hzwlg" (UID: "e973aa85-c8e0-457a-bb77-543b9e5180d8") : secret "image-registry-tls" not found Jul 10 17:11:44.842892 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:44.842873 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 17:11:44.842940 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:44.842921 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4ebb446-d269-4e98-b1e4-258948073db1-metrics-tls podName:d4ebb446-d269-4e98-b1e4-258948073db1 nodeName:}" failed. No retries permitted until 2026-07-10 17:11:46.842906885 +0000 UTC m=+39.170604409 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4ebb446-d269-4e98-b1e4-258948073db1-metrics-tls") pod "dns-default-r9l2z" (UID: "d4ebb446-d269-4e98-b1e4-258948073db1") : secret "dns-default-metrics-tls" not found Jul 10 17:11:45.403793 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:45.403764 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bq9tx_9369f652-8e06-4d0a-b00e-07ad43e2bec1/dns-node-resolver/0.log" Jul 10 17:11:45.749131 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:45.749098 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29817af8-c839-459c-a69d-733e4926b16b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fvjjb\" (UID: \"29817af8-c839-459c-a69d-733e4926b16b\") " pod="openshift-insights/insights-runtime-extractor-fvjjb" Jul 10 17:11:45.749131 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:45.749135 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29817af8-c839-459c-a69d-733e4926b16b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fvjjb\" (UID: \"29817af8-c839-459c-a69d-733e4926b16b\") " pod="openshift-insights/insights-runtime-extractor-fvjjb" Jul 10 17:11:45.749346 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:45.749235 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 17:11:45.749346 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:45.749316 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29817af8-c839-459c-a69d-733e4926b16b-kube-rbac-proxy-cm podName:29817af8-c839-459c-a69d-733e4926b16b nodeName:}" failed. No retries permitted until 2026-07-10 17:11:47.749292223 +0000 UTC m=+40.076989767 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/29817af8-c839-459c-a69d-733e4926b16b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fvjjb" (UID: "29817af8-c839-459c-a69d-733e4926b16b") : configmap "kube-rbac-proxy" not found Jul 10 17:11:45.749346 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:45.749237 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 17:11:45.749481 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:45.749384 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29817af8-c839-459c-a69d-733e4926b16b-insights-runtime-extractor-tls podName:29817af8-c839-459c-a69d-733e4926b16b nodeName:}" failed. No retries permitted until 2026-07-10 17:11:47.749370197 +0000 UTC m=+40.077067749 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29817af8-c839-459c-a69d-733e4926b16b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fvjjb" (UID: "29817af8-c839-459c-a69d-733e4926b16b") : secret "insights-runtime-extractor-tls" not found Jul 10 17:11:46.605294 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:46.605264 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5krz5_df9374c1-e474-4ceb-b6d6-a8879a52e376/node-ca/0.log" Jul 10 17:11:46.634306 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:46.634274 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-6vfzj"] Jul 10 17:11:46.638335 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:46.638316 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-6vfzj" Jul 10 17:11:46.640740 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:46.640723 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 10 17:11:46.641055 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:46.641035 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 10 17:11:46.641778 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:46.641761 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 10 17:11:46.641855 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:46.641838 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-r426d\"" Jul 10 17:11:46.643502 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:46.643483 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 10 17:11:46.649336 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:46.649317 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-6vfzj"] Jul 10 17:11:46.758298 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:46.758276 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-55q9h"] Jul 10 17:11:46.758632 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:46.758610 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh5c2\" (UniqueName: \"kubernetes.io/projected/0d0e872f-da71-455e-a988-23a51a0fb2a9-kube-api-access-lh5c2\") pod \"service-ca-79f68b9f4f-6vfzj\" (UID: \"0d0e872f-da71-455e-a988-23a51a0fb2a9\") " pod="openshift-service-ca/service-ca-79f68b9f4f-6vfzj" Jul 10 17:11:46.758698 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:46.758662 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0d0e872f-da71-455e-a988-23a51a0fb2a9-signing-cabundle\") pod \"service-ca-79f68b9f4f-6vfzj\" (UID: \"0d0e872f-da71-455e-a988-23a51a0fb2a9\") " pod="openshift-service-ca/service-ca-79f68b9f4f-6vfzj" Jul 10 17:11:46.758826 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:46.758802 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0d0e872f-da71-455e-a988-23a51a0fb2a9-signing-key\") pod \"service-ca-79f68b9f4f-6vfzj\" (UID: \"0d0e872f-da71-455e-a988-23a51a0fb2a9\") " pod="openshift-service-ca/service-ca-79f68b9f4f-6vfzj" Jul 10 17:11:46.769179 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:46.769159 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-55q9h" Jul 10 17:11:46.771675 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:46.771653 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 10 17:11:46.771768 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:46.771653 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 10 17:11:46.771768 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:46.771741 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-snbhp\"" Jul 10 17:11:46.776724 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:46.775903 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-55q9h"] Jul 10 17:11:46.859437 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:46.859192 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e973aa85-c8e0-457a-bb77-543b9e5180d8-registry-tls\") pod \"image-registry-995c6f9fc-hzwlg\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:11:46.859437 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:46.859352 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 17:11:46.859618 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:46.859441 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-995c6f9fc-hzwlg: secret "image-registry-tls" not found Jul 10 17:11:46.859618 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:46.859456 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0d0e872f-da71-455e-a988-23a51a0fb2a9-signing-key\") pod \"service-ca-79f68b9f4f-6vfzj\" (UID: \"0d0e872f-da71-455e-a988-23a51a0fb2a9\") " pod="openshift-service-ca/service-ca-79f68b9f4f-6vfzj" Jul 10 17:11:46.859618 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:46.859503 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e973aa85-c8e0-457a-bb77-543b9e5180d8-registry-tls podName:e973aa85-c8e0-457a-bb77-543b9e5180d8 nodeName:}" failed. No retries permitted until 2026-07-10 17:11:50.859484892 +0000 UTC m=+43.187182413 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e973aa85-c8e0-457a-bb77-543b9e5180d8-registry-tls") pod "image-registry-995c6f9fc-hzwlg" (UID: "e973aa85-c8e0-457a-bb77-543b9e5180d8") : secret "image-registry-tls" not found Jul 10 17:11:46.859618 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:46.859575 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4ebb446-d269-4e98-b1e4-258948073db1-metrics-tls\") pod \"dns-default-r9l2z\" (UID: \"d4ebb446-d269-4e98-b1e4-258948073db1\") " pod="openshift-dns/dns-default-r9l2z" Jul 10 17:11:46.859618 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:46.859595 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh5c2\" (UniqueName: \"kubernetes.io/projected/0d0e872f-da71-455e-a988-23a51a0fb2a9-kube-api-access-lh5c2\") pod \"service-ca-79f68b9f4f-6vfzj\" (UID: \"0d0e872f-da71-455e-a988-23a51a0fb2a9\") " pod="openshift-service-ca/service-ca-79f68b9f4f-6vfzj" Jul 10 17:11:46.859849 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:46.859653 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0d0e872f-da71-455e-a988-23a51a0fb2a9-signing-cabundle\") pod \"service-ca-79f68b9f4f-6vfzj\" (UID: \"0d0e872f-da71-455e-a988-23a51a0fb2a9\") " pod="openshift-service-ca/service-ca-79f68b9f4f-6vfzj" Jul 10 17:11:46.859849 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:46.859747 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 17:11:46.859849 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:46.859808 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4ebb446-d269-4e98-b1e4-258948073db1-metrics-tls podName:d4ebb446-d269-4e98-b1e4-258948073db1 nodeName:}" failed. No retries permitted until 2026-07-10 17:11:50.8597904 +0000 UTC m=+43.187487932 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4ebb446-d269-4e98-b1e4-258948073db1-metrics-tls") pod "dns-default-r9l2z" (UID: "d4ebb446-d269-4e98-b1e4-258948073db1") : secret "dns-default-metrics-tls" not found Jul 10 17:11:46.860361 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:46.860338 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0d0e872f-da71-455e-a988-23a51a0fb2a9-signing-cabundle\") pod \"service-ca-79f68b9f4f-6vfzj\" (UID: \"0d0e872f-da71-455e-a988-23a51a0fb2a9\") " pod="openshift-service-ca/service-ca-79f68b9f4f-6vfzj" Jul 10 17:11:46.863234 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:46.863207 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0d0e872f-da71-455e-a988-23a51a0fb2a9-signing-key\") pod \"service-ca-79f68b9f4f-6vfzj\" (UID: \"0d0e872f-da71-455e-a988-23a51a0fb2a9\") " pod="openshift-service-ca/service-ca-79f68b9f4f-6vfzj" Jul 10 17:11:46.868964 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:46.868912 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh5c2\" (UniqueName: \"kubernetes.io/projected/0d0e872f-da71-455e-a988-23a51a0fb2a9-kube-api-access-lh5c2\") pod \"service-ca-79f68b9f4f-6vfzj\" (UID: \"0d0e872f-da71-455e-a988-23a51a0fb2a9\") " pod="openshift-service-ca/service-ca-79f68b9f4f-6vfzj" Jul 10 17:11:46.947198 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:46.947159 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-6vfzj" Jul 10 17:11:46.960030 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:46.960003 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb7q6\" (UniqueName: \"kubernetes.io/projected/96d6166e-fa6f-42f6-81d9-18d199c2516a-kube-api-access-sb7q6\") pod \"migrator-7f59b54f6c-55q9h\" (UID: \"96d6166e-fa6f-42f6-81d9-18d199c2516a\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-55q9h" Jul 10 17:11:47.061032 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:47.061001 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb7q6\" (UniqueName: \"kubernetes.io/projected/96d6166e-fa6f-42f6-81d9-18d199c2516a-kube-api-access-sb7q6\") pod \"migrator-7f59b54f6c-55q9h\" (UID: \"96d6166e-fa6f-42f6-81d9-18d199c2516a\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-55q9h" Jul 10 17:11:47.071278 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:47.071256 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb7q6\" (UniqueName: \"kubernetes.io/projected/96d6166e-fa6f-42f6-81d9-18d199c2516a-kube-api-access-sb7q6\") pod \"migrator-7f59b54f6c-55q9h\" (UID: \"96d6166e-fa6f-42f6-81d9-18d199c2516a\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-55q9h" Jul 10 17:11:47.079163 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:47.079145 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-55q9h" Jul 10 17:11:47.082822 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:47.082781 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-6vfzj"] Jul 10 17:11:47.085999 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:47.085979 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d0e872f_da71_455e_a988_23a51a0fb2a9.slice/crio-4b697e3c10f1590c33e6c175a6e9a747e0c9e77a992d4a9ef5e6dbade6515066 WatchSource:0}: Error finding container 4b697e3c10f1590c33e6c175a6e9a747e0c9e77a992d4a9ef5e6dbade6515066: Status 404 returned error can't find the container with id 4b697e3c10f1590c33e6c175a6e9a747e0c9e77a992d4a9ef5e6dbade6515066 Jul 10 17:11:47.245756 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:47.245727 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-55q9h"] Jul 10 17:11:47.248365 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:47.248331 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d6166e_fa6f_42f6_81d9_18d199c2516a.slice/crio-ba58e95f0775019f7d899e5b7c2e64e3940245a455fd115009ebe073bb1013af WatchSource:0}: Error finding container ba58e95f0775019f7d899e5b7c2e64e3940245a455fd115009ebe073bb1013af: Status 404 returned error can't find the container with id ba58e95f0775019f7d899e5b7c2e64e3940245a455fd115009ebe073bb1013af Jul 10 17:11:47.459017 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:47.458982 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wr97q" event={"ID":"b8edbb95-6b40-46ed-a7ad-0de010dbf2dd","Type":"ContainerStarted","Data":"3da67ed13be2b6dbc8c47557c20d978b9bcabf9622adeeef8f5c083386fe0089"} Jul 10 17:11:47.460125 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:47.460101 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-55q9h" event={"ID":"96d6166e-fa6f-42f6-81d9-18d199c2516a","Type":"ContainerStarted","Data":"ba58e95f0775019f7d899e5b7c2e64e3940245a455fd115009ebe073bb1013af"} Jul 10 17:11:47.461133 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:47.461115 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-6vfzj" event={"ID":"0d0e872f-da71-455e-a988-23a51a0fb2a9","Type":"ContainerStarted","Data":"4b697e3c10f1590c33e6c175a6e9a747e0c9e77a992d4a9ef5e6dbade6515066"} Jul 10 17:11:47.463575 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:47.463543 2583 generic.go:356] "Generic (PLEG): container finished" podID="1e13ba88-7d4c-42b8-a992-95b3c447b222" containerID="0a5ef2e4c55ec2792216b418610c141d7992f2fa986f622457f9f65cc87320ab" exitCode=0 Jul 10 17:11:47.463667 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:47.463587 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vk6rc" event={"ID":"1e13ba88-7d4c-42b8-a992-95b3c447b222","Type":"ContainerDied","Data":"0a5ef2e4c55ec2792216b418610c141d7992f2fa986f622457f9f65cc87320ab"} Jul 10 17:11:47.490731 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:47.490687 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wr97q" podStartSLOduration=1.330802808 podStartE2EDuration="4.49066912s" totalImagesPullingTime="3.159866312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:11:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:11:47.488708846 +0000 UTC m=+39.816406390" watchObservedRunningTime="2026-07-10 17:11:47.49066912 +0000 UTC m=+39.818366658" Jul 10 17:11:47.766836 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:47.766606 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29817af8-c839-459c-a69d-733e4926b16b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fvjjb\" (UID: \"29817af8-c839-459c-a69d-733e4926b16b\") " pod="openshift-insights/insights-runtime-extractor-fvjjb" Jul 10 17:11:47.766836 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:47.766786 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29817af8-c839-459c-a69d-733e4926b16b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fvjjb\" (UID: \"29817af8-c839-459c-a69d-733e4926b16b\") " pod="openshift-insights/insights-runtime-extractor-fvjjb" Jul 10 17:11:47.767517 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:47.766734 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 17:11:47.767517 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:47.766910 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29817af8-c839-459c-a69d-733e4926b16b-kube-rbac-proxy-cm podName:29817af8-c839-459c-a69d-733e4926b16b nodeName:}" failed. No retries permitted until 2026-07-10 17:11:51.766892303 +0000 UTC m=+44.094589825 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/29817af8-c839-459c-a69d-733e4926b16b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fvjjb" (UID: "29817af8-c839-459c-a69d-733e4926b16b") : configmap "kube-rbac-proxy" not found Jul 10 17:11:47.767517 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:47.766937 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 17:11:47.767517 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:47.766982 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29817af8-c839-459c-a69d-733e4926b16b-insights-runtime-extractor-tls podName:29817af8-c839-459c-a69d-733e4926b16b nodeName:}" failed. No retries permitted until 2026-07-10 17:11:51.766971619 +0000 UTC m=+44.094669136 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29817af8-c839-459c-a69d-733e4926b16b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fvjjb" (UID: "29817af8-c839-459c-a69d-733e4926b16b") : secret "insights-runtime-extractor-tls" not found Jul 10 17:11:48.469568 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:48.469538 2583 generic.go:356] "Generic (PLEG): container finished" podID="1e13ba88-7d4c-42b8-a992-95b3c447b222" containerID="f7ab1b012f9c8607cfb224d29f8c9a6329420704e6636cac3fded87eead2fb9a" exitCode=0 Jul 10 17:11:48.469733 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:48.469609 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vk6rc" event={"ID":"1e13ba88-7d4c-42b8-a992-95b3c447b222","Type":"ContainerDied","Data":"f7ab1b012f9c8607cfb224d29f8c9a6329420704e6636cac3fded87eead2fb9a"} Jul 10 17:11:49.477095 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:49.477057 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vk6rc" event={"ID":"1e13ba88-7d4c-42b8-a992-95b3c447b222","Type":"ContainerStarted","Data":"460fcff47ae5f18981f715ee3c5f1faeb18cd9c904b6a9bca39301ce1f73ff8f"} Jul 10 17:11:49.509566 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:49.509506 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vk6rc" podStartSLOduration=7.515695707 podStartE2EDuration="41.509479615s" totalImagesPullingTime="33.993783908s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:11:08 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:11:49.508686505 +0000 UTC m=+41.836384045" watchObservedRunningTime="2026-07-10 17:11:49.509479615 +0000 UTC m=+41.837177155" Jul 10 17:11:50.481250 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:50.481215 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-55q9h" event={"ID":"96d6166e-fa6f-42f6-81d9-18d199c2516a","Type":"ContainerStarted","Data":"86d8f5c2b35496134c3f24e8720c662c566b4eac530da5968c0e00d0d923be4c"} Jul 10 17:11:50.481723 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:50.481275 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-55q9h" event={"ID":"96d6166e-fa6f-42f6-81d9-18d199c2516a","Type":"ContainerStarted","Data":"90fa0286bebf455fdedfdc13bc9542ebe73aab630fee829edaba2788f34e475d"} Jul 10 17:11:50.482771 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:50.482744 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-6vfzj" event={"ID":"0d0e872f-da71-455e-a988-23a51a0fb2a9","Type":"ContainerStarted","Data":"b16c7577b3781c8afe8cb1636f2d94502516a04566a2e2478c43a1a82fe2ccd0"} Jul 10 17:11:50.519376 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:50.519326 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-6vfzj" podStartSLOduration=1.566631587 podStartE2EDuration="4.519306749s" totalImagesPullingTime="2.952675162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:11:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:11:50.51905698 +0000 UTC m=+42.846754520" watchObservedRunningTime="2026-07-10 17:11:50.519306749 +0000 UTC m=+42.847004288" Jul 10 17:11:50.519573 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:50.519423 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-55q9h" podStartSLOduration=1.734573787 podStartE2EDuration="4.519418105s" totalImagesPullingTime="2.784844318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:11:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:11:50.49863276 +0000 UTC m=+42.826330300" watchObservedRunningTime="2026-07-10 17:11:50.519418105 +0000 UTC m=+42.847115644" Jul 10 17:11:50.893555 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:50.893450 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e973aa85-c8e0-457a-bb77-543b9e5180d8-registry-tls\") pod \"image-registry-995c6f9fc-hzwlg\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:11:50.893722 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:50.893609 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4ebb446-d269-4e98-b1e4-258948073db1-metrics-tls\") pod \"dns-default-r9l2z\" (UID: \"d4ebb446-d269-4e98-b1e4-258948073db1\") " pod="openshift-dns/dns-default-r9l2z" Jul 10 17:11:50.893722 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:50.893622 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 17:11:50.893722 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:50.893648 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-995c6f9fc-hzwlg: secret "image-registry-tls" not found Jul 10 17:11:50.893882 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:50.893733 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 17:11:50.893882 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:50.893735 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e973aa85-c8e0-457a-bb77-543b9e5180d8-registry-tls podName:e973aa85-c8e0-457a-bb77-543b9e5180d8 nodeName:}" failed. No retries permitted until 2026-07-10 17:11:58.893713125 +0000 UTC m=+51.221410647 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e973aa85-c8e0-457a-bb77-543b9e5180d8-registry-tls") pod "image-registry-995c6f9fc-hzwlg" (UID: "e973aa85-c8e0-457a-bb77-543b9e5180d8") : secret "image-registry-tls" not found Jul 10 17:11:50.893882 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:50.893793 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4ebb446-d269-4e98-b1e4-258948073db1-metrics-tls podName:d4ebb446-d269-4e98-b1e4-258948073db1 nodeName:}" failed. No retries permitted until 2026-07-10 17:11:58.893777975 +0000 UTC m=+51.221475507 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4ebb446-d269-4e98-b1e4-258948073db1-metrics-tls") pod "dns-default-r9l2z" (UID: "d4ebb446-d269-4e98-b1e4-258948073db1") : secret "dns-default-metrics-tls" not found Jul 10 17:11:51.800796 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:51.800754 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29817af8-c839-459c-a69d-733e4926b16b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fvjjb\" (UID: \"29817af8-c839-459c-a69d-733e4926b16b\") " pod="openshift-insights/insights-runtime-extractor-fvjjb" Jul 10 17:11:51.801266 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:51.800807 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29817af8-c839-459c-a69d-733e4926b16b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fvjjb\" (UID: \"29817af8-c839-459c-a69d-733e4926b16b\") " pod="openshift-insights/insights-runtime-extractor-fvjjb" Jul 10 17:11:51.801266 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:51.800879 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 17:11:51.801266 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:51.800919 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 17:11:51.801266 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:51.800970 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29817af8-c839-459c-a69d-733e4926b16b-kube-rbac-proxy-cm podName:29817af8-c839-459c-a69d-733e4926b16b nodeName:}" failed. No retries permitted until 2026-07-10 17:11:59.800949572 +0000 UTC m=+52.128647103 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/29817af8-c839-459c-a69d-733e4926b16b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fvjjb" (UID: "29817af8-c839-459c-a69d-733e4926b16b") : configmap "kube-rbac-proxy" not found Jul 10 17:11:51.801266 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:51.800991 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29817af8-c839-459c-a69d-733e4926b16b-insights-runtime-extractor-tls podName:29817af8-c839-459c-a69d-733e4926b16b nodeName:}" failed. No retries permitted until 2026-07-10 17:11:59.800981399 +0000 UTC m=+52.128678926 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29817af8-c839-459c-a69d-733e4926b16b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fvjjb" (UID: "29817af8-c839-459c-a69d-733e4926b16b") : secret "insights-runtime-extractor-tls" not found Jul 10 17:11:58.951603 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:58.951553 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e973aa85-c8e0-457a-bb77-543b9e5180d8-registry-tls\") pod \"image-registry-995c6f9fc-hzwlg\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:11:58.952149 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:58.951702 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4ebb446-d269-4e98-b1e4-258948073db1-metrics-tls\") pod \"dns-default-r9l2z\" (UID: \"d4ebb446-d269-4e98-b1e4-258948073db1\") " pod="openshift-dns/dns-default-r9l2z" Jul 10 17:11:58.954952 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:58.954927 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4ebb446-d269-4e98-b1e4-258948073db1-metrics-tls\") pod \"dns-default-r9l2z\" (UID: \"d4ebb446-d269-4e98-b1e4-258948073db1\") " pod="openshift-dns/dns-default-r9l2z" Jul 10 17:11:58.954952 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:58.954948 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e973aa85-c8e0-457a-bb77-543b9e5180d8-registry-tls\") pod \"image-registry-995c6f9fc-hzwlg\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:11:58.999543 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:58.999503 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:11:59.005611 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:59.005578 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r9l2z" Jul 10 17:11:59.176127 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:59.176091 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-995c6f9fc-hzwlg"] Jul 10 17:11:59.179245 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:59.179212 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode973aa85_c8e0_457a_bb77_543b9e5180d8.slice/crio-7db3726ab5f8b43f94fb8b528b213e407e5ed864521572b0b6619d77940a001f WatchSource:0}: Error finding container 7db3726ab5f8b43f94fb8b528b213e407e5ed864521572b0b6619d77940a001f: Status 404 returned error can't find the container with id 7db3726ab5f8b43f94fb8b528b213e407e5ed864521572b0b6619d77940a001f Jul 10 17:11:59.195150 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:59.195123 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r9l2z"] Jul 10 17:11:59.199826 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:11:59.199803 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4ebb446_d269_4e98_b1e4_258948073db1.slice/crio-f07982e0deabe2d41af783790f25b2bc4c2b6298e3540b8ce2283130c14c20ee WatchSource:0}: Error finding container f07982e0deabe2d41af783790f25b2bc4c2b6298e3540b8ce2283130c14c20ee: Status 404 returned error can't find the container with id f07982e0deabe2d41af783790f25b2bc4c2b6298e3540b8ce2283130c14c20ee Jul 10 17:11:59.508048 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:59.507959 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r9l2z" event={"ID":"d4ebb446-d269-4e98-b1e4-258948073db1","Type":"ContainerStarted","Data":"f07982e0deabe2d41af783790f25b2bc4c2b6298e3540b8ce2283130c14c20ee"} Jul 10 17:11:59.512533 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:59.512497 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" event={"ID":"e973aa85-c8e0-457a-bb77-543b9e5180d8","Type":"ContainerStarted","Data":"be93ca7d8403e0ac6676a8a944628457cafcf5c0bf2c4fab139c46d1acad4470"} Jul 10 17:11:59.512709 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:59.512555 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" event={"ID":"e973aa85-c8e0-457a-bb77-543b9e5180d8","Type":"ContainerStarted","Data":"7db3726ab5f8b43f94fb8b528b213e407e5ed864521572b0b6619d77940a001f"} Jul 10 17:11:59.512709 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:59.512644 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:11:59.553100 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:59.553035 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" podStartSLOduration=23.55301282 podStartE2EDuration="23.55301282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:11:59.552270329 +0000 UTC m=+51.879967872" watchObservedRunningTime="2026-07-10 17:11:59.55301282 +0000 UTC m=+51.880710360" Jul 10 17:11:59.861100 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:59.861007 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29817af8-c839-459c-a69d-733e4926b16b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fvjjb\" (UID: \"29817af8-c839-459c-a69d-733e4926b16b\") " pod="openshift-insights/insights-runtime-extractor-fvjjb" Jul 10 17:11:59.861271 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:59.861096 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29817af8-c839-459c-a69d-733e4926b16b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fvjjb\" (UID: \"29817af8-c839-459c-a69d-733e4926b16b\") " pod="openshift-insights/insights-runtime-extractor-fvjjb" Jul 10 17:11:59.861330 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:59.861267 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 17:11:59.861381 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:11:59.861337 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29817af8-c839-459c-a69d-733e4926b16b-insights-runtime-extractor-tls podName:29817af8-c839-459c-a69d-733e4926b16b nodeName:}" failed. No retries permitted until 2026-07-10 17:12:15.861318032 +0000 UTC m=+68.189015565 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29817af8-c839-459c-a69d-733e4926b16b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fvjjb" (UID: "29817af8-c839-459c-a69d-733e4926b16b") : secret "insights-runtime-extractor-tls" not found Jul 10 17:11:59.861649 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:11:59.861626 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29817af8-c839-459c-a69d-733e4926b16b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fvjjb\" (UID: \"29817af8-c839-459c-a69d-733e4926b16b\") " pod="openshift-insights/insights-runtime-extractor-fvjjb" Jul 10 17:12:02.524280 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:02.524247 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r9l2z" event={"ID":"d4ebb446-d269-4e98-b1e4-258948073db1","Type":"ContainerStarted","Data":"ec1c1584fc5221e4d062bbd55efcd4512cf606a47c8fed9737781f86ab98a9e2"} Jul 10 17:12:02.524688 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:02.524299 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r9l2z" event={"ID":"d4ebb446-d269-4e98-b1e4-258948073db1","Type":"ContainerStarted","Data":"a55b984d4c72497179546ca047ff55cbead13883d3a9a325ec776db03db31c99"} Jul 10 17:12:02.524688 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:02.524378 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-r9l2z" Jul 10 17:12:02.554940 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:02.554885 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-r9l2z" podStartSLOduration=17.194502995 podStartE2EDuration="19.554868872s" totalImagesPullingTime="2.360365877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:11:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:12:02.554051071 +0000 UTC m=+54.881748610" watchObservedRunningTime="2026-07-10 17:12:02.554868872 +0000 UTC m=+54.882566410" Jul 10 17:12:09.445536 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:09.445509 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-js69w" Jul 10 17:12:12.426417 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.426368 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c458bf5b-vnwqr"] Jul 10 17:12:12.430311 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.430290 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-774457d448-5pmtf"] Jul 10 17:12:12.430477 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.430430 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c458bf5b-vnwqr" Jul 10 17:12:12.432232 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.432215 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-774457d448-5pmtf" Jul 10 17:12:12.433344 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.433321 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 10 17:12:12.433525 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.433492 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 10 17:12:12.433525 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.433518 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 10 17:12:12.434838 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.434803 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 10 17:12:12.435805 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.435785 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-zvd8n\"" Jul 10 17:12:12.435938 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.435816 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 10 17:12:12.436172 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.435881 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 10 17:12:12.439672 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.439647 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 10 17:12:12.439825 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.439751 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 10 17:12:12.452434 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.452414 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c458bf5b-vnwqr"] Jul 10 17:12:12.456703 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.456681 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm7r7\" (UniqueName: \"kubernetes.io/projected/a62d79fa-d075-459b-be69-c6d799c01c9c-kube-api-access-lm7r7\") pod \"cluster-proxy-proxy-agent-75c458bf5b-vnwqr\" (UID: \"a62d79fa-d075-459b-be69-c6d799c01c9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c458bf5b-vnwqr" Jul 10 17:12:12.456799 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.456713 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a62d79fa-d075-459b-be69-c6d799c01c9c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75c458bf5b-vnwqr\" (UID: \"a62d79fa-d075-459b-be69-c6d799c01c9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c458bf5b-vnwqr" Jul 10 17:12:12.456799 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.456733 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a62d79fa-d075-459b-be69-c6d799c01c9c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75c458bf5b-vnwqr\" (UID: \"a62d79fa-d075-459b-be69-c6d799c01c9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c458bf5b-vnwqr" Jul 10 17:12:12.456799 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.456766 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37baf233-9a94-4ae8-8a44-c0336794c5b7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-774457d448-5pmtf\" (UID: \"37baf233-9a94-4ae8-8a44-c0336794c5b7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-774457d448-5pmtf" Jul 10 17:12:12.456931 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.456813 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a62d79fa-d075-459b-be69-c6d799c01c9c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75c458bf5b-vnwqr\" (UID: \"a62d79fa-d075-459b-be69-c6d799c01c9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c458bf5b-vnwqr" Jul 10 17:12:12.456931 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.456902 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a62d79fa-d075-459b-be69-c6d799c01c9c-hub\") pod \"cluster-proxy-proxy-agent-75c458bf5b-vnwqr\" (UID: \"a62d79fa-d075-459b-be69-c6d799c01c9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c458bf5b-vnwqr" Jul 10 17:12:12.457020 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.456939 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxwnf\" (UniqueName: \"kubernetes.io/projected/37baf233-9a94-4ae8-8a44-c0336794c5b7-kube-api-access-fxwnf\") pod \"managed-serviceaccount-addon-agent-774457d448-5pmtf\" (UID: \"37baf233-9a94-4ae8-8a44-c0336794c5b7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-774457d448-5pmtf" Jul 10 17:12:12.457020 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.456966 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a62d79fa-d075-459b-be69-c6d799c01c9c-ca\") pod \"cluster-proxy-proxy-agent-75c458bf5b-vnwqr\" (UID: \"a62d79fa-d075-459b-be69-c6d799c01c9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c458bf5b-vnwqr" Jul 10 17:12:12.472277 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.472256 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-774457d448-5pmtf"] Jul 10 17:12:12.483183 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.483162 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-995c6f9fc-hzwlg"] Jul 10 17:12:12.530308 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.530279 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-r9l2z" Jul 10 17:12:12.557349 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.557319 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a62d79fa-d075-459b-be69-c6d799c01c9c-ca\") pod \"cluster-proxy-proxy-agent-75c458bf5b-vnwqr\" (UID: \"a62d79fa-d075-459b-be69-c6d799c01c9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c458bf5b-vnwqr" Jul 10 17:12:12.557508 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.557360 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm7r7\" (UniqueName: \"kubernetes.io/projected/a62d79fa-d075-459b-be69-c6d799c01c9c-kube-api-access-lm7r7\") pod \"cluster-proxy-proxy-agent-75c458bf5b-vnwqr\" (UID: \"a62d79fa-d075-459b-be69-c6d799c01c9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c458bf5b-vnwqr" Jul 10 17:12:12.557508 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.557382 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a62d79fa-d075-459b-be69-c6d799c01c9c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75c458bf5b-vnwqr\" (UID: \"a62d79fa-d075-459b-be69-c6d799c01c9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c458bf5b-vnwqr" Jul 10 17:12:12.557508 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.557442 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a62d79fa-d075-459b-be69-c6d799c01c9c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75c458bf5b-vnwqr\" (UID: \"a62d79fa-d075-459b-be69-c6d799c01c9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c458bf5b-vnwqr" Jul 10 17:12:12.557508 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.557492 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37baf233-9a94-4ae8-8a44-c0336794c5b7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-774457d448-5pmtf\" (UID: \"37baf233-9a94-4ae8-8a44-c0336794c5b7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-774457d448-5pmtf" Jul 10 17:12:12.557710 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.557527 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a62d79fa-d075-459b-be69-c6d799c01c9c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75c458bf5b-vnwqr\" (UID: \"a62d79fa-d075-459b-be69-c6d799c01c9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c458bf5b-vnwqr" Jul 10 17:12:12.557710 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.557610 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a62d79fa-d075-459b-be69-c6d799c01c9c-hub\") pod \"cluster-proxy-proxy-agent-75c458bf5b-vnwqr\" (UID: \"a62d79fa-d075-459b-be69-c6d799c01c9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c458bf5b-vnwqr" Jul 10 17:12:12.557710 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.557674 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxwnf\" (UniqueName: \"kubernetes.io/projected/37baf233-9a94-4ae8-8a44-c0336794c5b7-kube-api-access-fxwnf\") pod \"managed-serviceaccount-addon-agent-774457d448-5pmtf\" (UID: \"37baf233-9a94-4ae8-8a44-c0336794c5b7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-774457d448-5pmtf" Jul 10 17:12:12.558344 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.558278 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a62d79fa-d075-459b-be69-c6d799c01c9c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75c458bf5b-vnwqr\" (UID: \"a62d79fa-d075-459b-be69-c6d799c01c9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c458bf5b-vnwqr" Jul 10 17:12:12.561036 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.560489 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a62d79fa-d075-459b-be69-c6d799c01c9c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75c458bf5b-vnwqr\" (UID: \"a62d79fa-d075-459b-be69-c6d799c01c9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c458bf5b-vnwqr" Jul 10 17:12:12.561036 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.560625 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37baf233-9a94-4ae8-8a44-c0336794c5b7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-774457d448-5pmtf\" (UID: \"37baf233-9a94-4ae8-8a44-c0336794c5b7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-774457d448-5pmtf" Jul 10 17:12:12.561036 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.560813 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a62d79fa-d075-459b-be69-c6d799c01c9c-hub\") pod \"cluster-proxy-proxy-agent-75c458bf5b-vnwqr\" (UID: \"a62d79fa-d075-459b-be69-c6d799c01c9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c458bf5b-vnwqr" Jul 10 17:12:12.561036 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.561014 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a62d79fa-d075-459b-be69-c6d799c01c9c-ca\") pod \"cluster-proxy-proxy-agent-75c458bf5b-vnwqr\" (UID: \"a62d79fa-d075-459b-be69-c6d799c01c9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c458bf5b-vnwqr" Jul 10 17:12:12.561036 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.561031 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a62d79fa-d075-459b-be69-c6d799c01c9c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75c458bf5b-vnwqr\" (UID: \"a62d79fa-d075-459b-be69-c6d799c01c9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c458bf5b-vnwqr" Jul 10 17:12:12.581189 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.581163 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxwnf\" (UniqueName: \"kubernetes.io/projected/37baf233-9a94-4ae8-8a44-c0336794c5b7-kube-api-access-fxwnf\") pod \"managed-serviceaccount-addon-agent-774457d448-5pmtf\" (UID: \"37baf233-9a94-4ae8-8a44-c0336794c5b7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-774457d448-5pmtf" Jul 10 17:12:12.581451 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.581431 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm7r7\" (UniqueName: \"kubernetes.io/projected/a62d79fa-d075-459b-be69-c6d799c01c9c-kube-api-access-lm7r7\") pod \"cluster-proxy-proxy-agent-75c458bf5b-vnwqr\" (UID: \"a62d79fa-d075-459b-be69-c6d799c01c9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c458bf5b-vnwqr" Jul 10 17:12:12.634270 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.634236 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-cb8684c9b-nckr9"] Jul 10 17:12:12.641003 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.640985 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cb8684c9b-nckr9" Jul 10 17:12:12.680035 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.679981 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cb8684c9b-nckr9"] Jul 10 17:12:12.759966 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.759941 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/44b85e98-52d3-4c78-84b7-7715d239e1bd-trusted-ca\") pod \"image-registry-cb8684c9b-nckr9\" (UID: \"44b85e98-52d3-4c78-84b7-7715d239e1bd\") " pod="openshift-image-registry/image-registry-cb8684c9b-nckr9" Jul 10 17:12:12.760081 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.759973 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/44b85e98-52d3-4c78-84b7-7715d239e1bd-ca-trust-extracted\") pod \"image-registry-cb8684c9b-nckr9\" (UID: \"44b85e98-52d3-4c78-84b7-7715d239e1bd\") " pod="openshift-image-registry/image-registry-cb8684c9b-nckr9" Jul 10 17:12:12.760081 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.759993 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/44b85e98-52d3-4c78-84b7-7715d239e1bd-installation-pull-secrets\") pod \"image-registry-cb8684c9b-nckr9\" (UID: \"44b85e98-52d3-4c78-84b7-7715d239e1bd\") " pod="openshift-image-registry/image-registry-cb8684c9b-nckr9" Jul 10 17:12:12.760081 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.760021 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/44b85e98-52d3-4c78-84b7-7715d239e1bd-image-registry-private-configuration\") pod \"image-registry-cb8684c9b-nckr9\" (UID: \"44b85e98-52d3-4c78-84b7-7715d239e1bd\") " pod="openshift-image-registry/image-registry-cb8684c9b-nckr9" Jul 10 17:12:12.760237 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.760107 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vmhq\" (UniqueName: \"kubernetes.io/projected/44b85e98-52d3-4c78-84b7-7715d239e1bd-kube-api-access-5vmhq\") pod \"image-registry-cb8684c9b-nckr9\" (UID: \"44b85e98-52d3-4c78-84b7-7715d239e1bd\") " pod="openshift-image-registry/image-registry-cb8684c9b-nckr9" Jul 10 17:12:12.760237 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.760154 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44b85e98-52d3-4c78-84b7-7715d239e1bd-bound-sa-token\") pod \"image-registry-cb8684c9b-nckr9\" (UID: \"44b85e98-52d3-4c78-84b7-7715d239e1bd\") " pod="openshift-image-registry/image-registry-cb8684c9b-nckr9" Jul 10 17:12:12.760237 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.760184 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/44b85e98-52d3-4c78-84b7-7715d239e1bd-registry-certificates\") pod \"image-registry-cb8684c9b-nckr9\" (UID: \"44b85e98-52d3-4c78-84b7-7715d239e1bd\") " pod="openshift-image-registry/image-registry-cb8684c9b-nckr9" Jul 10 17:12:12.760326 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.760254 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/44b85e98-52d3-4c78-84b7-7715d239e1bd-registry-tls\") pod \"image-registry-cb8684c9b-nckr9\" (UID: \"44b85e98-52d3-4c78-84b7-7715d239e1bd\") " pod="openshift-image-registry/image-registry-cb8684c9b-nckr9" Jul 10 17:12:12.761946 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.761929 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c458bf5b-vnwqr" Jul 10 17:12:12.766965 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.766951 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-774457d448-5pmtf" Jul 10 17:12:12.861237 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.861207 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/44b85e98-52d3-4c78-84b7-7715d239e1bd-trusted-ca\") pod \"image-registry-cb8684c9b-nckr9\" (UID: \"44b85e98-52d3-4c78-84b7-7715d239e1bd\") " pod="openshift-image-registry/image-registry-cb8684c9b-nckr9" Jul 10 17:12:12.861353 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.861263 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/44b85e98-52d3-4c78-84b7-7715d239e1bd-ca-trust-extracted\") pod \"image-registry-cb8684c9b-nckr9\" (UID: \"44b85e98-52d3-4c78-84b7-7715d239e1bd\") " pod="openshift-image-registry/image-registry-cb8684c9b-nckr9" Jul 10 17:12:12.861353 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.861296 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/44b85e98-52d3-4c78-84b7-7715d239e1bd-installation-pull-secrets\") pod \"image-registry-cb8684c9b-nckr9\" (UID: \"44b85e98-52d3-4c78-84b7-7715d239e1bd\") " pod="openshift-image-registry/image-registry-cb8684c9b-nckr9" Jul 10 17:12:12.861353 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.861345 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/44b85e98-52d3-4c78-84b7-7715d239e1bd-image-registry-private-configuration\") pod \"image-registry-cb8684c9b-nckr9\" (UID: \"44b85e98-52d3-4c78-84b7-7715d239e1bd\") " pod="openshift-image-registry/image-registry-cb8684c9b-nckr9" Jul 10 17:12:12.861563 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.861414 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vmhq\" (UniqueName: \"kubernetes.io/projected/44b85e98-52d3-4c78-84b7-7715d239e1bd-kube-api-access-5vmhq\") pod \"image-registry-cb8684c9b-nckr9\" (UID: \"44b85e98-52d3-4c78-84b7-7715d239e1bd\") " pod="openshift-image-registry/image-registry-cb8684c9b-nckr9" Jul 10 17:12:12.861563 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.861455 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44b85e98-52d3-4c78-84b7-7715d239e1bd-bound-sa-token\") pod \"image-registry-cb8684c9b-nckr9\" (UID: \"44b85e98-52d3-4c78-84b7-7715d239e1bd\") " pod="openshift-image-registry/image-registry-cb8684c9b-nckr9" Jul 10 17:12:12.861563 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.861486 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/44b85e98-52d3-4c78-84b7-7715d239e1bd-registry-certificates\") pod \"image-registry-cb8684c9b-nckr9\" (UID: \"44b85e98-52d3-4c78-84b7-7715d239e1bd\") " pod="openshift-image-registry/image-registry-cb8684c9b-nckr9" Jul 10 17:12:12.861563 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.861553 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/44b85e98-52d3-4c78-84b7-7715d239e1bd-registry-tls\") pod \"image-registry-cb8684c9b-nckr9\" (UID: \"44b85e98-52d3-4c78-84b7-7715d239e1bd\") " pod="openshift-image-registry/image-registry-cb8684c9b-nckr9" Jul 10 17:12:12.861811 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.861721 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/44b85e98-52d3-4c78-84b7-7715d239e1bd-ca-trust-extracted\") pod \"image-registry-cb8684c9b-nckr9\" (UID: \"44b85e98-52d3-4c78-84b7-7715d239e1bd\") " pod="openshift-image-registry/image-registry-cb8684c9b-nckr9" Jul 10 17:12:12.864859 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.864791 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/44b85e98-52d3-4c78-84b7-7715d239e1bd-registry-tls\") pod \"image-registry-cb8684c9b-nckr9\" (UID: \"44b85e98-52d3-4c78-84b7-7715d239e1bd\") " pod="openshift-image-registry/image-registry-cb8684c9b-nckr9" Jul 10 17:12:12.864958 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.864859 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/44b85e98-52d3-4c78-84b7-7715d239e1bd-trusted-ca\") pod \"image-registry-cb8684c9b-nckr9\" (UID: \"44b85e98-52d3-4c78-84b7-7715d239e1bd\") " pod="openshift-image-registry/image-registry-cb8684c9b-nckr9" Jul 10 17:12:12.864958 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.864896 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/44b85e98-52d3-4c78-84b7-7715d239e1bd-installation-pull-secrets\") pod \"image-registry-cb8684c9b-nckr9\" (UID: \"44b85e98-52d3-4c78-84b7-7715d239e1bd\") " pod="openshift-image-registry/image-registry-cb8684c9b-nckr9" Jul 10 17:12:12.865060 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.864984 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/44b85e98-52d3-4c78-84b7-7715d239e1bd-registry-certificates\") pod \"image-registry-cb8684c9b-nckr9\" (UID: \"44b85e98-52d3-4c78-84b7-7715d239e1bd\") " pod="openshift-image-registry/image-registry-cb8684c9b-nckr9" Jul 10 17:12:12.865060 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.865012 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/44b85e98-52d3-4c78-84b7-7715d239e1bd-image-registry-private-configuration\") pod \"image-registry-cb8684c9b-nckr9\" (UID: \"44b85e98-52d3-4c78-84b7-7715d239e1bd\") " pod="openshift-image-registry/image-registry-cb8684c9b-nckr9" Jul 10 17:12:12.878445 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.878421 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44b85e98-52d3-4c78-84b7-7715d239e1bd-bound-sa-token\") pod \"image-registry-cb8684c9b-nckr9\" (UID: \"44b85e98-52d3-4c78-84b7-7715d239e1bd\") " pod="openshift-image-registry/image-registry-cb8684c9b-nckr9" Jul 10 17:12:12.878670 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.878654 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vmhq\" (UniqueName: \"kubernetes.io/projected/44b85e98-52d3-4c78-84b7-7715d239e1bd-kube-api-access-5vmhq\") pod \"image-registry-cb8684c9b-nckr9\" (UID: \"44b85e98-52d3-4c78-84b7-7715d239e1bd\") " pod="openshift-image-registry/image-registry-cb8684c9b-nckr9" Jul 10 17:12:12.902209 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.902184 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c458bf5b-vnwqr"] Jul 10 17:12:12.905602 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:12:12.905579 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda62d79fa_d075_459b_be69_c6d799c01c9c.slice/crio-a9fec3747100aff102188ad596a329f93b024a4f93241baf6c343a6eeca64df4 WatchSource:0}: Error finding container a9fec3747100aff102188ad596a329f93b024a4f93241baf6c343a6eeca64df4: Status 404 returned error can't find the container with id a9fec3747100aff102188ad596a329f93b024a4f93241baf6c343a6eeca64df4 Jul 10 17:12:12.922205 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.922188 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-774457d448-5pmtf"] Jul 10 17:12:12.923658 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:12:12.923637 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37baf233_9a94_4ae8_8a44_c0336794c5b7.slice/crio-5b2556a1263166a52d53ef83f7e4af798d01c40537f65fc82f922dd3b213759b WatchSource:0}: Error finding container 5b2556a1263166a52d53ef83f7e4af798d01c40537f65fc82f922dd3b213759b: Status 404 returned error can't find the container with id 5b2556a1263166a52d53ef83f7e4af798d01c40537f65fc82f922dd3b213759b Jul 10 17:12:12.959967 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:12.959917 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cb8684c9b-nckr9" Jul 10 17:12:13.092850 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:13.092830 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cb8684c9b-nckr9"] Jul 10 17:12:13.094775 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:12:13.094752 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b85e98_52d3_4c78_84b7_7715d239e1bd.slice/crio-f8d77d914475a1aac791a2be6b68a27ef0a0c50816a8fee901f70f33b8fbd194 WatchSource:0}: Error finding container f8d77d914475a1aac791a2be6b68a27ef0a0c50816a8fee901f70f33b8fbd194: Status 404 returned error can't find the container with id f8d77d914475a1aac791a2be6b68a27ef0a0c50816a8fee901f70f33b8fbd194 Jul 10 17:12:13.561145 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:13.561099 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c458bf5b-vnwqr" event={"ID":"a62d79fa-d075-459b-be69-c6d799c01c9c","Type":"ContainerStarted","Data":"a9fec3747100aff102188ad596a329f93b024a4f93241baf6c343a6eeca64df4"} Jul 10 17:12:13.563142 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:13.563110 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cb8684c9b-nckr9" event={"ID":"44b85e98-52d3-4c78-84b7-7715d239e1bd","Type":"ContainerStarted","Data":"d23cd395e8abcefa514c0c0ced3239516c1ab76399f97e608b318ad2cd32fb69"} Jul 10 17:12:13.563264 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:13.563166 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cb8684c9b-nckr9" event={"ID":"44b85e98-52d3-4c78-84b7-7715d239e1bd","Type":"ContainerStarted","Data":"f8d77d914475a1aac791a2be6b68a27ef0a0c50816a8fee901f70f33b8fbd194"} Jul 10 17:12:13.563903 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:13.563879 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-cb8684c9b-nckr9" Jul 10 17:12:13.565291 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:13.565261 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-774457d448-5pmtf" event={"ID":"37baf233-9a94-4ae8-8a44-c0336794c5b7","Type":"ContainerStarted","Data":"5b2556a1263166a52d53ef83f7e4af798d01c40537f65fc82f922dd3b213759b"} Jul 10 17:12:13.591244 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:13.591203 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-cb8684c9b-nckr9" podStartSLOduration=1.59118901 podStartE2EDuration="1.59118901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:12:13.590339954 +0000 UTC m=+65.918037496" watchObservedRunningTime="2026-07-10 17:12:13.59118901 +0000 UTC m=+65.918886529" Jul 10 17:12:13.971965 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:13.971926 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsl5d\" (UniqueName: \"kubernetes.io/projected/7000764e-d442-4c90-aa00-97b83a65e4b6-kube-api-access-rsl5d\") pod \"network-check-target-7mnkc\" (UID: \"7000764e-d442-4c90-aa00-97b83a65e4b6\") " pod="openshift-network-diagnostics/network-check-target-7mnkc" Jul 10 17:12:13.972131 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:13.971979 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea2e6b67-bf39-4196-a657-7859b8b73199-original-pull-secret\") pod \"global-pull-secret-syncer-smz4h\" (UID: \"ea2e6b67-bf39-4196-a657-7859b8b73199\") " pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:12:13.972131 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:13.972026 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d177790b-fe2d-469e-9dff-c653997fe096-metrics-certs\") pod \"network-metrics-daemon-h9jx7\" (UID: \"d177790b-fe2d-469e-9dff-c653997fe096\") " pod="openshift-multus/network-metrics-daemon-h9jx7" Jul 10 17:12:13.974898 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:13.974872 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 17:12:13.975172 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:13.975154 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 17:12:13.975424 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:13.975405 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 17:12:13.985221 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:13.985200 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 17:12:13.992710 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:13.992511 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d177790b-fe2d-469e-9dff-c653997fe096-metrics-certs\") pod \"network-metrics-daemon-h9jx7\" (UID: \"d177790b-fe2d-469e-9dff-c653997fe096\") " pod="openshift-multus/network-metrics-daemon-h9jx7" Jul 10 17:12:13.992828 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:13.992413 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea2e6b67-bf39-4196-a657-7859b8b73199-original-pull-secret\") pod \"global-pull-secret-syncer-smz4h\" (UID: \"ea2e6b67-bf39-4196-a657-7859b8b73199\") " pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:12:14.007886 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:14.007833 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsl5d\" (UniqueName: \"kubernetes.io/projected/7000764e-d442-4c90-aa00-97b83a65e4b6-kube-api-access-rsl5d\") pod \"network-check-target-7mnkc\" (UID: \"7000764e-d442-4c90-aa00-97b83a65e4b6\") " pod="openshift-network-diagnostics/network-check-target-7mnkc" Jul 10 17:12:14.116075 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:14.116044 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rwncl\"" Jul 10 17:12:14.121319 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:14.121295 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smz4h" Jul 10 17:12:14.124204 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:14.123861 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9jx7" Jul 10 17:12:14.128380 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:14.128185 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-r2699\"" Jul 10 17:12:14.136782 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:14.135961 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7mnkc" Jul 10 17:12:14.312984 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:14.312949 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h9jx7"] Jul 10 17:12:14.346864 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:14.346835 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-smz4h"] Jul 10 17:12:14.371093 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:14.371000 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7mnkc"] Jul 10 17:12:14.374783 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:12:14.374746 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7000764e_d442_4c90_aa00_97b83a65e4b6.slice/crio-7b36d2390b7977b255c935c99c03278771cbd804a8c1e5dacdd0ecf89f2213c9 WatchSource:0}: Error finding container 7b36d2390b7977b255c935c99c03278771cbd804a8c1e5dacdd0ecf89f2213c9: Status 404 returned error can't find the container with id 7b36d2390b7977b255c935c99c03278771cbd804a8c1e5dacdd0ecf89f2213c9 Jul 10 17:12:14.571038 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:14.570950 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-smz4h" event={"ID":"ea2e6b67-bf39-4196-a657-7859b8b73199","Type":"ContainerStarted","Data":"9d87a94d575bf58afb3b95102f67de9c5b64bf0c5d14881fde6dd9228db98f3b"} Jul 10 17:12:14.572588 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:14.572541 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7mnkc" event={"ID":"7000764e-d442-4c90-aa00-97b83a65e4b6","Type":"ContainerStarted","Data":"7b36d2390b7977b255c935c99c03278771cbd804a8c1e5dacdd0ecf89f2213c9"} Jul 10 17:12:14.574024 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:14.573962 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h9jx7" event={"ID":"d177790b-fe2d-469e-9dff-c653997fe096","Type":"ContainerStarted","Data":"dad5041f4f9fbb2ca37eb095ab9c79b1db87650ee8999c16e1106b6f398b4401"} Jul 10 17:12:15.888673 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:15.888627 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29817af8-c839-459c-a69d-733e4926b16b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fvjjb\" (UID: \"29817af8-c839-459c-a69d-733e4926b16b\") " pod="openshift-insights/insights-runtime-extractor-fvjjb" Jul 10 17:12:15.891874 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:15.891848 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29817af8-c839-459c-a69d-733e4926b16b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fvjjb\" (UID: \"29817af8-c839-459c-a69d-733e4926b16b\") " pod="openshift-insights/insights-runtime-extractor-fvjjb" Jul 10 17:12:16.153764 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:16.153680 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-strpp\"" Jul 10 17:12:16.161805 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:16.161771 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fvjjb" Jul 10 17:12:20.879961 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:20.879935 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fvjjb"] Jul 10 17:12:20.887555 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:12:20.887500 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29817af8_c839_459c_a69d_733e4926b16b.slice/crio-94202aff1775b6e4273f26ef4c69464b3542010d8af623d5e2ed58a577f42e1f WatchSource:0}: Error finding container 94202aff1775b6e4273f26ef4c69464b3542010d8af623d5e2ed58a577f42e1f: Status 404 returned error can't find the container with id 94202aff1775b6e4273f26ef4c69464b3542010d8af623d5e2ed58a577f42e1f Jul 10 17:12:21.604344 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:21.604088 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-774457d448-5pmtf" event={"ID":"37baf233-9a94-4ae8-8a44-c0336794c5b7","Type":"ContainerStarted","Data":"2586c04f79647247b6854c87532e8baa260ea9d87333705936a449fd79d01e52"} Jul 10 17:12:21.607269 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:21.607235 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7mnkc" event={"ID":"7000764e-d442-4c90-aa00-97b83a65e4b6","Type":"ContainerStarted","Data":"f24b2f70d287a565388d99dcb5496f2f5ad28c74083dcb06bf58e148d9e0ee02"} Jul 10 17:12:21.607793 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:21.607770 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7mnkc" Jul 10 17:12:21.621207 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:21.621169 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fvjjb" event={"ID":"29817af8-c839-459c-a69d-733e4926b16b","Type":"ContainerStarted","Data":"94202aff1775b6e4273f26ef4c69464b3542010d8af623d5e2ed58a577f42e1f"} Jul 10 17:12:21.625933 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:21.625906 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h9jx7" event={"ID":"d177790b-fe2d-469e-9dff-c653997fe096","Type":"ContainerStarted","Data":"3fa1a63e3e4e48ed5d26c99bc8edd1e08b4351e09fa904e9dc5fc5bcc35a739e"} Jul 10 17:12:21.626733 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:21.626714 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h9jx7" event={"ID":"d177790b-fe2d-469e-9dff-c653997fe096","Type":"ContainerStarted","Data":"525a29b2ba6d7f1c43edcae0b0acc91d7c64e9c16e83f1c922424abe6ff241e8"} Jul 10 17:12:21.629759 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:21.629625 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c458bf5b-vnwqr" event={"ID":"a62d79fa-d075-459b-be69-c6d799c01c9c","Type":"ContainerStarted","Data":"2e7a24a29d15760b2e3c485ba7e986b7b0af41e95b3d42d7f141a1c0b97c838a"} Jul 10 17:12:21.629759 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:21.629664 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c458bf5b-vnwqr" event={"ID":"a62d79fa-d075-459b-be69-c6d799c01c9c","Type":"ContainerStarted","Data":"0e58c17d214619a196751afb5c45144947449c7a616300f3cb582a179f8cb4c1"} Jul 10 17:12:21.629759 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:21.629683 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c458bf5b-vnwqr" event={"ID":"a62d79fa-d075-459b-be69-c6d799c01c9c","Type":"ContainerStarted","Data":"8bf4c016620acb4d53f799742337b13fc8935ea5be203a9c0cd12e526bbc28ec"} Jul 10 17:12:21.633125 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:21.632723 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-smz4h" event={"ID":"ea2e6b67-bf39-4196-a657-7859b8b73199","Type":"ContainerStarted","Data":"271dc6a05bf76e059fdb63d2bf72f93485e8c2d3787c0dbfd2ce8a72f4c945bd"} Jul 10 17:12:21.665187 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:21.665123 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7mnkc" podStartSLOduration=67.310758186 podStartE2EDuration="1m13.665100399s" totalImagesPullingTime="6.354342213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:12:21.664784145 +0000 UTC m=+73.992481685" watchObservedRunningTime="2026-07-10 17:12:21.665100399 +0000 UTC m=+73.992797942" Jul 10 17:12:21.665347 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:21.665218 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-774457d448-5pmtf" podStartSLOduration=1.883892086 podStartE2EDuration="9.665210395s" totalImagesPullingTime="7.781318309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:12:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:12:21.623995347 +0000 UTC m=+73.951692887" watchObservedRunningTime="2026-07-10 17:12:21.665210395 +0000 UTC m=+73.992907934" Jul 10 17:12:21.721236 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:21.721173 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-smz4h" podStartSLOduration=67.345462632 podStartE2EDuration="1m13.721124983s" totalImagesPullingTime="6.375662351s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:12:21.694235677 +0000 UTC m=+74.021933210" watchObservedRunningTime="2026-07-10 17:12:21.721124983 +0000 UTC m=+74.048822530" Jul 10 17:12:21.722561 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:21.722515 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-h9jx7" podStartSLOduration=67.337014354 podStartE2EDuration="1m13.722502809s" totalImagesPullingTime="6.385488455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:12:21.720521318 +0000 UTC m=+74.048218858" watchObservedRunningTime="2026-07-10 17:12:21.722502809 +0000 UTC m=+74.050200349" Jul 10 17:12:21.755382 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:21.755311 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c458bf5b-vnwqr" podStartSLOduration=1.955905848 podStartE2EDuration="9.755286982s" totalImagesPullingTime="7.799381134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:12:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:12:21.753411922 +0000 UTC m=+74.081109453" watchObservedRunningTime="2026-07-10 17:12:21.755286982 +0000 UTC m=+74.082984521" Jul 10 17:12:22.490635 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:22.490607 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:12:25.650958 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:25.650866 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fvjjb" event={"ID":"29817af8-c839-459c-a69d-733e4926b16b","Type":"ContainerStarted","Data":"3a221aa2ee64e07a79e12a9634d41d1dab61d655750f11d5041a06497c213309"} Jul 10 17:12:25.650958 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:25.650933 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fvjjb" event={"ID":"29817af8-c839-459c-a69d-733e4926b16b","Type":"ContainerStarted","Data":"0f3055be6731a43b24cb7cdd9899a83778e6127c53de3d90b325ebe3d91daed5"} Jul 10 17:12:28.663744 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:28.663712 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fvjjb" event={"ID":"29817af8-c839-459c-a69d-733e4926b16b","Type":"ContainerStarted","Data":"89d4815f6f414c1f48cf1825311e96f97ba4cbf1b9322bbc35e81ae6211af9da"} Jul 10 17:12:28.689540 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:28.689489 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-fvjjb" podStartSLOduration=37.956087017 podStartE2EDuration="44.689472107s" totalImagesPullingTime="6.73338509s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:11:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:12:28.688068959 +0000 UTC m=+81.015766508" watchObservedRunningTime="2026-07-10 17:12:28.689472107 +0000 UTC m=+81.017169648" Jul 10 17:12:28.885529 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:28.885409 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vgp2l"] Jul 10 17:12:28.888311 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:28.888295 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vgp2l" Jul 10 17:12:28.891205 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:28.891183 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ngsl6\"" Jul 10 17:12:28.891480 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:28.891455 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 17:12:28.891718 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:28.891705 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 17:12:28.891768 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:28.891704 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 17:12:28.895292 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:28.895162 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 17:12:28.895546 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:28.895530 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 17:12:28.895624 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:28.895549 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 17:12:29.022151 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:29.022069 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f45c7ca-5e81-4c97-8b49-6efa2419024e-sys\") pod \"node-exporter-vgp2l\" (UID: \"5f45c7ca-5e81-4c97-8b49-6efa2419024e\") " pod="openshift-monitoring/node-exporter-vgp2l" Jul 10 17:12:29.022151 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:29.022109 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f45c7ca-5e81-4c97-8b49-6efa2419024e-metrics-client-ca\") pod \"node-exporter-vgp2l\" (UID: \"5f45c7ca-5e81-4c97-8b49-6efa2419024e\") " pod="openshift-monitoring/node-exporter-vgp2l" Jul 10 17:12:29.022151 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:29.022140 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5f45c7ca-5e81-4c97-8b49-6efa2419024e-root\") pod \"node-exporter-vgp2l\" (UID: \"5f45c7ca-5e81-4c97-8b49-6efa2419024e\") " pod="openshift-monitoring/node-exporter-vgp2l" Jul 10 17:12:29.022349 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:29.022163 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5f45c7ca-5e81-4c97-8b49-6efa2419024e-node-exporter-wtmp\") pod \"node-exporter-vgp2l\" (UID: \"5f45c7ca-5e81-4c97-8b49-6efa2419024e\") " pod="openshift-monitoring/node-exporter-vgp2l" Jul 10 17:12:29.022349 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:29.022192 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5f45c7ca-5e81-4c97-8b49-6efa2419024e-node-exporter-accelerators-collector-config\") pod \"node-exporter-vgp2l\" (UID: \"5f45c7ca-5e81-4c97-8b49-6efa2419024e\") " pod="openshift-monitoring/node-exporter-vgp2l" Jul 10 17:12:29.022349 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:29.022217 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5f45c7ca-5e81-4c97-8b49-6efa2419024e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vgp2l\" (UID: \"5f45c7ca-5e81-4c97-8b49-6efa2419024e\") " pod="openshift-monitoring/node-exporter-vgp2l" Jul 10 17:12:29.022349 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:29.022233 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fttw8\" (UniqueName: \"kubernetes.io/projected/5f45c7ca-5e81-4c97-8b49-6efa2419024e-kube-api-access-fttw8\") pod \"node-exporter-vgp2l\" (UID: \"5f45c7ca-5e81-4c97-8b49-6efa2419024e\") " pod="openshift-monitoring/node-exporter-vgp2l" Jul 10 17:12:29.022349 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:29.022302 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5f45c7ca-5e81-4c97-8b49-6efa2419024e-node-exporter-textfile\") pod \"node-exporter-vgp2l\" (UID: \"5f45c7ca-5e81-4c97-8b49-6efa2419024e\") " pod="openshift-monitoring/node-exporter-vgp2l" Jul 10 17:12:29.022349 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:29.022325 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5f45c7ca-5e81-4c97-8b49-6efa2419024e-node-exporter-tls\") pod \"node-exporter-vgp2l\" (UID: \"5f45c7ca-5e81-4c97-8b49-6efa2419024e\") " pod="openshift-monitoring/node-exporter-vgp2l" Jul 10 17:12:29.123350 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:29.123322 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5f45c7ca-5e81-4c97-8b49-6efa2419024e-root\") pod \"node-exporter-vgp2l\" (UID: \"5f45c7ca-5e81-4c97-8b49-6efa2419024e\") " pod="openshift-monitoring/node-exporter-vgp2l" Jul 10 17:12:29.123350 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:29.123354 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5f45c7ca-5e81-4c97-8b49-6efa2419024e-node-exporter-wtmp\") pod \"node-exporter-vgp2l\" (UID: \"5f45c7ca-5e81-4c97-8b49-6efa2419024e\") " pod="openshift-monitoring/node-exporter-vgp2l" Jul 10 17:12:29.123588 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:29.123383 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5f45c7ca-5e81-4c97-8b49-6efa2419024e-node-exporter-accelerators-collector-config\") pod \"node-exporter-vgp2l\" (UID: \"5f45c7ca-5e81-4c97-8b49-6efa2419024e\") " pod="openshift-monitoring/node-exporter-vgp2l" Jul 10 17:12:29.123588 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:29.123442 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5f45c7ca-5e81-4c97-8b49-6efa2419024e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vgp2l\" (UID: \"5f45c7ca-5e81-4c97-8b49-6efa2419024e\") " pod="openshift-monitoring/node-exporter-vgp2l" Jul 10 17:12:29.123588 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:29.123468 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5f45c7ca-5e81-4c97-8b49-6efa2419024e-root\") pod \"node-exporter-vgp2l\" (UID: \"5f45c7ca-5e81-4c97-8b49-6efa2419024e\") " pod="openshift-monitoring/node-exporter-vgp2l" Jul 10 17:12:29.123588 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:29.123470 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fttw8\" (UniqueName: \"kubernetes.io/projected/5f45c7ca-5e81-4c97-8b49-6efa2419024e-kube-api-access-fttw8\") pod \"node-exporter-vgp2l\" (UID: \"5f45c7ca-5e81-4c97-8b49-6efa2419024e\") " pod="openshift-monitoring/node-exporter-vgp2l" Jul 10 17:12:29.123588 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:29.123541 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5f45c7ca-5e81-4c97-8b49-6efa2419024e-node-exporter-wtmp\") pod \"node-exporter-vgp2l\" (UID: \"5f45c7ca-5e81-4c97-8b49-6efa2419024e\") " pod="openshift-monitoring/node-exporter-vgp2l" Jul 10 17:12:29.123588 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:29.123582 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5f45c7ca-5e81-4c97-8b49-6efa2419024e-node-exporter-textfile\") pod \"node-exporter-vgp2l\" (UID: \"5f45c7ca-5e81-4c97-8b49-6efa2419024e\") " pod="openshift-monitoring/node-exporter-vgp2l" Jul 10 17:12:29.123875 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:29.123609 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5f45c7ca-5e81-4c97-8b49-6efa2419024e-node-exporter-tls\") pod \"node-exporter-vgp2l\" (UID: \"5f45c7ca-5e81-4c97-8b49-6efa2419024e\") " pod="openshift-monitoring/node-exporter-vgp2l" Jul 10 17:12:29.123875 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:29.123648 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f45c7ca-5e81-4c97-8b49-6efa2419024e-sys\") pod \"node-exporter-vgp2l\" (UID: \"5f45c7ca-5e81-4c97-8b49-6efa2419024e\") " pod="openshift-monitoring/node-exporter-vgp2l" Jul 10 17:12:29.123875 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:29.123676 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f45c7ca-5e81-4c97-8b49-6efa2419024e-metrics-client-ca\") pod \"node-exporter-vgp2l\" (UID: \"5f45c7ca-5e81-4c97-8b49-6efa2419024e\") " pod="openshift-monitoring/node-exporter-vgp2l" Jul 10 17:12:29.124020 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:29.123878 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f45c7ca-5e81-4c97-8b49-6efa2419024e-sys\") pod \"node-exporter-vgp2l\" (UID: \"5f45c7ca-5e81-4c97-8b49-6efa2419024e\") " pod="openshift-monitoring/node-exporter-vgp2l" Jul 10 17:12:29.124162 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:29.124137 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5f45c7ca-5e81-4c97-8b49-6efa2419024e-node-exporter-accelerators-collector-config\") pod \"node-exporter-vgp2l\" (UID: \"5f45c7ca-5e81-4c97-8b49-6efa2419024e\") " pod="openshift-monitoring/node-exporter-vgp2l" Jul 10 17:12:29.124448 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:29.124422 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5f45c7ca-5e81-4c97-8b49-6efa2419024e-node-exporter-textfile\") pod \"node-exporter-vgp2l\" (UID: \"5f45c7ca-5e81-4c97-8b49-6efa2419024e\") " pod="openshift-monitoring/node-exporter-vgp2l" Jul 10 17:12:29.124570 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:29.124546 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f45c7ca-5e81-4c97-8b49-6efa2419024e-metrics-client-ca\") pod \"node-exporter-vgp2l\" (UID: \"5f45c7ca-5e81-4c97-8b49-6efa2419024e\") " pod="openshift-monitoring/node-exporter-vgp2l" Jul 10 17:12:29.126510 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:29.126481 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5f45c7ca-5e81-4c97-8b49-6efa2419024e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vgp2l\" (UID: \"5f45c7ca-5e81-4c97-8b49-6efa2419024e\") " pod="openshift-monitoring/node-exporter-vgp2l" Jul 10 17:12:29.126736 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:29.126711 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5f45c7ca-5e81-4c97-8b49-6efa2419024e-node-exporter-tls\") pod \"node-exporter-vgp2l\" (UID: \"5f45c7ca-5e81-4c97-8b49-6efa2419024e\") " pod="openshift-monitoring/node-exporter-vgp2l" Jul 10 17:12:29.135834 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:29.135796 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fttw8\" (UniqueName: \"kubernetes.io/projected/5f45c7ca-5e81-4c97-8b49-6efa2419024e-kube-api-access-fttw8\") pod \"node-exporter-vgp2l\" (UID: \"5f45c7ca-5e81-4c97-8b49-6efa2419024e\") " pod="openshift-monitoring/node-exporter-vgp2l" Jul 10 17:12:29.199127 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:29.199099 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vgp2l" Jul 10 17:12:29.210584 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:12:29.210558 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f45c7ca_5e81_4c97_8b49_6efa2419024e.slice/crio-7706536383a36f18bfcb8cac9fb3147cd5d60fbb80c10b205b4b99e27670169e WatchSource:0}: Error finding container 7706536383a36f18bfcb8cac9fb3147cd5d60fbb80c10b205b4b99e27670169e: Status 404 returned error can't find the container with id 7706536383a36f18bfcb8cac9fb3147cd5d60fbb80c10b205b4b99e27670169e Jul 10 17:12:29.667646 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:29.667605 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vgp2l" event={"ID":"5f45c7ca-5e81-4c97-8b49-6efa2419024e","Type":"ContainerStarted","Data":"7706536383a36f18bfcb8cac9fb3147cd5d60fbb80c10b205b4b99e27670169e"} Jul 10 17:12:30.673741 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:30.673704 2583 generic.go:356] "Generic (PLEG): container finished" podID="5f45c7ca-5e81-4c97-8b49-6efa2419024e" containerID="7bed69a5949bc1c2281d86fbea82df8453f43562ab90e805cecf438b1860758f" exitCode=0 Jul 10 17:12:30.674140 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:30.673766 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vgp2l" event={"ID":"5f45c7ca-5e81-4c97-8b49-6efa2419024e","Type":"ContainerDied","Data":"7bed69a5949bc1c2281d86fbea82df8453f43562ab90e805cecf438b1860758f"} Jul 10 17:12:31.680076 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:31.680036 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vgp2l" event={"ID":"5f45c7ca-5e81-4c97-8b49-6efa2419024e","Type":"ContainerStarted","Data":"0e446634136349e0ee456d81f0c0c5ffd8e4ff6afe2109008ed8c862348c9113"} Jul 10 17:12:31.680612 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:31.680088 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vgp2l" event={"ID":"5f45c7ca-5e81-4c97-8b49-6efa2419024e","Type":"ContainerStarted","Data":"62645c3a0c6dac04597bb3c179dbfe038ba695c1d318480680c19ed89cbbeb9c"} Jul 10 17:12:31.704822 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:31.704769 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vgp2l" podStartSLOduration=2.59870277 podStartE2EDuration="3.704751877s" totalImagesPullingTime="1.035263209s" totalInitContainerRuntime="70.785898ms" isStatefulPod=true podCreationTimestamp="2026-07-10 17:12:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:12:31.703420241 +0000 UTC m=+84.031117774" watchObservedRunningTime="2026-07-10 17:12:31.704751877 +0000 UTC m=+84.032449466" Jul 10 17:12:35.583011 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:35.582983 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-cb8684c9b-nckr9" Jul 10 17:12:37.505371 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:37.505335 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" podUID="e973aa85-c8e0-457a-bb77-543b9e5180d8" containerName="registry" containerID="cri-o://be93ca7d8403e0ac6676a8a944628457cafcf5c0bf2c4fab139c46d1acad4470" gracePeriod=30 Jul 10 17:12:37.706542 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:37.706508 2583 generic.go:356] "Generic (PLEG): container finished" podID="e973aa85-c8e0-457a-bb77-543b9e5180d8" containerID="be93ca7d8403e0ac6676a8a944628457cafcf5c0bf2c4fab139c46d1acad4470" exitCode=0 Jul 10 17:12:37.706673 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:37.706587 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" event={"ID":"e973aa85-c8e0-457a-bb77-543b9e5180d8","Type":"ContainerDied","Data":"be93ca7d8403e0ac6676a8a944628457cafcf5c0bf2c4fab139c46d1acad4470"} Jul 10 17:12:37.741681 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:37.741661 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:12:37.887299 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:37.887217 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e973aa85-c8e0-457a-bb77-543b9e5180d8-bound-sa-token\") pod \"e973aa85-c8e0-457a-bb77-543b9e5180d8\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " Jul 10 17:12:37.887299 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:37.887267 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e973aa85-c8e0-457a-bb77-543b9e5180d8-ca-trust-extracted\") pod \"e973aa85-c8e0-457a-bb77-543b9e5180d8\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " Jul 10 17:12:37.887299 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:37.887287 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e973aa85-c8e0-457a-bb77-543b9e5180d8-image-registry-private-configuration\") pod \"e973aa85-c8e0-457a-bb77-543b9e5180d8\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " Jul 10 17:12:37.887525 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:37.887428 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e973aa85-c8e0-457a-bb77-543b9e5180d8-trusted-ca\") pod \"e973aa85-c8e0-457a-bb77-543b9e5180d8\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " Jul 10 17:12:37.887525 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:37.887483 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bb7j4\" (UniqueName: \"kubernetes.io/projected/e973aa85-c8e0-457a-bb77-543b9e5180d8-kube-api-access-bb7j4\") pod \"e973aa85-c8e0-457a-bb77-543b9e5180d8\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " Jul 10 17:12:37.887600 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:37.887532 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e973aa85-c8e0-457a-bb77-543b9e5180d8-registry-tls\") pod \"e973aa85-c8e0-457a-bb77-543b9e5180d8\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " Jul 10 17:12:37.887600 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:37.887563 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e973aa85-c8e0-457a-bb77-543b9e5180d8-installation-pull-secrets\") pod \"e973aa85-c8e0-457a-bb77-543b9e5180d8\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " Jul 10 17:12:37.887691 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:37.887612 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e973aa85-c8e0-457a-bb77-543b9e5180d8-registry-certificates\") pod \"e973aa85-c8e0-457a-bb77-543b9e5180d8\" (UID: \"e973aa85-c8e0-457a-bb77-543b9e5180d8\") " Jul 10 17:12:37.888023 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:37.887847 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e973aa85-c8e0-457a-bb77-543b9e5180d8-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e973aa85-c8e0-457a-bb77-543b9e5180d8" (UID: "e973aa85-c8e0-457a-bb77-543b9e5180d8"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:12:37.888412 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:37.888367 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e973aa85-c8e0-457a-bb77-543b9e5180d8-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e973aa85-c8e0-457a-bb77-543b9e5180d8" (UID: "e973aa85-c8e0-457a-bb77-543b9e5180d8"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:12:37.889872 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:37.889845 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e973aa85-c8e0-457a-bb77-543b9e5180d8-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e973aa85-c8e0-457a-bb77-543b9e5180d8" (UID: "e973aa85-c8e0-457a-bb77-543b9e5180d8"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:12:37.889967 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:37.889870 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e973aa85-c8e0-457a-bb77-543b9e5180d8-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e973aa85-c8e0-457a-bb77-543b9e5180d8" (UID: "e973aa85-c8e0-457a-bb77-543b9e5180d8"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:12:37.890120 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:37.890104 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e973aa85-c8e0-457a-bb77-543b9e5180d8-kube-api-access-bb7j4" (OuterVolumeSpecName: "kube-api-access-bb7j4") pod "e973aa85-c8e0-457a-bb77-543b9e5180d8" (UID: "e973aa85-c8e0-457a-bb77-543b9e5180d8"). InnerVolumeSpecName "kube-api-access-bb7j4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:12:37.890432 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:37.890406 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e973aa85-c8e0-457a-bb77-543b9e5180d8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e973aa85-c8e0-457a-bb77-543b9e5180d8" (UID: "e973aa85-c8e0-457a-bb77-543b9e5180d8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:12:37.890512 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:37.890409 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e973aa85-c8e0-457a-bb77-543b9e5180d8-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e973aa85-c8e0-457a-bb77-543b9e5180d8" (UID: "e973aa85-c8e0-457a-bb77-543b9e5180d8"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:12:37.896120 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:37.896099 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e973aa85-c8e0-457a-bb77-543b9e5180d8-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e973aa85-c8e0-457a-bb77-543b9e5180d8" (UID: "e973aa85-c8e0-457a-bb77-543b9e5180d8"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:12:37.989014 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:37.988982 2583 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e973aa85-c8e0-457a-bb77-543b9e5180d8-registry-tls\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:12:37.989014 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:37.989012 2583 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e973aa85-c8e0-457a-bb77-543b9e5180d8-installation-pull-secrets\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:12:37.989014 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:37.989023 2583 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e973aa85-c8e0-457a-bb77-543b9e5180d8-registry-certificates\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:12:37.989179 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:37.989031 2583 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e973aa85-c8e0-457a-bb77-543b9e5180d8-bound-sa-token\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:12:37.989179 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:37.989039 2583 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e973aa85-c8e0-457a-bb77-543b9e5180d8-ca-trust-extracted\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:12:37.989179 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:37.989047 2583 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e973aa85-c8e0-457a-bb77-543b9e5180d8-image-registry-private-configuration\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:12:37.989179 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:37.989057 2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e973aa85-c8e0-457a-bb77-543b9e5180d8-trusted-ca\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:12:37.989179 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:37.989065 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb7j4\" (UniqueName: \"kubernetes.io/projected/e973aa85-c8e0-457a-bb77-543b9e5180d8-kube-api-access-bb7j4\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:12:38.714735 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:38.714711 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" Jul 10 17:12:38.715171 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:38.714702 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-995c6f9fc-hzwlg" event={"ID":"e973aa85-c8e0-457a-bb77-543b9e5180d8","Type":"ContainerDied","Data":"7db3726ab5f8b43f94fb8b528b213e407e5ed864521572b0b6619d77940a001f"} Jul 10 17:12:38.715171 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:38.714864 2583 scope.go:117] "RemoveContainer" containerID="be93ca7d8403e0ac6676a8a944628457cafcf5c0bf2c4fab139c46d1acad4470" Jul 10 17:12:38.734831 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:38.734804 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-995c6f9fc-hzwlg"] Jul 10 17:12:38.740342 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:38.740322 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-995c6f9fc-hzwlg"] Jul 10 17:12:40.204831 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:40.204801 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e973aa85-c8e0-457a-bb77-543b9e5180d8" path="/var/lib/kubelet/pods/e973aa85-c8e0-457a-bb77-543b9e5180d8/volumes" Jul 10 17:12:53.643620 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:53.643585 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7mnkc" Jul 10 17:12:54.346743 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:12:54.346711 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wr97q_b8edbb95-6b40-46ed-a7ad-0de010dbf2dd/serve-healthcheck-canary/0.log" Jul 10 17:14:42.393502 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:14:42.393460 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jv6sq"] Jul 10 17:14:42.394382 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:14:42.393806 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e973aa85-c8e0-457a-bb77-543b9e5180d8" containerName="registry" Jul 10 17:14:42.394382 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:14:42.393823 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e973aa85-c8e0-457a-bb77-543b9e5180d8" containerName="registry" Jul 10 17:14:42.394382 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:14:42.393895 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e973aa85-c8e0-457a-bb77-543b9e5180d8" containerName="registry" Jul 10 17:14:42.399106 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:14:42.399089 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jv6sq" Jul 10 17:14:42.401771 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:14:42.401751 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 17:14:42.402532 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:14:42.402515 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r9sxv\"" Jul 10 17:14:42.402532 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:14:42.402511 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 17:14:42.405405 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:14:42.405368 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jv6sq"] Jul 10 17:14:42.466240 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:14:42.466208 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2mdc\" (UniqueName: \"kubernetes.io/projected/2c9d3432-3f08-4957-adec-d09276049015-kube-api-access-s2mdc\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jv6sq\" (UID: \"2c9d3432-3f08-4957-adec-d09276049015\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jv6sq" Jul 10 17:14:42.466240 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:14:42.466246 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c9d3432-3f08-4957-adec-d09276049015-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jv6sq\" (UID: \"2c9d3432-3f08-4957-adec-d09276049015\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jv6sq" Jul 10 17:14:42.466466 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:14:42.466338 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c9d3432-3f08-4957-adec-d09276049015-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jv6sq\" (UID: \"2c9d3432-3f08-4957-adec-d09276049015\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jv6sq" Jul 10 17:14:42.567547 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:14:42.567519 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c9d3432-3f08-4957-adec-d09276049015-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jv6sq\" (UID: \"2c9d3432-3f08-4957-adec-d09276049015\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jv6sq" Jul 10 17:14:42.567705 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:14:42.567556 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2mdc\" (UniqueName: \"kubernetes.io/projected/2c9d3432-3f08-4957-adec-d09276049015-kube-api-access-s2mdc\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jv6sq\" (UID: \"2c9d3432-3f08-4957-adec-d09276049015\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jv6sq" Jul 10 17:14:42.567705 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:14:42.567578 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c9d3432-3f08-4957-adec-d09276049015-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jv6sq\" (UID: \"2c9d3432-3f08-4957-adec-d09276049015\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jv6sq" Jul 10 17:14:42.567971 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:14:42.567950 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c9d3432-3f08-4957-adec-d09276049015-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jv6sq\" (UID: \"2c9d3432-3f08-4957-adec-d09276049015\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jv6sq" Jul 10 17:14:42.568010 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:14:42.567963 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c9d3432-3f08-4957-adec-d09276049015-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jv6sq\" (UID: \"2c9d3432-3f08-4957-adec-d09276049015\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jv6sq" Jul 10 17:14:42.577015 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:14:42.576982 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2mdc\" (UniqueName: \"kubernetes.io/projected/2c9d3432-3f08-4957-adec-d09276049015-kube-api-access-s2mdc\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jv6sq\" (UID: \"2c9d3432-3f08-4957-adec-d09276049015\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jv6sq" Jul 10 17:14:42.708839 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:14:42.708801 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jv6sq" Jul 10 17:14:42.833868 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:14:42.833846 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jv6sq"] Jul 10 17:14:42.836071 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:14:42.836044 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c9d3432_3f08_4957_adec_d09276049015.slice/crio-d76c6dbef2781ea2d3623caf47eb99e18b7ea97bc43c357939401594ed6bef30 WatchSource:0}: Error finding container d76c6dbef2781ea2d3623caf47eb99e18b7ea97bc43c357939401594ed6bef30: Status 404 returned error can't find the container with id d76c6dbef2781ea2d3623caf47eb99e18b7ea97bc43c357939401594ed6bef30 Jul 10 17:14:43.136647 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:14:43.136556 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jv6sq" event={"ID":"2c9d3432-3f08-4957-adec-d09276049015","Type":"ContainerStarted","Data":"d76c6dbef2781ea2d3623caf47eb99e18b7ea97bc43c357939401594ed6bef30"} Jul 10 17:14:51.168235 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:14:51.168198 2583 generic.go:356] "Generic (PLEG): container finished" podID="2c9d3432-3f08-4957-adec-d09276049015" containerID="8cb270262f3af2f7877b7affc1a79edf7b1a97bfeb6743d509b1f48a3518b310" exitCode=0 Jul 10 17:14:51.168761 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:14:51.168277 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jv6sq" event={"ID":"2c9d3432-3f08-4957-adec-d09276049015","Type":"ContainerDied","Data":"8cb270262f3af2f7877b7affc1a79edf7b1a97bfeb6743d509b1f48a3518b310"} Jul 10 17:14:53.179286 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:14:53.179245 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jv6sq" event={"ID":"2c9d3432-3f08-4957-adec-d09276049015","Type":"ContainerStarted","Data":"ca1d678e03c52183c7388d5d5b60e825955efc46cc39e697e1d117baa7eb58e9"} Jul 10 17:14:54.185238 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:14:54.185202 2583 generic.go:356] "Generic (PLEG): container finished" podID="2c9d3432-3f08-4957-adec-d09276049015" containerID="ca1d678e03c52183c7388d5d5b60e825955efc46cc39e697e1d117baa7eb58e9" exitCode=0 Jul 10 17:14:54.185651 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:14:54.185273 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jv6sq" event={"ID":"2c9d3432-3f08-4957-adec-d09276049015","Type":"ContainerDied","Data":"ca1d678e03c52183c7388d5d5b60e825955efc46cc39e697e1d117baa7eb58e9"} Jul 10 17:15:03.219413 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:03.219365 2583 generic.go:356] "Generic (PLEG): container finished" podID="2c9d3432-3f08-4957-adec-d09276049015" containerID="e2db8f89f61fad23f23ed8ff734048b5e2fa65380a10d7d35ceca050ff8deb3a" exitCode=0 Jul 10 17:15:03.219811 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:03.219432 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jv6sq" event={"ID":"2c9d3432-3f08-4957-adec-d09276049015","Type":"ContainerDied","Data":"e2db8f89f61fad23f23ed8ff734048b5e2fa65380a10d7d35ceca050ff8deb3a"} Jul 10 17:15:04.368783 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:04.368747 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jv6sq" Jul 10 17:15:04.455894 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:04.455851 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c9d3432-3f08-4957-adec-d09276049015-bundle\") pod \"2c9d3432-3f08-4957-adec-d09276049015\" (UID: \"2c9d3432-3f08-4957-adec-d09276049015\") " Jul 10 17:15:04.456145 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:04.455953 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2mdc\" (UniqueName: \"kubernetes.io/projected/2c9d3432-3f08-4957-adec-d09276049015-kube-api-access-s2mdc\") pod \"2c9d3432-3f08-4957-adec-d09276049015\" (UID: \"2c9d3432-3f08-4957-adec-d09276049015\") " Jul 10 17:15:04.456145 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:04.455982 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c9d3432-3f08-4957-adec-d09276049015-util\") pod \"2c9d3432-3f08-4957-adec-d09276049015\" (UID: \"2c9d3432-3f08-4957-adec-d09276049015\") " Jul 10 17:15:04.456658 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:04.456632 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c9d3432-3f08-4957-adec-d09276049015-bundle" (OuterVolumeSpecName: "bundle") pod "2c9d3432-3f08-4957-adec-d09276049015" (UID: "2c9d3432-3f08-4957-adec-d09276049015"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:15:04.458335 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:04.458312 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c9d3432-3f08-4957-adec-d09276049015-kube-api-access-s2mdc" (OuterVolumeSpecName: "kube-api-access-s2mdc") pod "2c9d3432-3f08-4957-adec-d09276049015" (UID: "2c9d3432-3f08-4957-adec-d09276049015"). InnerVolumeSpecName "kube-api-access-s2mdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:15:04.460333 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:04.460308 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c9d3432-3f08-4957-adec-d09276049015-util" (OuterVolumeSpecName: "util") pod "2c9d3432-3f08-4957-adec-d09276049015" (UID: "2c9d3432-3f08-4957-adec-d09276049015"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:15:04.557167 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:04.557082 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c9d3432-3f08-4957-adec-d09276049015-bundle\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:15:04.557167 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:04.557115 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2mdc\" (UniqueName: \"kubernetes.io/projected/2c9d3432-3f08-4957-adec-d09276049015-kube-api-access-s2mdc\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:15:04.557167 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:04.557127 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c9d3432-3f08-4957-adec-d09276049015-util\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:15:05.228545 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:05.228512 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jv6sq" event={"ID":"2c9d3432-3f08-4957-adec-d09276049015","Type":"ContainerDied","Data":"d76c6dbef2781ea2d3623caf47eb99e18b7ea97bc43c357939401594ed6bef30"} Jul 10 17:15:05.228545 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:05.228543 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jv6sq" Jul 10 17:15:05.228545 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:05.228556 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d76c6dbef2781ea2d3623caf47eb99e18b7ea97bc43c357939401594ed6bef30" Jul 10 17:15:14.852375 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:14.852340 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqcbdl"] Jul 10 17:15:14.852781 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:14.852579 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c9d3432-3f08-4957-adec-d09276049015" containerName="pull" Jul 10 17:15:14.852781 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:14.852590 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c9d3432-3f08-4957-adec-d09276049015" containerName="pull" Jul 10 17:15:14.852781 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:14.852607 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c9d3432-3f08-4957-adec-d09276049015" containerName="extract" Jul 10 17:15:14.852781 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:14.852612 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c9d3432-3f08-4957-adec-d09276049015" containerName="extract" Jul 10 17:15:14.852781 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:14.852621 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c9d3432-3f08-4957-adec-d09276049015" containerName="util" Jul 10 17:15:14.852781 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:14.852627 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c9d3432-3f08-4957-adec-d09276049015" containerName="util" Jul 10 17:15:14.852781 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:14.852668 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c9d3432-3f08-4957-adec-d09276049015" containerName="extract" Jul 10 17:15:14.856244 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:14.856223 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqcbdl" Jul 10 17:15:14.859227 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:14.859208 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 17:15:14.859227 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:14.859227 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 17:15:14.861949 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:14.861933 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r9sxv\"" Jul 10 17:15:14.871121 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:14.871093 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqcbdl"] Jul 10 17:15:15.030472 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:15.030432 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5a7a1905-16d1-45d3-998a-32f5ecd32a65-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqcbdl\" (UID: \"5a7a1905-16d1-45d3-998a-32f5ecd32a65\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqcbdl" Jul 10 17:15:15.030661 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:15.030486 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5a7a1905-16d1-45d3-998a-32f5ecd32a65-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqcbdl\" (UID: \"5a7a1905-16d1-45d3-998a-32f5ecd32a65\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqcbdl" Jul 10 17:15:15.030661 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:15.030581 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c976z\" (UniqueName: \"kubernetes.io/projected/5a7a1905-16d1-45d3-998a-32f5ecd32a65-kube-api-access-c976z\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqcbdl\" (UID: \"5a7a1905-16d1-45d3-998a-32f5ecd32a65\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqcbdl" Jul 10 17:15:15.131795 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:15.131709 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5a7a1905-16d1-45d3-998a-32f5ecd32a65-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqcbdl\" (UID: \"5a7a1905-16d1-45d3-998a-32f5ecd32a65\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqcbdl" Jul 10 17:15:15.131795 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:15.131754 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5a7a1905-16d1-45d3-998a-32f5ecd32a65-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqcbdl\" (UID: \"5a7a1905-16d1-45d3-998a-32f5ecd32a65\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqcbdl" Jul 10 17:15:15.131795 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:15.131782 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c976z\" (UniqueName: \"kubernetes.io/projected/5a7a1905-16d1-45d3-998a-32f5ecd32a65-kube-api-access-c976z\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqcbdl\" (UID: \"5a7a1905-16d1-45d3-998a-32f5ecd32a65\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqcbdl" Jul 10 17:15:15.132157 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:15.132136 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5a7a1905-16d1-45d3-998a-32f5ecd32a65-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqcbdl\" (UID: \"5a7a1905-16d1-45d3-998a-32f5ecd32a65\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqcbdl" Jul 10 17:15:15.132236 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:15.132202 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5a7a1905-16d1-45d3-998a-32f5ecd32a65-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqcbdl\" (UID: \"5a7a1905-16d1-45d3-998a-32f5ecd32a65\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqcbdl" Jul 10 17:15:15.142532 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:15.142509 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c976z\" (UniqueName: \"kubernetes.io/projected/5a7a1905-16d1-45d3-998a-32f5ecd32a65-kube-api-access-c976z\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqcbdl\" (UID: \"5a7a1905-16d1-45d3-998a-32f5ecd32a65\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqcbdl" Jul 10 17:15:15.164476 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:15.164446 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqcbdl" Jul 10 17:15:15.295360 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:15.295289 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqcbdl"] Jul 10 17:15:15.298141 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:15:15.298110 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a7a1905_16d1_45d3_998a_32f5ecd32a65.slice/crio-52668e1a6710dfa1388f76321fe1ac09541ea6c9f9937054f1877cef452b32e7 WatchSource:0}: Error finding container 52668e1a6710dfa1388f76321fe1ac09541ea6c9f9937054f1877cef452b32e7: Status 404 returned error can't find the container with id 52668e1a6710dfa1388f76321fe1ac09541ea6c9f9937054f1877cef452b32e7 Jul 10 17:15:16.272584 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:16.272554 2583 generic.go:356] "Generic (PLEG): container finished" podID="5a7a1905-16d1-45d3-998a-32f5ecd32a65" containerID="c9107aa7f7c24660558c57ae25598ded783a15302e1eb0c5149c9fefbec5509d" exitCode=0 Jul 10 17:15:16.272980 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:16.272605 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqcbdl" event={"ID":"5a7a1905-16d1-45d3-998a-32f5ecd32a65","Type":"ContainerDied","Data":"c9107aa7f7c24660558c57ae25598ded783a15302e1eb0c5149c9fefbec5509d"} Jul 10 17:15:16.272980 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:16.272647 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqcbdl" event={"ID":"5a7a1905-16d1-45d3-998a-32f5ecd32a65","Type":"ContainerStarted","Data":"52668e1a6710dfa1388f76321fe1ac09541ea6c9f9937054f1877cef452b32e7"} Jul 10 17:15:19.284632 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:19.284601 2583 generic.go:356] "Generic (PLEG): container finished" podID="5a7a1905-16d1-45d3-998a-32f5ecd32a65" containerID="c95c3cd186bfcd12c98d30911b1a392de3e2ebbf7992d5a9988998533219838b" exitCode=0 Jul 10 17:15:19.285091 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:19.284688 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqcbdl" event={"ID":"5a7a1905-16d1-45d3-998a-32f5ecd32a65","Type":"ContainerDied","Data":"c95c3cd186bfcd12c98d30911b1a392de3e2ebbf7992d5a9988998533219838b"} Jul 10 17:15:20.290247 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:20.290214 2583 generic.go:356] "Generic (PLEG): container finished" podID="5a7a1905-16d1-45d3-998a-32f5ecd32a65" containerID="bd2d1dbddc1cedefa9100af7aa84d953a706260e902da8ac471ef7f2cba39bdd" exitCode=0 Jul 10 17:15:20.290692 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:20.290311 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqcbdl" event={"ID":"5a7a1905-16d1-45d3-998a-32f5ecd32a65","Type":"ContainerDied","Data":"bd2d1dbddc1cedefa9100af7aa84d953a706260e902da8ac471ef7f2cba39bdd"} Jul 10 17:15:21.412071 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:21.412047 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqcbdl" Jul 10 17:15:21.482534 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:21.482499 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c976z\" (UniqueName: \"kubernetes.io/projected/5a7a1905-16d1-45d3-998a-32f5ecd32a65-kube-api-access-c976z\") pod \"5a7a1905-16d1-45d3-998a-32f5ecd32a65\" (UID: \"5a7a1905-16d1-45d3-998a-32f5ecd32a65\") " Jul 10 17:15:21.482534 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:21.482540 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5a7a1905-16d1-45d3-998a-32f5ecd32a65-util\") pod \"5a7a1905-16d1-45d3-998a-32f5ecd32a65\" (UID: \"5a7a1905-16d1-45d3-998a-32f5ecd32a65\") " Jul 10 17:15:21.482739 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:21.482566 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5a7a1905-16d1-45d3-998a-32f5ecd32a65-bundle\") pod \"5a7a1905-16d1-45d3-998a-32f5ecd32a65\" (UID: \"5a7a1905-16d1-45d3-998a-32f5ecd32a65\") " Jul 10 17:15:21.482986 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:21.482956 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a7a1905-16d1-45d3-998a-32f5ecd32a65-bundle" (OuterVolumeSpecName: "bundle") pod "5a7a1905-16d1-45d3-998a-32f5ecd32a65" (UID: "5a7a1905-16d1-45d3-998a-32f5ecd32a65"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:15:21.484766 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:21.484734 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a7a1905-16d1-45d3-998a-32f5ecd32a65-kube-api-access-c976z" (OuterVolumeSpecName: "kube-api-access-c976z") pod "5a7a1905-16d1-45d3-998a-32f5ecd32a65" (UID: "5a7a1905-16d1-45d3-998a-32f5ecd32a65"). InnerVolumeSpecName "kube-api-access-c976z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:15:21.487274 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:21.487254 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a7a1905-16d1-45d3-998a-32f5ecd32a65-util" (OuterVolumeSpecName: "util") pod "5a7a1905-16d1-45d3-998a-32f5ecd32a65" (UID: "5a7a1905-16d1-45d3-998a-32f5ecd32a65"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:15:21.583110 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:21.583012 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c976z\" (UniqueName: \"kubernetes.io/projected/5a7a1905-16d1-45d3-998a-32f5ecd32a65-kube-api-access-c976z\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:15:21.583110 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:21.583054 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5a7a1905-16d1-45d3-998a-32f5ecd32a65-util\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:15:21.583110 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:21.583063 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5a7a1905-16d1-45d3-998a-32f5ecd32a65-bundle\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:15:22.299011 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:22.298968 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqcbdl" event={"ID":"5a7a1905-16d1-45d3-998a-32f5ecd32a65","Type":"ContainerDied","Data":"52668e1a6710dfa1388f76321fe1ac09541ea6c9f9937054f1877cef452b32e7"} Jul 10 17:15:22.299011 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:22.298989 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqcbdl" Jul 10 17:15:22.299011 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:22.299022 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52668e1a6710dfa1388f76321fe1ac09541ea6c9f9937054f1877cef452b32e7" Jul 10 17:15:40.539118 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:40.539084 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxtvl"] Jul 10 17:15:40.539526 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:40.539306 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a7a1905-16d1-45d3-998a-32f5ecd32a65" containerName="extract" Jul 10 17:15:40.539526 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:40.539316 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a7a1905-16d1-45d3-998a-32f5ecd32a65" containerName="extract" Jul 10 17:15:40.539526 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:40.539328 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a7a1905-16d1-45d3-998a-32f5ecd32a65" containerName="pull" Jul 10 17:15:40.539526 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:40.539333 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a7a1905-16d1-45d3-998a-32f5ecd32a65" containerName="pull" Jul 10 17:15:40.539526 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:40.539341 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a7a1905-16d1-45d3-998a-32f5ecd32a65" containerName="util" Jul 10 17:15:40.539526 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:40.539346 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a7a1905-16d1-45d3-998a-32f5ecd32a65" containerName="util" Jul 10 17:15:40.539526 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:40.539402 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a7a1905-16d1-45d3-998a-32f5ecd32a65" containerName="extract" Jul 10 17:15:40.544106 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:40.544087 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxtvl" Jul 10 17:15:40.546510 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:40.546492 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 17:15:40.547299 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:40.547280 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r9sxv\"" Jul 10 17:15:40.547337 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:40.547296 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 17:15:40.553679 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:40.551693 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxtvl"] Jul 10 17:15:40.609012 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:40.608975 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llt29\" (UniqueName: \"kubernetes.io/projected/c57a55cd-a507-4c7e-a37a-7669e443643e-kube-api-access-llt29\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxtvl\" (UID: \"c57a55cd-a507-4c7e-a37a-7669e443643e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxtvl" Jul 10 17:15:40.609176 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:40.609020 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c57a55cd-a507-4c7e-a37a-7669e443643e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxtvl\" (UID: \"c57a55cd-a507-4c7e-a37a-7669e443643e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxtvl" Jul 10 17:15:40.609176 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:40.609069 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c57a55cd-a507-4c7e-a37a-7669e443643e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxtvl\" (UID: \"c57a55cd-a507-4c7e-a37a-7669e443643e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxtvl" Jul 10 17:15:40.709756 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:40.709721 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llt29\" (UniqueName: \"kubernetes.io/projected/c57a55cd-a507-4c7e-a37a-7669e443643e-kube-api-access-llt29\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxtvl\" (UID: \"c57a55cd-a507-4c7e-a37a-7669e443643e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxtvl" Jul 10 17:15:40.709756 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:40.709762 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c57a55cd-a507-4c7e-a37a-7669e443643e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxtvl\" (UID: \"c57a55cd-a507-4c7e-a37a-7669e443643e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxtvl" Jul 10 17:15:40.709959 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:40.709784 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c57a55cd-a507-4c7e-a37a-7669e443643e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxtvl\" (UID: \"c57a55cd-a507-4c7e-a37a-7669e443643e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxtvl" Jul 10 17:15:40.710107 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:40.710093 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c57a55cd-a507-4c7e-a37a-7669e443643e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxtvl\" (UID: \"c57a55cd-a507-4c7e-a37a-7669e443643e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxtvl" Jul 10 17:15:40.710145 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:40.710128 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c57a55cd-a507-4c7e-a37a-7669e443643e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxtvl\" (UID: \"c57a55cd-a507-4c7e-a37a-7669e443643e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxtvl" Jul 10 17:15:40.718544 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:40.718519 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llt29\" (UniqueName: \"kubernetes.io/projected/c57a55cd-a507-4c7e-a37a-7669e443643e-kube-api-access-llt29\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxtvl\" (UID: \"c57a55cd-a507-4c7e-a37a-7669e443643e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxtvl" Jul 10 17:15:40.857586 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:40.857505 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxtvl" Jul 10 17:15:40.986605 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:40.986409 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxtvl"] Jul 10 17:15:40.989161 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:15:40.989136 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc57a55cd_a507_4c7e_a37a_7669e443643e.slice/crio-c978a9311bba3d63723ac0d04f67d131540ced7708b603ef691071196b3c8507 WatchSource:0}: Error finding container c978a9311bba3d63723ac0d04f67d131540ced7708b603ef691071196b3c8507: Status 404 returned error can't find the container with id c978a9311bba3d63723ac0d04f67d131540ced7708b603ef691071196b3c8507 Jul 10 17:15:41.368849 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:41.368801 2583 generic.go:356] "Generic (PLEG): container finished" podID="c57a55cd-a507-4c7e-a37a-7669e443643e" containerID="9f9cd4ee46a1a5cb7a2a8d123c42d21059c50ead23e1e182f5bd0485e2e30a92" exitCode=0 Jul 10 17:15:41.369016 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:41.368863 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxtvl" event={"ID":"c57a55cd-a507-4c7e-a37a-7669e443643e","Type":"ContainerDied","Data":"9f9cd4ee46a1a5cb7a2a8d123c42d21059c50ead23e1e182f5bd0485e2e30a92"} Jul 10 17:15:41.369016 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:41.368916 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxtvl" event={"ID":"c57a55cd-a507-4c7e-a37a-7669e443643e","Type":"ContainerStarted","Data":"c978a9311bba3d63723ac0d04f67d131540ced7708b603ef691071196b3c8507"} Jul 10 17:15:42.373925 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:42.373829 2583 generic.go:356] "Generic (PLEG): container finished" podID="c57a55cd-a507-4c7e-a37a-7669e443643e" containerID="dcf4bdcdd9f18885d70931b27e38d29ffa3bf6ff9a423dff5647e93d7961caec" exitCode=0 Jul 10 17:15:42.374341 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:42.373920 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxtvl" event={"ID":"c57a55cd-a507-4c7e-a37a-7669e443643e","Type":"ContainerDied","Data":"dcf4bdcdd9f18885d70931b27e38d29ffa3bf6ff9a423dff5647e93d7961caec"} Jul 10 17:15:43.379643 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:43.379601 2583 generic.go:356] "Generic (PLEG): container finished" podID="c57a55cd-a507-4c7e-a37a-7669e443643e" containerID="3db65c11f006b4b2acba2bceeed787f7dff9d2a01c364a72b733db479ce331d4" exitCode=0 Jul 10 17:15:43.380050 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:43.379690 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxtvl" event={"ID":"c57a55cd-a507-4c7e-a37a-7669e443643e","Type":"ContainerDied","Data":"3db65c11f006b4b2acba2bceeed787f7dff9d2a01c364a72b733db479ce331d4"} Jul 10 17:15:44.504875 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:44.504854 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxtvl" Jul 10 17:15:44.540009 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:44.539969 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c57a55cd-a507-4c7e-a37a-7669e443643e-util\") pod \"c57a55cd-a507-4c7e-a37a-7669e443643e\" (UID: \"c57a55cd-a507-4c7e-a37a-7669e443643e\") " Jul 10 17:15:44.540167 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:44.540029 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c57a55cd-a507-4c7e-a37a-7669e443643e-bundle\") pod \"c57a55cd-a507-4c7e-a37a-7669e443643e\" (UID: \"c57a55cd-a507-4c7e-a37a-7669e443643e\") " Jul 10 17:15:44.540167 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:44.540061 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llt29\" (UniqueName: \"kubernetes.io/projected/c57a55cd-a507-4c7e-a37a-7669e443643e-kube-api-access-llt29\") pod \"c57a55cd-a507-4c7e-a37a-7669e443643e\" (UID: \"c57a55cd-a507-4c7e-a37a-7669e443643e\") " Jul 10 17:15:44.540925 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:44.540863 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57a55cd-a507-4c7e-a37a-7669e443643e-bundle" (OuterVolumeSpecName: "bundle") pod "c57a55cd-a507-4c7e-a37a-7669e443643e" (UID: "c57a55cd-a507-4c7e-a37a-7669e443643e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:15:44.542364 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:44.542336 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57a55cd-a507-4c7e-a37a-7669e443643e-kube-api-access-llt29" (OuterVolumeSpecName: "kube-api-access-llt29") pod "c57a55cd-a507-4c7e-a37a-7669e443643e" (UID: "c57a55cd-a507-4c7e-a37a-7669e443643e"). InnerVolumeSpecName "kube-api-access-llt29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:15:44.546734 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:44.546710 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57a55cd-a507-4c7e-a37a-7669e443643e-util" (OuterVolumeSpecName: "util") pod "c57a55cd-a507-4c7e-a37a-7669e443643e" (UID: "c57a55cd-a507-4c7e-a37a-7669e443643e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:15:44.640940 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:44.640844 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c57a55cd-a507-4c7e-a37a-7669e443643e-util\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:15:44.640940 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:44.640888 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c57a55cd-a507-4c7e-a37a-7669e443643e-bundle\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:15:44.640940 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:44.640898 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llt29\" (UniqueName: \"kubernetes.io/projected/c57a55cd-a507-4c7e-a37a-7669e443643e-kube-api-access-llt29\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:15:45.388255 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:45.388221 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxtvl" event={"ID":"c57a55cd-a507-4c7e-a37a-7669e443643e","Type":"ContainerDied","Data":"c978a9311bba3d63723ac0d04f67d131540ced7708b603ef691071196b3c8507"} Jul 10 17:15:45.388451 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:45.388274 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c978a9311bba3d63723ac0d04f67d131540ced7708b603ef691071196b3c8507" Jul 10 17:15:45.388451 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:45.388236 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkxtvl" Jul 10 17:15:52.616468 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:52.616433 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b884cfc77-g4rrm"] Jul 10 17:15:52.616934 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:52.616708 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c57a55cd-a507-4c7e-a37a-7669e443643e" containerName="pull" Jul 10 17:15:52.616934 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:52.616721 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c57a55cd-a507-4c7e-a37a-7669e443643e" containerName="pull" Jul 10 17:15:52.616934 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:52.616732 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c57a55cd-a507-4c7e-a37a-7669e443643e" containerName="extract" Jul 10 17:15:52.616934 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:52.616738 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c57a55cd-a507-4c7e-a37a-7669e443643e" containerName="extract" Jul 10 17:15:52.616934 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:52.616758 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c57a55cd-a507-4c7e-a37a-7669e443643e" containerName="util" Jul 10 17:15:52.616934 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:52.616764 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c57a55cd-a507-4c7e-a37a-7669e443643e" containerName="util" Jul 10 17:15:52.616934 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:52.616813 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c57a55cd-a507-4c7e-a37a-7669e443643e" containerName="extract" Jul 10 17:15:52.618986 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:52.618969 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-g4rrm" Jul 10 17:15:52.622215 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:52.622186 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 17:15:52.622848 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:52.622825 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 17:15:52.623148 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:52.623128 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-znxdl\"" Jul 10 17:15:52.623148 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:52.623127 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 17:15:52.623305 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:52.623153 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 17:15:52.636588 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:52.636559 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b884cfc77-g4rrm"] Jul 10 17:15:52.698480 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:52.698443 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwsd6\" (UniqueName: \"kubernetes.io/projected/c545fd6c-d7a9-4ad4-845c-2504f3cf59fc-kube-api-access-zwsd6\") pod \"opendatahub-operator-controller-manager-7b884cfc77-g4rrm\" (UID: \"c545fd6c-d7a9-4ad4-845c-2504f3cf59fc\") " pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-g4rrm" Jul 10 17:15:52.698665 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:52.698511 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c545fd6c-d7a9-4ad4-845c-2504f3cf59fc-manifests\") pod \"opendatahub-operator-controller-manager-7b884cfc77-g4rrm\" (UID: \"c545fd6c-d7a9-4ad4-845c-2504f3cf59fc\") " pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-g4rrm" Jul 10 17:15:52.698665 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:52.698539 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c545fd6c-d7a9-4ad4-845c-2504f3cf59fc-tmp\") pod \"opendatahub-operator-controller-manager-7b884cfc77-g4rrm\" (UID: \"c545fd6c-d7a9-4ad4-845c-2504f3cf59fc\") " pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-g4rrm" Jul 10 17:15:52.698665 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:52.698563 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c545fd6c-d7a9-4ad4-845c-2504f3cf59fc-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b884cfc77-g4rrm\" (UID: \"c545fd6c-d7a9-4ad4-845c-2504f3cf59fc\") " pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-g4rrm" Jul 10 17:15:52.698665 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:52.698589 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c545fd6c-d7a9-4ad4-845c-2504f3cf59fc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b884cfc77-g4rrm\" (UID: \"c545fd6c-d7a9-4ad4-845c-2504f3cf59fc\") " pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-g4rrm" Jul 10 17:15:52.799354 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:52.799313 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c545fd6c-d7a9-4ad4-845c-2504f3cf59fc-manifests\") pod \"opendatahub-operator-controller-manager-7b884cfc77-g4rrm\" (UID: \"c545fd6c-d7a9-4ad4-845c-2504f3cf59fc\") " pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-g4rrm" Jul 10 17:15:52.799354 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:52.799352 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c545fd6c-d7a9-4ad4-845c-2504f3cf59fc-tmp\") pod \"opendatahub-operator-controller-manager-7b884cfc77-g4rrm\" (UID: \"c545fd6c-d7a9-4ad4-845c-2504f3cf59fc\") " pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-g4rrm" Jul 10 17:15:52.799587 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:52.799380 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c545fd6c-d7a9-4ad4-845c-2504f3cf59fc-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b884cfc77-g4rrm\" (UID: \"c545fd6c-d7a9-4ad4-845c-2504f3cf59fc\") " pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-g4rrm" Jul 10 17:15:52.799587 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:52.799438 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c545fd6c-d7a9-4ad4-845c-2504f3cf59fc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b884cfc77-g4rrm\" (UID: \"c545fd6c-d7a9-4ad4-845c-2504f3cf59fc\") " pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-g4rrm" Jul 10 17:15:52.799587 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:52.799464 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwsd6\" (UniqueName: \"kubernetes.io/projected/c545fd6c-d7a9-4ad4-845c-2504f3cf59fc-kube-api-access-zwsd6\") pod \"opendatahub-operator-controller-manager-7b884cfc77-g4rrm\" (UID: \"c545fd6c-d7a9-4ad4-845c-2504f3cf59fc\") " pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-g4rrm" Jul 10 17:15:52.799798 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:52.799773 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c545fd6c-d7a9-4ad4-845c-2504f3cf59fc-manifests\") pod \"opendatahub-operator-controller-manager-7b884cfc77-g4rrm\" (UID: \"c545fd6c-d7a9-4ad4-845c-2504f3cf59fc\") " pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-g4rrm" Jul 10 17:15:52.801883 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:52.801862 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c545fd6c-d7a9-4ad4-845c-2504f3cf59fc-tmp\") pod \"opendatahub-operator-controller-manager-7b884cfc77-g4rrm\" (UID: \"c545fd6c-d7a9-4ad4-845c-2504f3cf59fc\") " pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-g4rrm" Jul 10 17:15:52.802082 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:52.802064 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c545fd6c-d7a9-4ad4-845c-2504f3cf59fc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b884cfc77-g4rrm\" (UID: \"c545fd6c-d7a9-4ad4-845c-2504f3cf59fc\") " pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-g4rrm" Jul 10 17:15:52.802205 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:52.802182 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c545fd6c-d7a9-4ad4-845c-2504f3cf59fc-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b884cfc77-g4rrm\" (UID: \"c545fd6c-d7a9-4ad4-845c-2504f3cf59fc\") " pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-g4rrm" Jul 10 17:15:52.813178 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:52.813153 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwsd6\" (UniqueName: \"kubernetes.io/projected/c545fd6c-d7a9-4ad4-845c-2504f3cf59fc-kube-api-access-zwsd6\") pod \"opendatahub-operator-controller-manager-7b884cfc77-g4rrm\" (UID: \"c545fd6c-d7a9-4ad4-845c-2504f3cf59fc\") " pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-g4rrm" Jul 10 17:15:52.929426 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:52.929368 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-g4rrm" Jul 10 17:15:53.068303 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:53.068231 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b884cfc77-g4rrm"] Jul 10 17:15:53.070686 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:15:53.070660 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc545fd6c_d7a9_4ad4_845c_2504f3cf59fc.slice/crio-91b69b59562bd513bca2a7c838bab4e5340312cd3265ee4860232ae796f7fd51 WatchSource:0}: Error finding container 91b69b59562bd513bca2a7c838bab4e5340312cd3265ee4860232ae796f7fd51: Status 404 returned error can't find the container with id 91b69b59562bd513bca2a7c838bab4e5340312cd3265ee4860232ae796f7fd51 Jul 10 17:15:53.417357 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:53.417321 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-g4rrm" event={"ID":"c545fd6c-d7a9-4ad4-845c-2504f3cf59fc","Type":"ContainerStarted","Data":"91b69b59562bd513bca2a7c838bab4e5340312cd3265ee4860232ae796f7fd51"} Jul 10 17:15:56.432084 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:56.432041 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-g4rrm" event={"ID":"c545fd6c-d7a9-4ad4-845c-2504f3cf59fc","Type":"ContainerStarted","Data":"abbe1eb6d715f1574c7e83125ec4da34e1b1ef79e4bf858c15e4e97860198ecd"} Jul 10 17:15:57.438429 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:57.438378 2583 generic.go:356] "Generic (PLEG): container finished" podID="c545fd6c-d7a9-4ad4-845c-2504f3cf59fc" containerID="abbe1eb6d715f1574c7e83125ec4da34e1b1ef79e4bf858c15e4e97860198ecd" exitCode=0 Jul 10 17:15:57.438880 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:57.438437 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-g4rrm" event={"ID":"c545fd6c-d7a9-4ad4-845c-2504f3cf59fc","Type":"ContainerDied","Data":"abbe1eb6d715f1574c7e83125ec4da34e1b1ef79e4bf858c15e4e97860198ecd"} Jul 10 17:15:58.443718 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:58.443684 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-g4rrm" event={"ID":"c545fd6c-d7a9-4ad4-845c-2504f3cf59fc","Type":"ContainerStarted","Data":"74dd37cf07d51872796a7923a2a5f29b4875b2e6e4e001f2d8b9e146f1f4667d"} Jul 10 17:15:58.444095 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:15:58.443756 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-g4rrm" Jul 10 17:16:08.058257 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:08.058220 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js69w_e41baaf7-4171-4df1-8bdd-d0697a61fdce/ovn-acl-logging/0.log" Jul 10 17:16:08.061985 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:08.061959 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js69w_e41baaf7-4171-4df1-8bdd-d0697a61fdce/ovn-acl-logging/0.log" Jul 10 17:16:08.062697 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:08.062676 2583 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 17:16:09.449547 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:09.449513 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-g4rrm" Jul 10 17:16:09.471596 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:09.471552 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7b884cfc77-g4rrm" podStartSLOduration=12.861002473 podStartE2EDuration="17.471535104s" totalImagesPullingTime="3.563135882s" totalInitContainerRuntime="1.047396749s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:15:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:15:58.477469036 +0000 UTC m=+290.805166575" watchObservedRunningTime="2026-07-10 17:16:09.471535104 +0000 UTC m=+301.799232642" Jul 10 17:16:17.670049 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:17.670012 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-545f68d885-qzl6x"] Jul 10 17:16:17.677911 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:17.677882 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-545f68d885-qzl6x" Jul 10 17:16:17.680599 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:17.680562 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 10 17:16:17.680750 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:17.680629 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 10 17:16:17.680750 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:17.680643 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 17:16:17.681299 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:17.681285 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 17:16:17.681345 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:17.681291 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-42f7l\"" Jul 10 17:16:17.684031 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:17.684006 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-545f68d885-qzl6x"] Jul 10 17:16:17.776278 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:17.776243 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7875067d-0704-4751-9605-5e851f2421a8-tls-certs\") pod \"kube-auth-proxy-545f68d885-qzl6x\" (UID: \"7875067d-0704-4751-9605-5e851f2421a8\") " pod="openshift-ingress/kube-auth-proxy-545f68d885-qzl6x" Jul 10 17:16:17.776487 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:17.776302 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgnbp\" (UniqueName: \"kubernetes.io/projected/7875067d-0704-4751-9605-5e851f2421a8-kube-api-access-xgnbp\") pod \"kube-auth-proxy-545f68d885-qzl6x\" (UID: \"7875067d-0704-4751-9605-5e851f2421a8\") " pod="openshift-ingress/kube-auth-proxy-545f68d885-qzl6x" Jul 10 17:16:17.776487 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:17.776356 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7875067d-0704-4751-9605-5e851f2421a8-tmp\") pod \"kube-auth-proxy-545f68d885-qzl6x\" (UID: \"7875067d-0704-4751-9605-5e851f2421a8\") " pod="openshift-ingress/kube-auth-proxy-545f68d885-qzl6x" Jul 10 17:16:17.836473 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:17.836431 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-h8qcx"] Jul 10 17:16:17.838916 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:17.838881 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-h8qcx" Jul 10 17:16:17.841238 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:17.841215 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 10 17:16:17.841363 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:17.841328 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-bntm7\"" Jul 10 17:16:17.851563 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:17.851525 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-h8qcx"] Jul 10 17:16:17.879882 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:17.879842 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7875067d-0704-4751-9605-5e851f2421a8-tls-certs\") pod \"kube-auth-proxy-545f68d885-qzl6x\" (UID: \"7875067d-0704-4751-9605-5e851f2421a8\") " pod="openshift-ingress/kube-auth-proxy-545f68d885-qzl6x" Jul 10 17:16:17.880066 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:17.879935 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgnbp\" (UniqueName: \"kubernetes.io/projected/7875067d-0704-4751-9605-5e851f2421a8-kube-api-access-xgnbp\") pod \"kube-auth-proxy-545f68d885-qzl6x\" (UID: \"7875067d-0704-4751-9605-5e851f2421a8\") " pod="openshift-ingress/kube-auth-proxy-545f68d885-qzl6x" Jul 10 17:16:17.880066 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:17.879973 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/313596b5-3368-4ca3-8f71-0be58e50e7f4-cert\") pod \"odh-model-controller-7998c67dc4-h8qcx\" (UID: \"313596b5-3368-4ca3-8f71-0be58e50e7f4\") " pod="opendatahub/odh-model-controller-7998c67dc4-h8qcx" Jul 10 17:16:17.880066 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:17.880006 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r8mb\" (UniqueName: \"kubernetes.io/projected/313596b5-3368-4ca3-8f71-0be58e50e7f4-kube-api-access-5r8mb\") pod \"odh-model-controller-7998c67dc4-h8qcx\" (UID: \"313596b5-3368-4ca3-8f71-0be58e50e7f4\") " pod="opendatahub/odh-model-controller-7998c67dc4-h8qcx" Jul 10 17:16:17.880066 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:17.880040 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7875067d-0704-4751-9605-5e851f2421a8-tmp\") pod \"kube-auth-proxy-545f68d885-qzl6x\" (UID: \"7875067d-0704-4751-9605-5e851f2421a8\") " pod="openshift-ingress/kube-auth-proxy-545f68d885-qzl6x" Jul 10 17:16:17.880943 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:16:17.880918 2583 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 10 17:16:17.881055 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:16:17.881015 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7875067d-0704-4751-9605-5e851f2421a8-tls-certs podName:7875067d-0704-4751-9605-5e851f2421a8 nodeName:}" failed. No retries permitted until 2026-07-10 17:16:18.380984433 +0000 UTC m=+310.708681953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7875067d-0704-4751-9605-5e851f2421a8-tls-certs") pod "kube-auth-proxy-545f68d885-qzl6x" (UID: "7875067d-0704-4751-9605-5e851f2421a8") : secret "kube-auth-proxy-tls" not found Jul 10 17:16:17.884568 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:17.884540 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7875067d-0704-4751-9605-5e851f2421a8-tmp\") pod \"kube-auth-proxy-545f68d885-qzl6x\" (UID: \"7875067d-0704-4751-9605-5e851f2421a8\") " pod="openshift-ingress/kube-auth-proxy-545f68d885-qzl6x" Jul 10 17:16:17.889576 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:17.889547 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgnbp\" (UniqueName: \"kubernetes.io/projected/7875067d-0704-4751-9605-5e851f2421a8-kube-api-access-xgnbp\") pod \"kube-auth-proxy-545f68d885-qzl6x\" (UID: \"7875067d-0704-4751-9605-5e851f2421a8\") " pod="openshift-ingress/kube-auth-proxy-545f68d885-qzl6x" Jul 10 17:16:17.980723 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:17.980621 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/313596b5-3368-4ca3-8f71-0be58e50e7f4-cert\") pod \"odh-model-controller-7998c67dc4-h8qcx\" (UID: \"313596b5-3368-4ca3-8f71-0be58e50e7f4\") " pod="opendatahub/odh-model-controller-7998c67dc4-h8qcx" Jul 10 17:16:17.980723 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:17.980664 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r8mb\" (UniqueName: \"kubernetes.io/projected/313596b5-3368-4ca3-8f71-0be58e50e7f4-kube-api-access-5r8mb\") pod \"odh-model-controller-7998c67dc4-h8qcx\" (UID: \"313596b5-3368-4ca3-8f71-0be58e50e7f4\") " pod="opendatahub/odh-model-controller-7998c67dc4-h8qcx" Jul 10 17:16:17.980917 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:16:17.980768 2583 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 10 17:16:17.980917 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:16:17.980844 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/313596b5-3368-4ca3-8f71-0be58e50e7f4-cert podName:313596b5-3368-4ca3-8f71-0be58e50e7f4 nodeName:}" failed. No retries permitted until 2026-07-10 17:16:18.480826259 +0000 UTC m=+310.808523777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/313596b5-3368-4ca3-8f71-0be58e50e7f4-cert") pod "odh-model-controller-7998c67dc4-h8qcx" (UID: "313596b5-3368-4ca3-8f71-0be58e50e7f4") : secret "odh-model-controller-webhook-cert" not found Jul 10 17:16:17.994138 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:17.994106 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r8mb\" (UniqueName: \"kubernetes.io/projected/313596b5-3368-4ca3-8f71-0be58e50e7f4-kube-api-access-5r8mb\") pod \"odh-model-controller-7998c67dc4-h8qcx\" (UID: \"313596b5-3368-4ca3-8f71-0be58e50e7f4\") " pod="opendatahub/odh-model-controller-7998c67dc4-h8qcx" Jul 10 17:16:18.383048 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.382956 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7875067d-0704-4751-9605-5e851f2421a8-tls-certs\") pod \"kube-auth-proxy-545f68d885-qzl6x\" (UID: \"7875067d-0704-4751-9605-5e851f2421a8\") " pod="openshift-ingress/kube-auth-proxy-545f68d885-qzl6x" Jul 10 17:16:18.385619 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.385600 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7875067d-0704-4751-9605-5e851f2421a8-tls-certs\") pod \"kube-auth-proxy-545f68d885-qzl6x\" (UID: \"7875067d-0704-4751-9605-5e851f2421a8\") " pod="openshift-ingress/kube-auth-proxy-545f68d885-qzl6x" Jul 10 17:16:18.484010 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.483973 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/313596b5-3368-4ca3-8f71-0be58e50e7f4-cert\") pod \"odh-model-controller-7998c67dc4-h8qcx\" (UID: \"313596b5-3368-4ca3-8f71-0be58e50e7f4\") " pod="opendatahub/odh-model-controller-7998c67dc4-h8qcx" Jul 10 17:16:18.486733 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.486707 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/313596b5-3368-4ca3-8f71-0be58e50e7f4-cert\") pod \"odh-model-controller-7998c67dc4-h8qcx\" (UID: \"313596b5-3368-4ca3-8f71-0be58e50e7f4\") " pod="opendatahub/odh-model-controller-7998c67dc4-h8qcx" Jul 10 17:16:18.586165 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.586116 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-7cxn7"] Jul 10 17:16:18.588495 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.588475 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-7cxn7" Jul 10 17:16:18.589451 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.589371 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-545f68d885-qzl6x" Jul 10 17:16:18.590945 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.590929 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 10 17:16:18.607127 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.607094 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-7cxn7"] Jul 10 17:16:18.686150 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.686054 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dae95bcd-c4af-4c58-bf11-2aef952610c5-tls-certs\") pod \"model-serving-api-54d89bd5fc-7cxn7\" (UID: \"dae95bcd-c4af-4c58-bf11-2aef952610c5\") " pod="opendatahub/model-serving-api-54d89bd5fc-7cxn7" Jul 10 17:16:18.686150 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.686140 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8mfl\" (UniqueName: \"kubernetes.io/projected/dae95bcd-c4af-4c58-bf11-2aef952610c5-kube-api-access-w8mfl\") pod \"model-serving-api-54d89bd5fc-7cxn7\" (UID: \"dae95bcd-c4af-4c58-bf11-2aef952610c5\") " pod="opendatahub/model-serving-api-54d89bd5fc-7cxn7" Jul 10 17:16:18.735750 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.735624 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-545f68d885-qzl6x"] Jul 10 17:16:18.738494 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:16:18.738449 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7875067d_0704_4751_9605_5e851f2421a8.slice/crio-df231c192e23d64785f5d8ec3baf09a2d57952316a9a9f9bd7b3f9cb4608603b WatchSource:0}: Error finding container df231c192e23d64785f5d8ec3baf09a2d57952316a9a9f9bd7b3f9cb4608603b: Status 404 returned error can't find the container with id df231c192e23d64785f5d8ec3baf09a2d57952316a9a9f9bd7b3f9cb4608603b Jul 10 17:16:18.740355 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.740337 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 17:16:18.750057 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.750033 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-h8qcx" Jul 10 17:16:18.787355 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.787320 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dae95bcd-c4af-4c58-bf11-2aef952610c5-tls-certs\") pod \"model-serving-api-54d89bd5fc-7cxn7\" (UID: \"dae95bcd-c4af-4c58-bf11-2aef952610c5\") " pod="opendatahub/model-serving-api-54d89bd5fc-7cxn7" Jul 10 17:16:18.787544 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.787434 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8mfl\" (UniqueName: \"kubernetes.io/projected/dae95bcd-c4af-4c58-bf11-2aef952610c5-kube-api-access-w8mfl\") pod \"model-serving-api-54d89bd5fc-7cxn7\" (UID: \"dae95bcd-c4af-4c58-bf11-2aef952610c5\") " pod="opendatahub/model-serving-api-54d89bd5fc-7cxn7" Jul 10 17:16:18.789826 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.789806 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dae95bcd-c4af-4c58-bf11-2aef952610c5-tls-certs\") pod \"model-serving-api-54d89bd5fc-7cxn7\" (UID: \"dae95bcd-c4af-4c58-bf11-2aef952610c5\") " pod="opendatahub/model-serving-api-54d89bd5fc-7cxn7" Jul 10 17:16:18.797223 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.797189 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8mfl\" (UniqueName: \"kubernetes.io/projected/dae95bcd-c4af-4c58-bf11-2aef952610c5-kube-api-access-w8mfl\") pod \"model-serving-api-54d89bd5fc-7cxn7\" (UID: \"dae95bcd-c4af-4c58-bf11-2aef952610c5\") " pod="opendatahub/model-serving-api-54d89bd5fc-7cxn7" Jul 10 17:16:18.810019 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.809988 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6"] Jul 10 17:16:18.815234 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.813887 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6" Jul 10 17:16:18.818524 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.816679 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-d2dq4\"" Jul 10 17:16:18.818524 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.816993 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 10 17:16:18.818524 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.817407 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 17:16:18.818524 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.817585 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 10 17:16:18.818524 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.817748 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 10 17:16:18.828042 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.828016 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6"] Jul 10 17:16:18.887829 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.887798 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/834b6019-1993-45d6-8a07-aeeee385304a-istio-token\") pod \"istiod-openshift-gateway-7574459656-hrrs6\" (UID: \"834b6019-1993-45d6-8a07-aeeee385304a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6" Jul 10 17:16:18.887997 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.887835 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/834b6019-1993-45d6-8a07-aeeee385304a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-hrrs6\" (UID: \"834b6019-1993-45d6-8a07-aeeee385304a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6" Jul 10 17:16:18.887997 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.887865 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnsfq\" (UniqueName: \"kubernetes.io/projected/834b6019-1993-45d6-8a07-aeeee385304a-kube-api-access-gnsfq\") pod \"istiod-openshift-gateway-7574459656-hrrs6\" (UID: \"834b6019-1993-45d6-8a07-aeeee385304a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6" Jul 10 17:16:18.887997 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.887892 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/834b6019-1993-45d6-8a07-aeeee385304a-local-certs\") pod \"istiod-openshift-gateway-7574459656-hrrs6\" (UID: \"834b6019-1993-45d6-8a07-aeeee385304a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6" Jul 10 17:16:18.887997 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.887934 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/834b6019-1993-45d6-8a07-aeeee385304a-cacerts\") pod \"istiod-openshift-gateway-7574459656-hrrs6\" (UID: \"834b6019-1993-45d6-8a07-aeeee385304a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6" Jul 10 17:16:18.887997 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.887984 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/834b6019-1993-45d6-8a07-aeeee385304a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-hrrs6\" (UID: \"834b6019-1993-45d6-8a07-aeeee385304a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6" Jul 10 17:16:18.888182 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.888012 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/834b6019-1993-45d6-8a07-aeeee385304a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-hrrs6\" (UID: \"834b6019-1993-45d6-8a07-aeeee385304a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6" Jul 10 17:16:18.902440 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.902351 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-h8qcx"] Jul 10 17:16:18.902440 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.902418 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-7cxn7" Jul 10 17:16:18.905542 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:16:18.905496 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod313596b5_3368_4ca3_8f71_0be58e50e7f4.slice/crio-87f7e13d0962f6435a52bfb4bd92157a685dd16375edacb8a45814eb809abf92 WatchSource:0}: Error finding container 87f7e13d0962f6435a52bfb4bd92157a685dd16375edacb8a45814eb809abf92: Status 404 returned error can't find the container with id 87f7e13d0962f6435a52bfb4bd92157a685dd16375edacb8a45814eb809abf92 Jul 10 17:16:18.988802 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.988764 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/834b6019-1993-45d6-8a07-aeeee385304a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-hrrs6\" (UID: \"834b6019-1993-45d6-8a07-aeeee385304a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6" Jul 10 17:16:18.988934 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.988820 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/834b6019-1993-45d6-8a07-aeeee385304a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-hrrs6\" (UID: \"834b6019-1993-45d6-8a07-aeeee385304a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6" Jul 10 17:16:18.988934 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.988888 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/834b6019-1993-45d6-8a07-aeeee385304a-istio-token\") pod \"istiod-openshift-gateway-7574459656-hrrs6\" (UID: \"834b6019-1993-45d6-8a07-aeeee385304a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6" Jul 10 17:16:18.988934 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.988915 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/834b6019-1993-45d6-8a07-aeeee385304a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-hrrs6\" (UID: \"834b6019-1993-45d6-8a07-aeeee385304a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6" Jul 10 17:16:18.989082 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.988945 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnsfq\" (UniqueName: \"kubernetes.io/projected/834b6019-1993-45d6-8a07-aeeee385304a-kube-api-access-gnsfq\") pod \"istiod-openshift-gateway-7574459656-hrrs6\" (UID: \"834b6019-1993-45d6-8a07-aeeee385304a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6" Jul 10 17:16:18.989082 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.988972 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/834b6019-1993-45d6-8a07-aeeee385304a-local-certs\") pod \"istiod-openshift-gateway-7574459656-hrrs6\" (UID: \"834b6019-1993-45d6-8a07-aeeee385304a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6" Jul 10 17:16:18.989082 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.989023 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/834b6019-1993-45d6-8a07-aeeee385304a-cacerts\") pod \"istiod-openshift-gateway-7574459656-hrrs6\" (UID: \"834b6019-1993-45d6-8a07-aeeee385304a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6" Jul 10 17:16:18.990227 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.989600 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/834b6019-1993-45d6-8a07-aeeee385304a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-hrrs6\" (UID: \"834b6019-1993-45d6-8a07-aeeee385304a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6" Jul 10 17:16:18.992116 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.992090 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/834b6019-1993-45d6-8a07-aeeee385304a-cacerts\") pod \"istiod-openshift-gateway-7574459656-hrrs6\" (UID: \"834b6019-1993-45d6-8a07-aeeee385304a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6" Jul 10 17:16:18.993312 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.992987 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/834b6019-1993-45d6-8a07-aeeee385304a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-hrrs6\" (UID: \"834b6019-1993-45d6-8a07-aeeee385304a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6" Jul 10 17:16:18.993312 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.992998 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/834b6019-1993-45d6-8a07-aeeee385304a-local-certs\") pod \"istiod-openshift-gateway-7574459656-hrrs6\" (UID: \"834b6019-1993-45d6-8a07-aeeee385304a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6" Jul 10 17:16:18.993513 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.993331 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/834b6019-1993-45d6-8a07-aeeee385304a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-hrrs6\" (UID: \"834b6019-1993-45d6-8a07-aeeee385304a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6" Jul 10 17:16:18.998426 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.998351 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnsfq\" (UniqueName: \"kubernetes.io/projected/834b6019-1993-45d6-8a07-aeeee385304a-kube-api-access-gnsfq\") pod \"istiod-openshift-gateway-7574459656-hrrs6\" (UID: \"834b6019-1993-45d6-8a07-aeeee385304a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6" Jul 10 17:16:18.998561 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:18.998524 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/834b6019-1993-45d6-8a07-aeeee385304a-istio-token\") pod \"istiod-openshift-gateway-7574459656-hrrs6\" (UID: \"834b6019-1993-45d6-8a07-aeeee385304a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6" Jul 10 17:16:19.043582 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:19.043553 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-7cxn7"] Jul 10 17:16:19.046472 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:16:19.046444 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddae95bcd_c4af_4c58_bf11_2aef952610c5.slice/crio-f8b586f91a0fa20032d0a81b7db4b8155c264e9b31b28a89d93e4dcf572a1a4a WatchSource:0}: Error finding container f8b586f91a0fa20032d0a81b7db4b8155c264e9b31b28a89d93e4dcf572a1a4a: Status 404 returned error can't find the container with id f8b586f91a0fa20032d0a81b7db4b8155c264e9b31b28a89d93e4dcf572a1a4a Jul 10 17:16:19.129901 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:19.129865 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6" Jul 10 17:16:19.268384 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:19.268350 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6"] Jul 10 17:16:19.272615 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:16:19.272587 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod834b6019_1993_45d6_8a07_aeeee385304a.slice/crio-c1a10f90b1c7524cb67d636f6782260d18aa4154316a33e1147e4f2e1242ef7d WatchSource:0}: Error finding container c1a10f90b1c7524cb67d636f6782260d18aa4154316a33e1147e4f2e1242ef7d: Status 404 returned error can't find the container with id c1a10f90b1c7524cb67d636f6782260d18aa4154316a33e1147e4f2e1242ef7d Jul 10 17:16:19.525112 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:19.524999 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-545f68d885-qzl6x" event={"ID":"7875067d-0704-4751-9605-5e851f2421a8","Type":"ContainerStarted","Data":"df231c192e23d64785f5d8ec3baf09a2d57952316a9a9f9bd7b3f9cb4608603b"} Jul 10 17:16:19.526322 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:19.526291 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-7cxn7" event={"ID":"dae95bcd-c4af-4c58-bf11-2aef952610c5","Type":"ContainerStarted","Data":"f8b586f91a0fa20032d0a81b7db4b8155c264e9b31b28a89d93e4dcf572a1a4a"} Jul 10 17:16:19.527508 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:19.527473 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-h8qcx" event={"ID":"313596b5-3368-4ca3-8f71-0be58e50e7f4","Type":"ContainerStarted","Data":"87f7e13d0962f6435a52bfb4bd92157a685dd16375edacb8a45814eb809abf92"} Jul 10 17:16:19.528800 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:19.528777 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6" event={"ID":"834b6019-1993-45d6-8a07-aeeee385304a","Type":"ContainerStarted","Data":"c1a10f90b1c7524cb67d636f6782260d18aa4154316a33e1147e4f2e1242ef7d"} Jul 10 17:16:24.312610 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:24.312476 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:16:24.312610 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:24.312540 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:16:24.567647 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:24.567540 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6" event={"ID":"834b6019-1993-45d6-8a07-aeeee385304a","Type":"ContainerStarted","Data":"850853b800ea569ddd2b1dd1a01462a78e2750ca883cf7f40c3527327b3b12b0"} Jul 10 17:16:24.567817 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:24.567707 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6" Jul 10 17:16:24.568994 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:24.568970 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-545f68d885-qzl6x" event={"ID":"7875067d-0704-4751-9605-5e851f2421a8","Type":"ContainerStarted","Data":"06aea7c6beb7ae11aa7a709327f45132e010f95c6afd65bf0a9b1d5515737c0f"} Jul 10 17:16:24.570593 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:24.570571 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-7cxn7" event={"ID":"dae95bcd-c4af-4c58-bf11-2aef952610c5","Type":"ContainerStarted","Data":"7bb0e0848dfc4498c5634fc1b5c27bc7b8082cd2f0f77dd32ffcbf17b014a2bc"} Jul 10 17:16:24.570712 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:24.570632 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-7cxn7" Jul 10 17:16:24.571951 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:24.571932 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-h8qcx" event={"ID":"313596b5-3368-4ca3-8f71-0be58e50e7f4","Type":"ContainerStarted","Data":"8119ef6afd8ddca0aaa04e6828acc38d94366a675ad91a913626033395ca850b"} Jul 10 17:16:24.572073 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:24.572060 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-h8qcx" Jul 10 17:16:24.585957 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:24.585915 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6" podStartSLOduration=1.548665929 podStartE2EDuration="6.585903213s" totalImagesPullingTime="5.037237284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:16:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:16:24.585164117 +0000 UTC m=+316.912861697" watchObservedRunningTime="2026-07-10 17:16:24.585903213 +0000 UTC m=+316.913600753" Jul 10 17:16:24.601831 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:24.601789 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-545f68d885-qzl6x" podStartSLOduration=2.040913715 podStartE2EDuration="7.60177532s" totalImagesPullingTime="5.560861605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:16:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:16:24.600047276 +0000 UTC m=+316.927744816" watchObservedRunningTime="2026-07-10 17:16:24.60177532 +0000 UTC m=+316.929472858" Jul 10 17:16:24.641830 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:24.641781 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-h8qcx" podStartSLOduration=2.237271985 podStartE2EDuration="7.641764871s" totalImagesPullingTime="5.404492886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:16:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:16:24.625744062 +0000 UTC m=+316.953441602" watchObservedRunningTime="2026-07-10 17:16:24.641764871 +0000 UTC m=+316.969462410" Jul 10 17:16:24.642852 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:24.642822 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-7cxn7" podStartSLOduration=1.390906033 podStartE2EDuration="6.642813957s" totalImagesPullingTime="5.251907924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:16:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:16:24.640785904 +0000 UTC m=+316.968483444" watchObservedRunningTime="2026-07-10 17:16:24.642813957 +0000 UTC m=+316.970511495" Jul 10 17:16:25.578185 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:25.578149 2583 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-hrrs6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 17:16:25.578587 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:25.578199 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6" podUID="834b6019-1993-45d6-8a07-aeeee385304a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 17:16:28.577745 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:28.577707 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6" Jul 10 17:16:29.268601 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:29.268569 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-8htph"] Jul 10 17:16:29.271470 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:29.271451 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8htph" Jul 10 17:16:29.275176 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:29.275150 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 10 17:16:29.278895 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:29.278605 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-ghzkz\"" Jul 10 17:16:29.281768 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:29.281744 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-p7lgl"] Jul 10 17:16:29.286113 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:29.284966 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p7lgl" Jul 10 17:16:29.286113 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:29.285244 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-8htph"] Jul 10 17:16:29.287640 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:29.287618 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 10 17:16:29.288476 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:29.288457 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-4ccwg\"" Jul 10 17:16:29.304987 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:29.304956 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-p7lgl"] Jul 10 17:16:29.388675 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:29.388636 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f94sx\" (UniqueName: \"kubernetes.io/projected/9b800003-9484-432d-b004-452a75d4765b-kube-api-access-f94sx\") pod \"kserve-controller-manager-d6f4b65b8-8htph\" (UID: \"9b800003-9484-432d-b004-452a75d4765b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8htph" Jul 10 17:16:29.388853 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:29.388697 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b800003-9484-432d-b004-452a75d4765b-cert\") pod \"kserve-controller-manager-d6f4b65b8-8htph\" (UID: \"9b800003-9484-432d-b004-452a75d4765b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8htph" Jul 10 17:16:29.388853 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:29.388722 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86283d78-f4af-4cd5-aaab-a075694c5a37-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-p7lgl\" (UID: \"86283d78-f4af-4cd5-aaab-a075694c5a37\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p7lgl" Jul 10 17:16:29.388853 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:29.388757 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llqfj\" (UniqueName: \"kubernetes.io/projected/86283d78-f4af-4cd5-aaab-a075694c5a37-kube-api-access-llqfj\") pod \"llmisvc-controller-manager-777c6bd7d5-p7lgl\" (UID: \"86283d78-f4af-4cd5-aaab-a075694c5a37\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p7lgl" Jul 10 17:16:29.489609 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:29.489574 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f94sx\" (UniqueName: \"kubernetes.io/projected/9b800003-9484-432d-b004-452a75d4765b-kube-api-access-f94sx\") pod \"kserve-controller-manager-d6f4b65b8-8htph\" (UID: \"9b800003-9484-432d-b004-452a75d4765b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8htph" Jul 10 17:16:29.489790 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:29.489621 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b800003-9484-432d-b004-452a75d4765b-cert\") pod \"kserve-controller-manager-d6f4b65b8-8htph\" (UID: \"9b800003-9484-432d-b004-452a75d4765b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8htph" Jul 10 17:16:29.489790 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:29.489638 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86283d78-f4af-4cd5-aaab-a075694c5a37-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-p7lgl\" (UID: \"86283d78-f4af-4cd5-aaab-a075694c5a37\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p7lgl" Jul 10 17:16:29.489790 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:29.489671 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llqfj\" (UniqueName: \"kubernetes.io/projected/86283d78-f4af-4cd5-aaab-a075694c5a37-kube-api-access-llqfj\") pod \"llmisvc-controller-manager-777c6bd7d5-p7lgl\" (UID: \"86283d78-f4af-4cd5-aaab-a075694c5a37\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p7lgl" Jul 10 17:16:29.489790 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:16:29.489745 2583 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 10 17:16:29.489938 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:16:29.489817 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86283d78-f4af-4cd5-aaab-a075694c5a37-cert podName:86283d78-f4af-4cd5-aaab-a075694c5a37 nodeName:}" failed. No retries permitted until 2026-07-10 17:16:29.989795656 +0000 UTC m=+322.317493178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/86283d78-f4af-4cd5-aaab-a075694c5a37-cert") pod "llmisvc-controller-manager-777c6bd7d5-p7lgl" (UID: "86283d78-f4af-4cd5-aaab-a075694c5a37") : secret "llmisvc-webhook-server-cert" not found Jul 10 17:16:29.492273 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:29.492247 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b800003-9484-432d-b004-452a75d4765b-cert\") pod \"kserve-controller-manager-d6f4b65b8-8htph\" (UID: \"9b800003-9484-432d-b004-452a75d4765b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8htph" Jul 10 17:16:29.504835 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:29.504812 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llqfj\" (UniqueName: \"kubernetes.io/projected/86283d78-f4af-4cd5-aaab-a075694c5a37-kube-api-access-llqfj\") pod \"llmisvc-controller-manager-777c6bd7d5-p7lgl\" (UID: \"86283d78-f4af-4cd5-aaab-a075694c5a37\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p7lgl" Jul 10 17:16:29.504979 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:29.504836 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f94sx\" (UniqueName: \"kubernetes.io/projected/9b800003-9484-432d-b004-452a75d4765b-kube-api-access-f94sx\") pod \"kserve-controller-manager-d6f4b65b8-8htph\" (UID: \"9b800003-9484-432d-b004-452a75d4765b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-8htph" Jul 10 17:16:29.583761 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:29.583671 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8htph" Jul 10 17:16:29.729929 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:29.729892 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-8htph"] Jul 10 17:16:29.733912 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:16:29.733885 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b800003_9484_432d_b004_452a75d4765b.slice/crio-990b2829f24d905619d6e4469d021d1197c3a21224f9e5a1dde20e2dd1804795 WatchSource:0}: Error finding container 990b2829f24d905619d6e4469d021d1197c3a21224f9e5a1dde20e2dd1804795: Status 404 returned error can't find the container with id 990b2829f24d905619d6e4469d021d1197c3a21224f9e5a1dde20e2dd1804795 Jul 10 17:16:29.994989 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:29.994953 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86283d78-f4af-4cd5-aaab-a075694c5a37-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-p7lgl\" (UID: \"86283d78-f4af-4cd5-aaab-a075694c5a37\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p7lgl" Jul 10 17:16:29.995188 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:16:29.995080 2583 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 10 17:16:29.995188 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:16:29.995139 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86283d78-f4af-4cd5-aaab-a075694c5a37-cert podName:86283d78-f4af-4cd5-aaab-a075694c5a37 nodeName:}" failed. No retries permitted until 2026-07-10 17:16:30.995122779 +0000 UTC m=+323.322820298 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/86283d78-f4af-4cd5-aaab-a075694c5a37-cert") pod "llmisvc-controller-manager-777c6bd7d5-p7lgl" (UID: "86283d78-f4af-4cd5-aaab-a075694c5a37") : secret "llmisvc-webhook-server-cert" not found Jul 10 17:16:30.598221 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:30.598176 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8htph" event={"ID":"9b800003-9484-432d-b004-452a75d4765b","Type":"ContainerStarted","Data":"990b2829f24d905619d6e4469d021d1197c3a21224f9e5a1dde20e2dd1804795"} Jul 10 17:16:31.003620 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:31.003582 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86283d78-f4af-4cd5-aaab-a075694c5a37-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-p7lgl\" (UID: \"86283d78-f4af-4cd5-aaab-a075694c5a37\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p7lgl" Jul 10 17:16:31.006201 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:31.006168 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86283d78-f4af-4cd5-aaab-a075694c5a37-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-p7lgl\" (UID: \"86283d78-f4af-4cd5-aaab-a075694c5a37\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p7lgl" Jul 10 17:16:31.102911 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:31.102857 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p7lgl" Jul 10 17:16:31.270168 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:31.270143 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-p7lgl"] Jul 10 17:16:31.272277 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:16:31.272244 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod86283d78_f4af_4cd5_aaab_a075694c5a37.slice/crio-0b03cdb578927142beb0a4131d8a60b5b3c5b91791ffebcaf82a593efd42c5b5 WatchSource:0}: Error finding container 0b03cdb578927142beb0a4131d8a60b5b3c5b91791ffebcaf82a593efd42c5b5: Status 404 returned error can't find the container with id 0b03cdb578927142beb0a4131d8a60b5b3c5b91791ffebcaf82a593efd42c5b5 Jul 10 17:16:31.603870 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:31.603798 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p7lgl" event={"ID":"86283d78-f4af-4cd5-aaab-a075694c5a37","Type":"ContainerStarted","Data":"0b03cdb578927142beb0a4131d8a60b5b3c5b91791ffebcaf82a593efd42c5b5"} Jul 10 17:16:32.624742 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:32.624710 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6"] Jul 10 17:16:32.625151 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:32.624935 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6" podUID="834b6019-1993-45d6-8a07-aeeee385304a" containerName="discovery" containerID="cri-o://850853b800ea569ddd2b1dd1a01462a78e2750ca883cf7f40c3527327b3b12b0" gracePeriod=30 Jul 10 17:16:32.894715 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:32.894692 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6" Jul 10 17:16:33.022101 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:33.022007 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/834b6019-1993-45d6-8a07-aeeee385304a-istio-kubeconfig\") pod \"834b6019-1993-45d6-8a07-aeeee385304a\" (UID: \"834b6019-1993-45d6-8a07-aeeee385304a\") " Jul 10 17:16:33.022101 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:33.022068 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnsfq\" (UniqueName: \"kubernetes.io/projected/834b6019-1993-45d6-8a07-aeeee385304a-kube-api-access-gnsfq\") pod \"834b6019-1993-45d6-8a07-aeeee385304a\" (UID: \"834b6019-1993-45d6-8a07-aeeee385304a\") " Jul 10 17:16:33.022101 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:33.022094 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/834b6019-1993-45d6-8a07-aeeee385304a-cacerts\") pod \"834b6019-1993-45d6-8a07-aeeee385304a\" (UID: \"834b6019-1993-45d6-8a07-aeeee385304a\") " Jul 10 17:16:33.022373 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:33.022132 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/834b6019-1993-45d6-8a07-aeeee385304a-local-certs\") pod \"834b6019-1993-45d6-8a07-aeeee385304a\" (UID: \"834b6019-1993-45d6-8a07-aeeee385304a\") " Jul 10 17:16:33.022373 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:33.022188 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/834b6019-1993-45d6-8a07-aeeee385304a-istio-csr-dns-cert\") pod \"834b6019-1993-45d6-8a07-aeeee385304a\" (UID: \"834b6019-1993-45d6-8a07-aeeee385304a\") " Jul 10 17:16:33.022373 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:33.022226 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/834b6019-1993-45d6-8a07-aeeee385304a-istio-csr-ca-configmap\") pod \"834b6019-1993-45d6-8a07-aeeee385304a\" (UID: \"834b6019-1993-45d6-8a07-aeeee385304a\") " Jul 10 17:16:33.022373 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:33.022248 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/834b6019-1993-45d6-8a07-aeeee385304a-istio-token\") pod \"834b6019-1993-45d6-8a07-aeeee385304a\" (UID: \"834b6019-1993-45d6-8a07-aeeee385304a\") " Jul 10 17:16:33.023413 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:33.023344 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/834b6019-1993-45d6-8a07-aeeee385304a-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "834b6019-1993-45d6-8a07-aeeee385304a" (UID: "834b6019-1993-45d6-8a07-aeeee385304a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:16:33.025366 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:33.025336 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/834b6019-1993-45d6-8a07-aeeee385304a-cacerts" (OuterVolumeSpecName: "cacerts") pod "834b6019-1993-45d6-8a07-aeeee385304a" (UID: "834b6019-1993-45d6-8a07-aeeee385304a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:16:33.027170 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:33.025684 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/834b6019-1993-45d6-8a07-aeeee385304a-kube-api-access-gnsfq" (OuterVolumeSpecName: "kube-api-access-gnsfq") pod "834b6019-1993-45d6-8a07-aeeee385304a" (UID: "834b6019-1993-45d6-8a07-aeeee385304a"). InnerVolumeSpecName "kube-api-access-gnsfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:16:33.027170 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:33.025728 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/834b6019-1993-45d6-8a07-aeeee385304a-istio-token" (OuterVolumeSpecName: "istio-token") pod "834b6019-1993-45d6-8a07-aeeee385304a" (UID: "834b6019-1993-45d6-8a07-aeeee385304a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:16:33.027170 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:33.026426 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/834b6019-1993-45d6-8a07-aeeee385304a-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "834b6019-1993-45d6-8a07-aeeee385304a" (UID: "834b6019-1993-45d6-8a07-aeeee385304a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:16:33.027425 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:33.027382 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/834b6019-1993-45d6-8a07-aeeee385304a-local-certs" (OuterVolumeSpecName: "local-certs") pod "834b6019-1993-45d6-8a07-aeeee385304a" (UID: "834b6019-1993-45d6-8a07-aeeee385304a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:16:33.027998 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:33.027978 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/834b6019-1993-45d6-8a07-aeeee385304a-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "834b6019-1993-45d6-8a07-aeeee385304a" (UID: "834b6019-1993-45d6-8a07-aeeee385304a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:16:33.123463 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:33.123433 2583 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/834b6019-1993-45d6-8a07-aeeee385304a-local-certs\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:16:33.123463 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:33.123468 2583 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/834b6019-1993-45d6-8a07-aeeee385304a-istio-csr-dns-cert\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:16:33.123620 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:33.123482 2583 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/834b6019-1993-45d6-8a07-aeeee385304a-istio-csr-ca-configmap\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:16:33.123620 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:33.123496 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/834b6019-1993-45d6-8a07-aeeee385304a-istio-token\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:16:33.123620 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:33.123508 2583 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/834b6019-1993-45d6-8a07-aeeee385304a-istio-kubeconfig\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:16:33.123620 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:33.123521 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnsfq\" (UniqueName: \"kubernetes.io/projected/834b6019-1993-45d6-8a07-aeeee385304a-kube-api-access-gnsfq\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:16:33.123620 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:33.123535 2583 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/834b6019-1993-45d6-8a07-aeeee385304a-cacerts\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:16:33.615510 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:33.615472 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8htph" event={"ID":"9b800003-9484-432d-b004-452a75d4765b","Type":"ContainerStarted","Data":"851f6b16c445765a3c8dd4a14097dbc0deb70df789bcc1b7080ca1300124d413"} Jul 10 17:16:33.615702 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:33.615597 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8htph" Jul 10 17:16:33.616819 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:33.616796 2583 generic.go:356] "Generic (PLEG): container finished" podID="834b6019-1993-45d6-8a07-aeeee385304a" containerID="850853b800ea569ddd2b1dd1a01462a78e2750ca883cf7f40c3527327b3b12b0" exitCode=0 Jul 10 17:16:33.616954 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:33.616835 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6" Jul 10 17:16:33.616954 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:33.616856 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6" event={"ID":"834b6019-1993-45d6-8a07-aeeee385304a","Type":"ContainerDied","Data":"850853b800ea569ddd2b1dd1a01462a78e2750ca883cf7f40c3527327b3b12b0"} Jul 10 17:16:33.616954 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:33.616895 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6" event={"ID":"834b6019-1993-45d6-8a07-aeeee385304a","Type":"ContainerDied","Data":"c1a10f90b1c7524cb67d636f6782260d18aa4154316a33e1147e4f2e1242ef7d"} Jul 10 17:16:33.616954 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:33.616923 2583 scope.go:117] "RemoveContainer" containerID="850853b800ea569ddd2b1dd1a01462a78e2750ca883cf7f40c3527327b3b12b0" Jul 10 17:16:33.627288 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:33.627267 2583 scope.go:117] "RemoveContainer" containerID="850853b800ea569ddd2b1dd1a01462a78e2750ca883cf7f40c3527327b3b12b0" Jul 10 17:16:33.627668 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:16:33.627648 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"850853b800ea569ddd2b1dd1a01462a78e2750ca883cf7f40c3527327b3b12b0\": container with ID starting with 850853b800ea569ddd2b1dd1a01462a78e2750ca883cf7f40c3527327b3b12b0 not found: ID does not exist" containerID="850853b800ea569ddd2b1dd1a01462a78e2750ca883cf7f40c3527327b3b12b0" Jul 10 17:16:33.627736 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:33.627677 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"850853b800ea569ddd2b1dd1a01462a78e2750ca883cf7f40c3527327b3b12b0"} err="failed to get container status \"850853b800ea569ddd2b1dd1a01462a78e2750ca883cf7f40c3527327b3b12b0\": rpc error: code = NotFound desc = could not find container \"850853b800ea569ddd2b1dd1a01462a78e2750ca883cf7f40c3527327b3b12b0\": container with ID starting with 850853b800ea569ddd2b1dd1a01462a78e2750ca883cf7f40c3527327b3b12b0 not found: ID does not exist" Jul 10 17:16:33.633636 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:33.633589 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8htph" podStartSLOduration=1.877998076 podStartE2EDuration="4.633574523s" totalImagesPullingTime="2.755576447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:16:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:16:33.632138901 +0000 UTC m=+325.959836442" watchObservedRunningTime="2026-07-10 17:16:33.633574523 +0000 UTC m=+325.961272061" Jul 10 17:16:33.647352 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:33.647327 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6"] Jul 10 17:16:33.654769 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:33.654750 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-hrrs6"] Jul 10 17:16:34.207036 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:34.207003 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="834b6019-1993-45d6-8a07-aeeee385304a" path="/var/lib/kubelet/pods/834b6019-1993-45d6-8a07-aeeee385304a/volumes" Jul 10 17:16:35.582566 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:35.582545 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-7cxn7" Jul 10 17:16:35.592897 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:35.592875 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-h8qcx" Jul 10 17:16:35.630723 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:35.630685 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p7lgl" event={"ID":"86283d78-f4af-4cd5-aaab-a075694c5a37","Type":"ContainerStarted","Data":"27dcdbbb95bd40c282ac9599439c851ff5bba41207f3192b2824b4a0bde6ab52"} Jul 10 17:16:35.630877 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:35.630763 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p7lgl" Jul 10 17:16:35.653322 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:16:35.653234 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p7lgl" podStartSLOduration=2.365176165 podStartE2EDuration="6.648129048s" totalImagesPullingTime="4.282952883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:16:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:16:35.646714897 +0000 UTC m=+327.974412444" watchObservedRunningTime="2026-07-10 17:16:35.648129048 +0000 UTC m=+327.975826584" Jul 10 17:17:04.628677 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:04.628593 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-8htph" Jul 10 17:17:06.637786 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:06.637751 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-p7lgl" Jul 10 17:17:13.402886 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:13.402854 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hsc2f"] Jul 10 17:17:13.403277 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:13.403191 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="834b6019-1993-45d6-8a07-aeeee385304a" containerName="discovery" Jul 10 17:17:13.403277 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:13.403210 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="834b6019-1993-45d6-8a07-aeeee385304a" containerName="discovery" Jul 10 17:17:13.403352 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:13.403299 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="834b6019-1993-45d6-8a07-aeeee385304a" containerName="discovery" Jul 10 17:17:13.408186 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:13.408169 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hsc2f" Jul 10 17:17:13.411828 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:13.411807 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-4856q\"" Jul 10 17:17:13.411828 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:13.411821 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 17:17:13.412019 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:13.411831 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 17:17:13.414972 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:13.414953 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hsc2f"] Jul 10 17:17:13.553170 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:13.553136 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sdc8\" (UniqueName: \"kubernetes.io/projected/3e29702f-d085-4e99-bf69-776613034f5d-kube-api-access-7sdc8\") pod \"kuadrant-operator-catalog-hsc2f\" (UID: \"3e29702f-d085-4e99-bf69-776613034f5d\") " pod="kuadrant-system/kuadrant-operator-catalog-hsc2f" Jul 10 17:17:13.653645 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:13.653568 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sdc8\" (UniqueName: \"kubernetes.io/projected/3e29702f-d085-4e99-bf69-776613034f5d-kube-api-access-7sdc8\") pod \"kuadrant-operator-catalog-hsc2f\" (UID: \"3e29702f-d085-4e99-bf69-776613034f5d\") " pod="kuadrant-system/kuadrant-operator-catalog-hsc2f" Jul 10 17:17:13.661785 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:13.661759 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sdc8\" (UniqueName: \"kubernetes.io/projected/3e29702f-d085-4e99-bf69-776613034f5d-kube-api-access-7sdc8\") pod \"kuadrant-operator-catalog-hsc2f\" (UID: \"3e29702f-d085-4e99-bf69-776613034f5d\") " pod="kuadrant-system/kuadrant-operator-catalog-hsc2f" Jul 10 17:17:13.719904 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:13.719877 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hsc2f" Jul 10 17:17:13.767239 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:13.767202 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hsc2f"] Jul 10 17:17:13.850607 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:13.850538 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hsc2f"] Jul 10 17:17:13.853349 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:17:13.853321 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e29702f_d085_4e99_bf69_776613034f5d.slice/crio-d4e5240746621525326221e56b567781d03972c60f18db062bb9fd364a047df3 WatchSource:0}: Error finding container d4e5240746621525326221e56b567781d03972c60f18db062bb9fd364a047df3: Status 404 returned error can't find the container with id d4e5240746621525326221e56b567781d03972c60f18db062bb9fd364a047df3 Jul 10 17:17:13.978285 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:13.976512 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rf9pk"] Jul 10 17:17:13.981247 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:13.981224 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rf9pk" Jul 10 17:17:13.986175 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:13.986151 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rf9pk"] Jul 10 17:17:14.056521 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:14.056484 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzv6x\" (UniqueName: \"kubernetes.io/projected/88d0b0b6-91bf-4f00-be4c-6c8191e5176e-kube-api-access-pzv6x\") pod \"kuadrant-operator-catalog-rf9pk\" (UID: \"88d0b0b6-91bf-4f00-be4c-6c8191e5176e\") " pod="kuadrant-system/kuadrant-operator-catalog-rf9pk" Jul 10 17:17:14.158019 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:14.157981 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzv6x\" (UniqueName: \"kubernetes.io/projected/88d0b0b6-91bf-4f00-be4c-6c8191e5176e-kube-api-access-pzv6x\") pod \"kuadrant-operator-catalog-rf9pk\" (UID: \"88d0b0b6-91bf-4f00-be4c-6c8191e5176e\") " pod="kuadrant-system/kuadrant-operator-catalog-rf9pk" Jul 10 17:17:14.170114 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:14.170082 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzv6x\" (UniqueName: \"kubernetes.io/projected/88d0b0b6-91bf-4f00-be4c-6c8191e5176e-kube-api-access-pzv6x\") pod \"kuadrant-operator-catalog-rf9pk\" (UID: \"88d0b0b6-91bf-4f00-be4c-6c8191e5176e\") " pod="kuadrant-system/kuadrant-operator-catalog-rf9pk" Jul 10 17:17:14.291747 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:14.291656 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rf9pk" Jul 10 17:17:14.440246 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:14.440217 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rf9pk"] Jul 10 17:17:14.443002 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:17:14.442975 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d0b0b6_91bf_4f00_be4c_6c8191e5176e.slice/crio-1b957e475f66228b067c8e42097ab2030fd7d0ab8be4a65a68cd5ea8d2e9df40 WatchSource:0}: Error finding container 1b957e475f66228b067c8e42097ab2030fd7d0ab8be4a65a68cd5ea8d2e9df40: Status 404 returned error can't find the container with id 1b957e475f66228b067c8e42097ab2030fd7d0ab8be4a65a68cd5ea8d2e9df40 Jul 10 17:17:14.806010 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:14.805967 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rf9pk" event={"ID":"88d0b0b6-91bf-4f00-be4c-6c8191e5176e","Type":"ContainerStarted","Data":"1b957e475f66228b067c8e42097ab2030fd7d0ab8be4a65a68cd5ea8d2e9df40"} Jul 10 17:17:14.807192 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:14.807164 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hsc2f" event={"ID":"3e29702f-d085-4e99-bf69-776613034f5d","Type":"ContainerStarted","Data":"d4e5240746621525326221e56b567781d03972c60f18db062bb9fd364a047df3"} Jul 10 17:17:16.817775 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:16.817730 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hsc2f" event={"ID":"3e29702f-d085-4e99-bf69-776613034f5d","Type":"ContainerStarted","Data":"8332f8a8ce8e9e20e86783b11aaa7db3c27c72b9ace2af2fa8516e540e16bf6c"} Jul 10 17:17:16.818254 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:16.817805 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-hsc2f" podUID="3e29702f-d085-4e99-bf69-776613034f5d" containerName="registry-server" containerID="cri-o://8332f8a8ce8e9e20e86783b11aaa7db3c27c72b9ace2af2fa8516e540e16bf6c" gracePeriod=2 Jul 10 17:17:16.819164 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:16.819140 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rf9pk" event={"ID":"88d0b0b6-91bf-4f00-be4c-6c8191e5176e","Type":"ContainerStarted","Data":"60c63852bab1e30a0a08f1c955a5d7eaa7894d493677acffec9f9022405e433a"} Jul 10 17:17:16.833712 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:16.833670 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-hsc2f" podStartSLOduration=1.5897177679999999 podStartE2EDuration="3.833654302s" totalImagesPullingTime="2.243936534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:17:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:17:16.832109293 +0000 UTC m=+369.159806832" watchObservedRunningTime="2026-07-10 17:17:16.833654302 +0000 UTC m=+369.161351840" Jul 10 17:17:16.847748 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:16.847705 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-rf9pk" podStartSLOduration=2.197545053 podStartE2EDuration="3.847689678s" totalImagesPullingTime="1.650144625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:17:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:17:16.846059445 +0000 UTC m=+369.173756982" watchObservedRunningTime="2026-07-10 17:17:16.847689678 +0000 UTC m=+369.175387217" Jul 10 17:17:17.053077 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:17.053050 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hsc2f" Jul 10 17:17:17.183968 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:17.183934 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sdc8\" (UniqueName: \"kubernetes.io/projected/3e29702f-d085-4e99-bf69-776613034f5d-kube-api-access-7sdc8\") pod \"3e29702f-d085-4e99-bf69-776613034f5d\" (UID: \"3e29702f-d085-4e99-bf69-776613034f5d\") " Jul 10 17:17:17.186310 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:17.186284 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e29702f-d085-4e99-bf69-776613034f5d-kube-api-access-7sdc8" (OuterVolumeSpecName: "kube-api-access-7sdc8") pod "3e29702f-d085-4e99-bf69-776613034f5d" (UID: "3e29702f-d085-4e99-bf69-776613034f5d"). InnerVolumeSpecName "kube-api-access-7sdc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:17:17.285042 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:17.284990 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sdc8\" (UniqueName: \"kubernetes.io/projected/3e29702f-d085-4e99-bf69-776613034f5d-kube-api-access-7sdc8\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:17:17.825123 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:17.825087 2583 generic.go:356] "Generic (PLEG): container finished" podID="3e29702f-d085-4e99-bf69-776613034f5d" containerID="8332f8a8ce8e9e20e86783b11aaa7db3c27c72b9ace2af2fa8516e540e16bf6c" exitCode=0 Jul 10 17:17:17.825629 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:17.825133 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hsc2f" Jul 10 17:17:17.825629 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:17.825173 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hsc2f" event={"ID":"3e29702f-d085-4e99-bf69-776613034f5d","Type":"ContainerDied","Data":"8332f8a8ce8e9e20e86783b11aaa7db3c27c72b9ace2af2fa8516e540e16bf6c"} Jul 10 17:17:17.825629 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:17.825232 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hsc2f" event={"ID":"3e29702f-d085-4e99-bf69-776613034f5d","Type":"ContainerDied","Data":"d4e5240746621525326221e56b567781d03972c60f18db062bb9fd364a047df3"} Jul 10 17:17:17.825629 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:17.825258 2583 scope.go:117] "RemoveContainer" containerID="8332f8a8ce8e9e20e86783b11aaa7db3c27c72b9ace2af2fa8516e540e16bf6c" Jul 10 17:17:17.834045 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:17.834028 2583 scope.go:117] "RemoveContainer" containerID="8332f8a8ce8e9e20e86783b11aaa7db3c27c72b9ace2af2fa8516e540e16bf6c" Jul 10 17:17:17.834304 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:17:17.834283 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8332f8a8ce8e9e20e86783b11aaa7db3c27c72b9ace2af2fa8516e540e16bf6c\": container with ID starting with 8332f8a8ce8e9e20e86783b11aaa7db3c27c72b9ace2af2fa8516e540e16bf6c not found: ID does not exist" containerID="8332f8a8ce8e9e20e86783b11aaa7db3c27c72b9ace2af2fa8516e540e16bf6c" Jul 10 17:17:17.834363 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:17.834317 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8332f8a8ce8e9e20e86783b11aaa7db3c27c72b9ace2af2fa8516e540e16bf6c"} err="failed to get container status \"8332f8a8ce8e9e20e86783b11aaa7db3c27c72b9ace2af2fa8516e540e16bf6c\": rpc error: code = NotFound desc = could not find container \"8332f8a8ce8e9e20e86783b11aaa7db3c27c72b9ace2af2fa8516e540e16bf6c\": container with ID starting with 8332f8a8ce8e9e20e86783b11aaa7db3c27c72b9ace2af2fa8516e540e16bf6c not found: ID does not exist" Jul 10 17:17:17.844948 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:17.844926 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hsc2f"] Jul 10 17:17:17.850222 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:17.850199 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hsc2f"] Jul 10 17:17:18.205813 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:18.205784 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e29702f-d085-4e99-bf69-776613034f5d" path="/var/lib/kubelet/pods/3e29702f-d085-4e99-bf69-776613034f5d/volumes" Jul 10 17:17:24.292344 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:24.292307 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-rf9pk" Jul 10 17:17:24.292778 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:24.292564 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-rf9pk" Jul 10 17:17:24.315831 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:24.315803 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-rf9pk" Jul 10 17:17:24.879269 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:24.879241 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-rf9pk" Jul 10 17:17:28.809096 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:28.809062 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s"] Jul 10 17:17:28.809491 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:28.809339 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e29702f-d085-4e99-bf69-776613034f5d" containerName="registry-server" Jul 10 17:17:28.809491 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:28.809349 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e29702f-d085-4e99-bf69-776613034f5d" containerName="registry-server" Jul 10 17:17:28.809569 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:28.809507 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e29702f-d085-4e99-bf69-776613034f5d" containerName="registry-server" Jul 10 17:17:28.811503 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:28.811489 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s" Jul 10 17:17:28.813650 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:28.813624 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2qjm5\"" Jul 10 17:17:28.820045 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:28.820021 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s"] Jul 10 17:17:28.980834 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:28.980809 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-495pc\" (UniqueName: \"kubernetes.io/projected/59ca63e6-8478-4612-814f-44f062905e67-kube-api-access-495pc\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s\" (UID: \"59ca63e6-8478-4612-814f-44f062905e67\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s" Jul 10 17:17:28.981005 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:28.980870 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/59ca63e6-8478-4612-814f-44f062905e67-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s\" (UID: \"59ca63e6-8478-4612-814f-44f062905e67\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s" Jul 10 17:17:28.981005 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:28.980897 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/59ca63e6-8478-4612-814f-44f062905e67-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s\" (UID: \"59ca63e6-8478-4612-814f-44f062905e67\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s" Jul 10 17:17:29.081871 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:29.081777 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/59ca63e6-8478-4612-814f-44f062905e67-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s\" (UID: \"59ca63e6-8478-4612-814f-44f062905e67\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s" Jul 10 17:17:29.081871 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:29.081821 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/59ca63e6-8478-4612-814f-44f062905e67-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s\" (UID: \"59ca63e6-8478-4612-814f-44f062905e67\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s" Jul 10 17:17:29.082094 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:29.081874 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-495pc\" (UniqueName: \"kubernetes.io/projected/59ca63e6-8478-4612-814f-44f062905e67-kube-api-access-495pc\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s\" (UID: \"59ca63e6-8478-4612-814f-44f062905e67\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s" Jul 10 17:17:29.082146 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:29.082130 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/59ca63e6-8478-4612-814f-44f062905e67-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s\" (UID: \"59ca63e6-8478-4612-814f-44f062905e67\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s" Jul 10 17:17:29.082195 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:29.082157 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/59ca63e6-8478-4612-814f-44f062905e67-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s\" (UID: \"59ca63e6-8478-4612-814f-44f062905e67\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s" Jul 10 17:17:29.090076 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:29.090045 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-495pc\" (UniqueName: \"kubernetes.io/projected/59ca63e6-8478-4612-814f-44f062905e67-kube-api-access-495pc\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s\" (UID: \"59ca63e6-8478-4612-814f-44f062905e67\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s" Jul 10 17:17:29.120966 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:29.120942 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s" Jul 10 17:17:29.266029 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:29.265890 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s"] Jul 10 17:17:29.268799 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:17:29.268769 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ca63e6_8478_4612_814f_44f062905e67.slice/crio-913447c89ca980f92866265b1be5746886ac52dc8b2f0efe718028eb430c2216 WatchSource:0}: Error finding container 913447c89ca980f92866265b1be5746886ac52dc8b2f0efe718028eb430c2216: Status 404 returned error can't find the container with id 913447c89ca980f92866265b1be5746886ac52dc8b2f0efe718028eb430c2216 Jul 10 17:17:29.409901 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:29.409869 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb"] Jul 10 17:17:29.412078 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:29.412062 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb" Jul 10 17:17:29.420083 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:29.420058 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb"] Jul 10 17:17:29.585011 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:29.584981 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8db3a343-675d-40b5-a0a9-5a8a02374227-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb\" (UID: \"8db3a343-675d-40b5-a0a9-5a8a02374227\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb" Jul 10 17:17:29.585216 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:29.585018 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjk4g\" (UniqueName: \"kubernetes.io/projected/8db3a343-675d-40b5-a0a9-5a8a02374227-kube-api-access-rjk4g\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb\" (UID: \"8db3a343-675d-40b5-a0a9-5a8a02374227\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb" Jul 10 17:17:29.585216 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:29.585045 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8db3a343-675d-40b5-a0a9-5a8a02374227-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb\" (UID: \"8db3a343-675d-40b5-a0a9-5a8a02374227\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb" Jul 10 17:17:29.686196 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:29.686160 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8db3a343-675d-40b5-a0a9-5a8a02374227-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb\" (UID: \"8db3a343-675d-40b5-a0a9-5a8a02374227\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb" Jul 10 17:17:29.686360 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:29.686201 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjk4g\" (UniqueName: \"kubernetes.io/projected/8db3a343-675d-40b5-a0a9-5a8a02374227-kube-api-access-rjk4g\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb\" (UID: \"8db3a343-675d-40b5-a0a9-5a8a02374227\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb" Jul 10 17:17:29.686360 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:29.686228 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8db3a343-675d-40b5-a0a9-5a8a02374227-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb\" (UID: \"8db3a343-675d-40b5-a0a9-5a8a02374227\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb" Jul 10 17:17:29.686671 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:29.686650 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8db3a343-675d-40b5-a0a9-5a8a02374227-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb\" (UID: \"8db3a343-675d-40b5-a0a9-5a8a02374227\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb" Jul 10 17:17:29.686714 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:29.686661 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8db3a343-675d-40b5-a0a9-5a8a02374227-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb\" (UID: \"8db3a343-675d-40b5-a0a9-5a8a02374227\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb" Jul 10 17:17:29.694785 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:29.694759 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjk4g\" (UniqueName: \"kubernetes.io/projected/8db3a343-675d-40b5-a0a9-5a8a02374227-kube-api-access-rjk4g\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb\" (UID: \"8db3a343-675d-40b5-a0a9-5a8a02374227\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb" Jul 10 17:17:29.747806 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:29.747768 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb" Jul 10 17:17:29.877751 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:29.877691 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb"] Jul 10 17:17:29.879788 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:29.879756 2583 generic.go:356] "Generic (PLEG): container finished" podID="59ca63e6-8478-4612-814f-44f062905e67" containerID="97807566aef595b3be1c7068fa773bb41176cd9446e0df6613a28e646c7eefda" exitCode=0 Jul 10 17:17:29.879896 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:29.879804 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s" event={"ID":"59ca63e6-8478-4612-814f-44f062905e67","Type":"ContainerDied","Data":"97807566aef595b3be1c7068fa773bb41176cd9446e0df6613a28e646c7eefda"} Jul 10 17:17:29.879896 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:29.879845 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s" event={"ID":"59ca63e6-8478-4612-814f-44f062905e67","Type":"ContainerStarted","Data":"913447c89ca980f92866265b1be5746886ac52dc8b2f0efe718028eb430c2216"} Jul 10 17:17:29.880135 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:17:29.880114 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8db3a343_675d_40b5_a0a9_5a8a02374227.slice/crio-1315d25ec6e98f2b849d75357c8badb5310549ecf1c462ea4f527d135be5a13c WatchSource:0}: Error finding container 1315d25ec6e98f2b849d75357c8badb5310549ecf1c462ea4f527d135be5a13c: Status 404 returned error can't find the container with id 1315d25ec6e98f2b849d75357c8badb5310549ecf1c462ea4f527d135be5a13c Jul 10 17:17:30.010333 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.010298 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw"] Jul 10 17:17:30.012881 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.012856 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw" Jul 10 17:17:30.020708 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.020681 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw"] Jul 10 17:17:30.090712 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.090661 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d81b6b2e-0d3c-4f29-ae16-361c0b8682a6-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw\" (UID: \"d81b6b2e-0d3c-4f29-ae16-361c0b8682a6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw" Jul 10 17:17:30.090876 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.090719 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d81b6b2e-0d3c-4f29-ae16-361c0b8682a6-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw\" (UID: \"d81b6b2e-0d3c-4f29-ae16-361c0b8682a6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw" Jul 10 17:17:30.090876 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.090792 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64m5s\" (UniqueName: \"kubernetes.io/projected/d81b6b2e-0d3c-4f29-ae16-361c0b8682a6-kube-api-access-64m5s\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw\" (UID: \"d81b6b2e-0d3c-4f29-ae16-361c0b8682a6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw" Jul 10 17:17:30.191721 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.191686 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d81b6b2e-0d3c-4f29-ae16-361c0b8682a6-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw\" (UID: \"d81b6b2e-0d3c-4f29-ae16-361c0b8682a6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw" Jul 10 17:17:30.191721 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.191724 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d81b6b2e-0d3c-4f29-ae16-361c0b8682a6-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw\" (UID: \"d81b6b2e-0d3c-4f29-ae16-361c0b8682a6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw" Jul 10 17:17:30.191956 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.191758 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64m5s\" (UniqueName: \"kubernetes.io/projected/d81b6b2e-0d3c-4f29-ae16-361c0b8682a6-kube-api-access-64m5s\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw\" (UID: \"d81b6b2e-0d3c-4f29-ae16-361c0b8682a6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw" Jul 10 17:17:30.192084 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.192061 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d81b6b2e-0d3c-4f29-ae16-361c0b8682a6-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw\" (UID: \"d81b6b2e-0d3c-4f29-ae16-361c0b8682a6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw" Jul 10 17:17:30.192146 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.192097 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d81b6b2e-0d3c-4f29-ae16-361c0b8682a6-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw\" (UID: \"d81b6b2e-0d3c-4f29-ae16-361c0b8682a6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw" Jul 10 17:17:30.201920 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.201891 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64m5s\" (UniqueName: \"kubernetes.io/projected/d81b6b2e-0d3c-4f29-ae16-361c0b8682a6-kube-api-access-64m5s\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw\" (UID: \"d81b6b2e-0d3c-4f29-ae16-361c0b8682a6\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw" Jul 10 17:17:30.334190 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.334104 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw" Jul 10 17:17:30.473750 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.473725 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw"] Jul 10 17:17:30.478758 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:17:30.478725 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd81b6b2e_0d3c_4f29_ae16_361c0b8682a6.slice/crio-e73778b4cb3a8856e66a0d7a7f0fd91343722c3b2fe16fa93316d870cb4ac8c3 WatchSource:0}: Error finding container e73778b4cb3a8856e66a0d7a7f0fd91343722c3b2fe16fa93316d870cb4ac8c3: Status 404 returned error can't find the container with id e73778b4cb3a8856e66a0d7a7f0fd91343722c3b2fe16fa93316d870cb4ac8c3 Jul 10 17:17:30.612423 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.612378 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79"] Jul 10 17:17:30.615119 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.615102 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79" Jul 10 17:17:30.625049 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.625023 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79"] Jul 10 17:17:30.696912 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.696881 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjd27\" (UniqueName: \"kubernetes.io/projected/e55f2ebf-0019-408b-bf88-2739a8763f78-kube-api-access-rjd27\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79\" (UID: \"e55f2ebf-0019-408b-bf88-2739a8763f78\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79" Jul 10 17:17:30.696912 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.696924 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e55f2ebf-0019-408b-bf88-2739a8763f78-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79\" (UID: \"e55f2ebf-0019-408b-bf88-2739a8763f78\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79" Jul 10 17:17:30.697157 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.697053 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e55f2ebf-0019-408b-bf88-2739a8763f78-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79\" (UID: \"e55f2ebf-0019-408b-bf88-2739a8763f78\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79" Jul 10 17:17:30.797905 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.797864 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e55f2ebf-0019-408b-bf88-2739a8763f78-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79\" (UID: \"e55f2ebf-0019-408b-bf88-2739a8763f78\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79" Jul 10 17:17:30.798068 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.797926 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjd27\" (UniqueName: \"kubernetes.io/projected/e55f2ebf-0019-408b-bf88-2739a8763f78-kube-api-access-rjd27\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79\" (UID: \"e55f2ebf-0019-408b-bf88-2739a8763f78\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79" Jul 10 17:17:30.798068 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.797955 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e55f2ebf-0019-408b-bf88-2739a8763f78-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79\" (UID: \"e55f2ebf-0019-408b-bf88-2739a8763f78\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79" Jul 10 17:17:30.798280 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.798260 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e55f2ebf-0019-408b-bf88-2739a8763f78-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79\" (UID: \"e55f2ebf-0019-408b-bf88-2739a8763f78\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79" Jul 10 17:17:30.798330 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.798285 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e55f2ebf-0019-408b-bf88-2739a8763f78-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79\" (UID: \"e55f2ebf-0019-408b-bf88-2739a8763f78\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79" Jul 10 17:17:30.807111 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.807087 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjd27\" (UniqueName: \"kubernetes.io/projected/e55f2ebf-0019-408b-bf88-2739a8763f78-kube-api-access-rjd27\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79\" (UID: \"e55f2ebf-0019-408b-bf88-2739a8763f78\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79" Jul 10 17:17:30.888503 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.888411 2583 generic.go:356] "Generic (PLEG): container finished" podID="59ca63e6-8478-4612-814f-44f062905e67" containerID="12fd996dbda93af0a614c6c212877a0801c71b5d8284ca3b7149e19ace59ffd9" exitCode=0 Jul 10 17:17:30.888503 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.888492 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s" event={"ID":"59ca63e6-8478-4612-814f-44f062905e67","Type":"ContainerDied","Data":"12fd996dbda93af0a614c6c212877a0801c71b5d8284ca3b7149e19ace59ffd9"} Jul 10 17:17:30.890115 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.890093 2583 generic.go:356] "Generic (PLEG): container finished" podID="d81b6b2e-0d3c-4f29-ae16-361c0b8682a6" containerID="e23eb33e6de867cc870180952b8b3d5dab63a1083beabddc405616562b6391be" exitCode=0 Jul 10 17:17:30.890222 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.890179 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw" event={"ID":"d81b6b2e-0d3c-4f29-ae16-361c0b8682a6","Type":"ContainerDied","Data":"e23eb33e6de867cc870180952b8b3d5dab63a1083beabddc405616562b6391be"} Jul 10 17:17:30.890334 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.890244 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw" event={"ID":"d81b6b2e-0d3c-4f29-ae16-361c0b8682a6","Type":"ContainerStarted","Data":"e73778b4cb3a8856e66a0d7a7f0fd91343722c3b2fe16fa93316d870cb4ac8c3"} Jul 10 17:17:30.891938 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.891912 2583 generic.go:356] "Generic (PLEG): container finished" podID="8db3a343-675d-40b5-a0a9-5a8a02374227" containerID="fabb0eed26d820393c034fdba5ed9f6c207ced0631e1645fbf6b207461d69bbf" exitCode=0 Jul 10 17:17:30.892031 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.891990 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb" event={"ID":"8db3a343-675d-40b5-a0a9-5a8a02374227","Type":"ContainerDied","Data":"fabb0eed26d820393c034fdba5ed9f6c207ced0631e1645fbf6b207461d69bbf"} Jul 10 17:17:30.892078 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.892039 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb" event={"ID":"8db3a343-675d-40b5-a0a9-5a8a02374227","Type":"ContainerStarted","Data":"1315d25ec6e98f2b849d75357c8badb5310549ecf1c462ea4f527d135be5a13c"} Jul 10 17:17:30.983194 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:30.983163 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79" Jul 10 17:17:31.118294 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:31.118268 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79"] Jul 10 17:17:31.120147 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:17:31.120122 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode55f2ebf_0019_408b_bf88_2739a8763f78.slice/crio-f4f8cd58a917843698e3e998becbf8d6072b16c45f72451f32038c82337b2246 WatchSource:0}: Error finding container f4f8cd58a917843698e3e998becbf8d6072b16c45f72451f32038c82337b2246: Status 404 returned error can't find the container with id f4f8cd58a917843698e3e998becbf8d6072b16c45f72451f32038c82337b2246 Jul 10 17:17:31.898680 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:31.898637 2583 generic.go:356] "Generic (PLEG): container finished" podID="e55f2ebf-0019-408b-bf88-2739a8763f78" containerID="f049fb653fd348fe5cb1d222bdd86a5bb6fe6d74188f2b1c10945ff65c1fcd9e" exitCode=0 Jul 10 17:17:31.899105 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:31.898702 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79" event={"ID":"e55f2ebf-0019-408b-bf88-2739a8763f78","Type":"ContainerDied","Data":"f049fb653fd348fe5cb1d222bdd86a5bb6fe6d74188f2b1c10945ff65c1fcd9e"} Jul 10 17:17:31.899105 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:31.898852 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79" event={"ID":"e55f2ebf-0019-408b-bf88-2739a8763f78","Type":"ContainerStarted","Data":"f4f8cd58a917843698e3e998becbf8d6072b16c45f72451f32038c82337b2246"} Jul 10 17:17:31.901120 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:31.901083 2583 generic.go:356] "Generic (PLEG): container finished" podID="59ca63e6-8478-4612-814f-44f062905e67" containerID="92adee55fdee1be7e0120b9186e1572a5c0b3f1c889f1c4e96ec3a2a7b46e1f9" exitCode=0 Jul 10 17:17:31.901120 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:31.901120 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s" event={"ID":"59ca63e6-8478-4612-814f-44f062905e67","Type":"ContainerDied","Data":"92adee55fdee1be7e0120b9186e1572a5c0b3f1c889f1c4e96ec3a2a7b46e1f9"} Jul 10 17:17:32.909853 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:32.909815 2583 generic.go:356] "Generic (PLEG): container finished" podID="e55f2ebf-0019-408b-bf88-2739a8763f78" containerID="7f17c0e9546e2fdfec7310074144051a9eb8b60975fd44c2deb057d710ffcc07" exitCode=0 Jul 10 17:17:32.910281 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:32.909908 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79" event={"ID":"e55f2ebf-0019-408b-bf88-2739a8763f78","Type":"ContainerDied","Data":"7f17c0e9546e2fdfec7310074144051a9eb8b60975fd44c2deb057d710ffcc07"} Jul 10 17:17:32.911667 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:32.911643 2583 generic.go:356] "Generic (PLEG): container finished" podID="d81b6b2e-0d3c-4f29-ae16-361c0b8682a6" containerID="c93bebddc58a5a5cb56122e4bc0449308275e5b86d6ae11fdf4a0f0cfcbbdc8d" exitCode=0 Jul 10 17:17:32.911762 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:32.911714 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw" event={"ID":"d81b6b2e-0d3c-4f29-ae16-361c0b8682a6","Type":"ContainerDied","Data":"c93bebddc58a5a5cb56122e4bc0449308275e5b86d6ae11fdf4a0f0cfcbbdc8d"} Jul 10 17:17:32.913631 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:32.913608 2583 generic.go:356] "Generic (PLEG): container finished" podID="8db3a343-675d-40b5-a0a9-5a8a02374227" containerID="e4229433906cfeedcf673093cfa4e13e0381b12712b2bf3aabd4e91e570a28b0" exitCode=0 Jul 10 17:17:32.913710 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:32.913657 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb" event={"ID":"8db3a343-675d-40b5-a0a9-5a8a02374227","Type":"ContainerDied","Data":"e4229433906cfeedcf673093cfa4e13e0381b12712b2bf3aabd4e91e570a28b0"} Jul 10 17:17:33.059193 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:33.058944 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s" Jul 10 17:17:33.218155 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:33.218084 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-495pc\" (UniqueName: \"kubernetes.io/projected/59ca63e6-8478-4612-814f-44f062905e67-kube-api-access-495pc\") pod \"59ca63e6-8478-4612-814f-44f062905e67\" (UID: \"59ca63e6-8478-4612-814f-44f062905e67\") " Jul 10 17:17:33.218155 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:33.218131 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/59ca63e6-8478-4612-814f-44f062905e67-bundle\") pod \"59ca63e6-8478-4612-814f-44f062905e67\" (UID: \"59ca63e6-8478-4612-814f-44f062905e67\") " Jul 10 17:17:33.218301 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:33.218161 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/59ca63e6-8478-4612-814f-44f062905e67-util\") pod \"59ca63e6-8478-4612-814f-44f062905e67\" (UID: \"59ca63e6-8478-4612-814f-44f062905e67\") " Jul 10 17:17:33.218683 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:33.218657 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ca63e6-8478-4612-814f-44f062905e67-bundle" (OuterVolumeSpecName: "bundle") pod "59ca63e6-8478-4612-814f-44f062905e67" (UID: "59ca63e6-8478-4612-814f-44f062905e67"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:17:33.220235 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:33.220215 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ca63e6-8478-4612-814f-44f062905e67-kube-api-access-495pc" (OuterVolumeSpecName: "kube-api-access-495pc") pod "59ca63e6-8478-4612-814f-44f062905e67" (UID: "59ca63e6-8478-4612-814f-44f062905e67"). InnerVolumeSpecName "kube-api-access-495pc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:17:33.223456 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:33.223438 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ca63e6-8478-4612-814f-44f062905e67-util" (OuterVolumeSpecName: "util") pod "59ca63e6-8478-4612-814f-44f062905e67" (UID: "59ca63e6-8478-4612-814f-44f062905e67"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:17:33.319255 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:33.319220 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/59ca63e6-8478-4612-814f-44f062905e67-util\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:17:33.319255 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:33.319250 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-495pc\" (UniqueName: \"kubernetes.io/projected/59ca63e6-8478-4612-814f-44f062905e67-kube-api-access-495pc\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:17:33.319255 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:33.319262 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/59ca63e6-8478-4612-814f-44f062905e67-bundle\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:17:33.920560 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:33.920528 2583 generic.go:356] "Generic (PLEG): container finished" podID="8db3a343-675d-40b5-a0a9-5a8a02374227" containerID="8ef1bf2df2ca1c0a469105df8be5e15557c6cf370434a16fdb907c6a5a0f4e98" exitCode=0 Jul 10 17:17:33.920935 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:33.920615 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb" event={"ID":"8db3a343-675d-40b5-a0a9-5a8a02374227","Type":"ContainerDied","Data":"8ef1bf2df2ca1c0a469105df8be5e15557c6cf370434a16fdb907c6a5a0f4e98"} Jul 10 17:17:33.922699 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:33.922677 2583 generic.go:356] "Generic (PLEG): container finished" podID="e55f2ebf-0019-408b-bf88-2739a8763f78" containerID="acf6eceda1a65e918994fb7648e5b174b2eb89ae07c94a95f926a8a925cdf1d0" exitCode=0 Jul 10 17:17:33.922791 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:33.922755 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79" event={"ID":"e55f2ebf-0019-408b-bf88-2739a8763f78","Type":"ContainerDied","Data":"acf6eceda1a65e918994fb7648e5b174b2eb89ae07c94a95f926a8a925cdf1d0"} Jul 10 17:17:33.924559 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:33.924538 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s" event={"ID":"59ca63e6-8478-4612-814f-44f062905e67","Type":"ContainerDied","Data":"913447c89ca980f92866265b1be5746886ac52dc8b2f0efe718028eb430c2216"} Jul 10 17:17:33.924651 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:33.924576 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="913447c89ca980f92866265b1be5746886ac52dc8b2f0efe718028eb430c2216" Jul 10 17:17:33.924651 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:33.924545 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s" Jul 10 17:17:33.926616 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:33.926592 2583 generic.go:356] "Generic (PLEG): container finished" podID="d81b6b2e-0d3c-4f29-ae16-361c0b8682a6" containerID="41d222cbc574f40f8748661b81e8cba454973c38e7f6e104fa088263861db1c2" exitCode=0 Jul 10 17:17:33.926702 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:33.926661 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw" event={"ID":"d81b6b2e-0d3c-4f29-ae16-361c0b8682a6","Type":"ContainerDied","Data":"41d222cbc574f40f8748661b81e8cba454973c38e7f6e104fa088263861db1c2"} Jul 10 17:17:35.073033 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.073006 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw" Jul 10 17:17:35.116282 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.116259 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb" Jul 10 17:17:35.119002 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.118984 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79" Jul 10 17:17:35.237714 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.237600 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e55f2ebf-0019-408b-bf88-2739a8763f78-bundle\") pod \"e55f2ebf-0019-408b-bf88-2739a8763f78\" (UID: \"e55f2ebf-0019-408b-bf88-2739a8763f78\") " Jul 10 17:17:35.237714 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.237665 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d81b6b2e-0d3c-4f29-ae16-361c0b8682a6-util\") pod \"d81b6b2e-0d3c-4f29-ae16-361c0b8682a6\" (UID: \"d81b6b2e-0d3c-4f29-ae16-361c0b8682a6\") " Jul 10 17:17:35.237714 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.237702 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8db3a343-675d-40b5-a0a9-5a8a02374227-bundle\") pod \"8db3a343-675d-40b5-a0a9-5a8a02374227\" (UID: \"8db3a343-675d-40b5-a0a9-5a8a02374227\") " Jul 10 17:17:35.237984 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.237739 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e55f2ebf-0019-408b-bf88-2739a8763f78-util\") pod \"e55f2ebf-0019-408b-bf88-2739a8763f78\" (UID: \"e55f2ebf-0019-408b-bf88-2739a8763f78\") " Jul 10 17:17:35.237984 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.237797 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8db3a343-675d-40b5-a0a9-5a8a02374227-util\") pod \"8db3a343-675d-40b5-a0a9-5a8a02374227\" (UID: \"8db3a343-675d-40b5-a0a9-5a8a02374227\") " Jul 10 17:17:35.237984 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.237822 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d81b6b2e-0d3c-4f29-ae16-361c0b8682a6-bundle\") pod \"d81b6b2e-0d3c-4f29-ae16-361c0b8682a6\" (UID: \"d81b6b2e-0d3c-4f29-ae16-361c0b8682a6\") " Jul 10 17:17:35.238234 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.237852 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjk4g\" (UniqueName: \"kubernetes.io/projected/8db3a343-675d-40b5-a0a9-5a8a02374227-kube-api-access-rjk4g\") pod \"8db3a343-675d-40b5-a0a9-5a8a02374227\" (UID: \"8db3a343-675d-40b5-a0a9-5a8a02374227\") " Jul 10 17:17:35.238295 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.238271 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64m5s\" (UniqueName: \"kubernetes.io/projected/d81b6b2e-0d3c-4f29-ae16-361c0b8682a6-kube-api-access-64m5s\") pod \"d81b6b2e-0d3c-4f29-ae16-361c0b8682a6\" (UID: \"d81b6b2e-0d3c-4f29-ae16-361c0b8682a6\") " Jul 10 17:17:35.238346 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.238331 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjd27\" (UniqueName: \"kubernetes.io/projected/e55f2ebf-0019-408b-bf88-2739a8763f78-kube-api-access-rjd27\") pod \"e55f2ebf-0019-408b-bf88-2739a8763f78\" (UID: \"e55f2ebf-0019-408b-bf88-2739a8763f78\") " Jul 10 17:17:35.238846 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.238808 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e55f2ebf-0019-408b-bf88-2739a8763f78-bundle" (OuterVolumeSpecName: "bundle") pod "e55f2ebf-0019-408b-bf88-2739a8763f78" (UID: "e55f2ebf-0019-408b-bf88-2739a8763f78"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:17:35.238940 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.238866 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81b6b2e-0d3c-4f29-ae16-361c0b8682a6-bundle" (OuterVolumeSpecName: "bundle") pod "d81b6b2e-0d3c-4f29-ae16-361c0b8682a6" (UID: "d81b6b2e-0d3c-4f29-ae16-361c0b8682a6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:17:35.239230 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.239196 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8db3a343-675d-40b5-a0a9-5a8a02374227-bundle" (OuterVolumeSpecName: "bundle") pod "8db3a343-675d-40b5-a0a9-5a8a02374227" (UID: "8db3a343-675d-40b5-a0a9-5a8a02374227"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:17:35.241927 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.241897 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d81b6b2e-0d3c-4f29-ae16-361c0b8682a6-kube-api-access-64m5s" (OuterVolumeSpecName: "kube-api-access-64m5s") pod "d81b6b2e-0d3c-4f29-ae16-361c0b8682a6" (UID: "d81b6b2e-0d3c-4f29-ae16-361c0b8682a6"). InnerVolumeSpecName "kube-api-access-64m5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:17:35.242026 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.241964 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8db3a343-675d-40b5-a0a9-5a8a02374227-kube-api-access-rjk4g" (OuterVolumeSpecName: "kube-api-access-rjk4g") pod "8db3a343-675d-40b5-a0a9-5a8a02374227" (UID: "8db3a343-675d-40b5-a0a9-5a8a02374227"). InnerVolumeSpecName "kube-api-access-rjk4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:17:35.242026 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.241996 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e55f2ebf-0019-408b-bf88-2739a8763f78-kube-api-access-rjd27" (OuterVolumeSpecName: "kube-api-access-rjd27") pod "e55f2ebf-0019-408b-bf88-2739a8763f78" (UID: "e55f2ebf-0019-408b-bf88-2739a8763f78"). InnerVolumeSpecName "kube-api-access-rjd27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:17:35.247077 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.247024 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81b6b2e-0d3c-4f29-ae16-361c0b8682a6-util" (OuterVolumeSpecName: "util") pod "d81b6b2e-0d3c-4f29-ae16-361c0b8682a6" (UID: "d81b6b2e-0d3c-4f29-ae16-361c0b8682a6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:17:35.249867 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.249840 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e55f2ebf-0019-408b-bf88-2739a8763f78-util" (OuterVolumeSpecName: "util") pod "e55f2ebf-0019-408b-bf88-2739a8763f78" (UID: "e55f2ebf-0019-408b-bf88-2739a8763f78"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:17:35.252419 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.252383 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8db3a343-675d-40b5-a0a9-5a8a02374227-util" (OuterVolumeSpecName: "util") pod "8db3a343-675d-40b5-a0a9-5a8a02374227" (UID: "8db3a343-675d-40b5-a0a9-5a8a02374227"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:17:35.339524 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.339489 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d81b6b2e-0d3c-4f29-ae16-361c0b8682a6-util\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:17:35.339524 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.339517 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8db3a343-675d-40b5-a0a9-5a8a02374227-bundle\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:17:35.339524 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.339525 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e55f2ebf-0019-408b-bf88-2739a8763f78-util\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:17:35.339524 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.339532 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8db3a343-675d-40b5-a0a9-5a8a02374227-util\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:17:35.339754 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.339540 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d81b6b2e-0d3c-4f29-ae16-361c0b8682a6-bundle\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:17:35.339754 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.339549 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjk4g\" (UniqueName: \"kubernetes.io/projected/8db3a343-675d-40b5-a0a9-5a8a02374227-kube-api-access-rjk4g\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:17:35.339754 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.339559 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64m5s\" (UniqueName: \"kubernetes.io/projected/d81b6b2e-0d3c-4f29-ae16-361c0b8682a6-kube-api-access-64m5s\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:17:35.339754 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.339568 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjd27\" (UniqueName: \"kubernetes.io/projected/e55f2ebf-0019-408b-bf88-2739a8763f78-kube-api-access-rjd27\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:17:35.339754 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.339576 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e55f2ebf-0019-408b-bf88-2739a8763f78-bundle\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:17:35.937021 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.936986 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb" Jul 10 17:17:35.937021 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.936998 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb" event={"ID":"8db3a343-675d-40b5-a0a9-5a8a02374227","Type":"ContainerDied","Data":"1315d25ec6e98f2b849d75357c8badb5310549ecf1c462ea4f527d135be5a13c"} Jul 10 17:17:35.937326 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.937077 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1315d25ec6e98f2b849d75357c8badb5310549ecf1c462ea4f527d135be5a13c" Jul 10 17:17:35.938821 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.938795 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79" Jul 10 17:17:35.938947 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.938802 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79" event={"ID":"e55f2ebf-0019-408b-bf88-2739a8763f78","Type":"ContainerDied","Data":"f4f8cd58a917843698e3e998becbf8d6072b16c45f72451f32038c82337b2246"} Jul 10 17:17:35.938947 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.938915 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4f8cd58a917843698e3e998becbf8d6072b16c45f72451f32038c82337b2246" Jul 10 17:17:35.940747 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.940727 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw" event={"ID":"d81b6b2e-0d3c-4f29-ae16-361c0b8682a6","Type":"ContainerDied","Data":"e73778b4cb3a8856e66a0d7a7f0fd91343722c3b2fe16fa93316d870cb4ac8c3"} Jul 10 17:17:35.940841 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.940757 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e73778b4cb3a8856e66a0d7a7f0fd91343722c3b2fe16fa93316d870cb4ac8c3" Jul 10 17:17:35.940841 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:35.940774 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw" Jul 10 17:17:45.129685 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.129648 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-99lnw"] Jul 10 17:17:45.130051 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.129925 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8db3a343-675d-40b5-a0a9-5a8a02374227" containerName="util" Jul 10 17:17:45.130051 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.129935 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8db3a343-675d-40b5-a0a9-5a8a02374227" containerName="util" Jul 10 17:17:45.130051 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.129943 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d81b6b2e-0d3c-4f29-ae16-361c0b8682a6" containerName="pull" Jul 10 17:17:45.130051 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.129948 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d81b6b2e-0d3c-4f29-ae16-361c0b8682a6" containerName="pull" Jul 10 17:17:45.130051 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.129955 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8db3a343-675d-40b5-a0a9-5a8a02374227" containerName="pull" Jul 10 17:17:45.130051 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.129960 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8db3a343-675d-40b5-a0a9-5a8a02374227" containerName="pull" Jul 10 17:17:45.130051 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.129966 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d81b6b2e-0d3c-4f29-ae16-361c0b8682a6" containerName="util" Jul 10 17:17:45.130051 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.129970 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d81b6b2e-0d3c-4f29-ae16-361c0b8682a6" containerName="util" Jul 10 17:17:45.130051 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.129979 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e55f2ebf-0019-408b-bf88-2739a8763f78" containerName="extract" Jul 10 17:17:45.130051 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.129984 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e55f2ebf-0019-408b-bf88-2739a8763f78" containerName="extract" Jul 10 17:17:45.130051 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.129989 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d81b6b2e-0d3c-4f29-ae16-361c0b8682a6" containerName="extract" Jul 10 17:17:45.130051 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.129994 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d81b6b2e-0d3c-4f29-ae16-361c0b8682a6" containerName="extract" Jul 10 17:17:45.130051 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.130002 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59ca63e6-8478-4612-814f-44f062905e67" containerName="pull" Jul 10 17:17:45.130051 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.130007 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="59ca63e6-8478-4612-814f-44f062905e67" containerName="pull" Jul 10 17:17:45.130051 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.130012 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59ca63e6-8478-4612-814f-44f062905e67" containerName="extract" Jul 10 17:17:45.130051 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.130017 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="59ca63e6-8478-4612-814f-44f062905e67" containerName="extract" Jul 10 17:17:45.130051 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.130024 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59ca63e6-8478-4612-814f-44f062905e67" containerName="util" Jul 10 17:17:45.130051 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.130029 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="59ca63e6-8478-4612-814f-44f062905e67" containerName="util" Jul 10 17:17:45.130051 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.130033 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e55f2ebf-0019-408b-bf88-2739a8763f78" containerName="util" Jul 10 17:17:45.130051 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.130038 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e55f2ebf-0019-408b-bf88-2739a8763f78" containerName="util" Jul 10 17:17:45.130051 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.130043 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e55f2ebf-0019-408b-bf88-2739a8763f78" containerName="pull" Jul 10 17:17:45.130051 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.130048 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e55f2ebf-0019-408b-bf88-2739a8763f78" containerName="pull" Jul 10 17:17:45.130051 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.130057 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8db3a343-675d-40b5-a0a9-5a8a02374227" containerName="extract" Jul 10 17:17:45.130051 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.130062 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8db3a343-675d-40b5-a0a9-5a8a02374227" containerName="extract" Jul 10 17:17:45.130983 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.130114 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e55f2ebf-0019-408b-bf88-2739a8763f78" containerName="extract" Jul 10 17:17:45.130983 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.130120 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8db3a343-675d-40b5-a0a9-5a8a02374227" containerName="extract" Jul 10 17:17:45.130983 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.130127 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="59ca63e6-8478-4612-814f-44f062905e67" containerName="extract" Jul 10 17:17:45.130983 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.130131 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d81b6b2e-0d3c-4f29-ae16-361c0b8682a6" containerName="extract" Jul 10 17:17:45.134513 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.134495 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-99lnw" Jul 10 17:17:45.136927 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.136905 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-gwt5r\"" Jul 10 17:17:45.147346 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.147324 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-99lnw"] Jul 10 17:17:45.209575 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.209542 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p49jz\" (UniqueName: \"kubernetes.io/projected/ca7980e3-3509-4c5d-b71d-41d763d0cee7-kube-api-access-p49jz\") pod \"limitador-operator-controller-manager-859d7b78bd-99lnw\" (UID: \"ca7980e3-3509-4c5d-b71d-41d763d0cee7\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-99lnw" Jul 10 17:17:45.310082 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.310049 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p49jz\" (UniqueName: \"kubernetes.io/projected/ca7980e3-3509-4c5d-b71d-41d763d0cee7-kube-api-access-p49jz\") pod \"limitador-operator-controller-manager-859d7b78bd-99lnw\" (UID: \"ca7980e3-3509-4c5d-b71d-41d763d0cee7\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-99lnw" Jul 10 17:17:45.319599 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.319567 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p49jz\" (UniqueName: \"kubernetes.io/projected/ca7980e3-3509-4c5d-b71d-41d763d0cee7-kube-api-access-p49jz\") pod \"limitador-operator-controller-manager-859d7b78bd-99lnw\" (UID: \"ca7980e3-3509-4c5d-b71d-41d763d0cee7\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-99lnw" Jul 10 17:17:45.444449 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.444380 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-99lnw" Jul 10 17:17:45.578534 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.578491 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-99lnw"] Jul 10 17:17:45.581613 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:17:45.581582 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca7980e3_3509_4c5d_b71d_41d763d0cee7.slice/crio-87d1eb5138da00dcc4acccd149ee274dbc9ec430249075f37255504751b34db4 WatchSource:0}: Error finding container 87d1eb5138da00dcc4acccd149ee274dbc9ec430249075f37255504751b34db4: Status 404 returned error can't find the container with id 87d1eb5138da00dcc4acccd149ee274dbc9ec430249075f37255504751b34db4 Jul 10 17:17:45.986060 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:45.986028 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-99lnw" event={"ID":"ca7980e3-3509-4c5d-b71d-41d763d0cee7","Type":"ContainerStarted","Data":"87d1eb5138da00dcc4acccd149ee274dbc9ec430249075f37255504751b34db4"} Jul 10 17:17:47.997841 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:47.997798 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-99lnw" event={"ID":"ca7980e3-3509-4c5d-b71d-41d763d0cee7","Type":"ContainerStarted","Data":"52af461080b97c7c87e972583784502135f67ec91e105206db295d2e9b96858d"} Jul 10 17:17:47.998265 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:47.997919 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-99lnw" Jul 10 17:17:48.014496 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:48.014452 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-99lnw" podStartSLOduration=1.205266159 podStartE2EDuration="3.014439542s" totalImagesPullingTime="1.809173383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:17:48.01382258 +0000 UTC m=+400.341520119" watchObservedRunningTime="2026-07-10 17:17:48.014439542 +0000 UTC m=+400.342137080" Jul 10 17:17:48.601769 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:48.601741 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-b2kmw"] Jul 10 17:17:48.604907 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:48.604890 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b2kmw" Jul 10 17:17:48.607218 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:48.607199 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-txrsv\"" Jul 10 17:17:48.607218 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:48.607208 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 10 17:17:48.615287 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:48.615264 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-b2kmw"] Jul 10 17:17:48.631687 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:48.631660 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b5fg\" (UniqueName: \"kubernetes.io/projected/fd6d0d14-eced-40cb-bd67-12b7fbd9e539-kube-api-access-9b5fg\") pod \"dns-operator-controller-manager-864dd78c4c-b2kmw\" (UID: \"fd6d0d14-eced-40cb-bd67-12b7fbd9e539\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b2kmw" Jul 10 17:17:48.732980 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:48.732946 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b5fg\" (UniqueName: \"kubernetes.io/projected/fd6d0d14-eced-40cb-bd67-12b7fbd9e539-kube-api-access-9b5fg\") pod \"dns-operator-controller-manager-864dd78c4c-b2kmw\" (UID: \"fd6d0d14-eced-40cb-bd67-12b7fbd9e539\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b2kmw" Jul 10 17:17:48.742973 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:48.742948 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b5fg\" (UniqueName: \"kubernetes.io/projected/fd6d0d14-eced-40cb-bd67-12b7fbd9e539-kube-api-access-9b5fg\") pod \"dns-operator-controller-manager-864dd78c4c-b2kmw\" (UID: \"fd6d0d14-eced-40cb-bd67-12b7fbd9e539\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b2kmw" Jul 10 17:17:48.915668 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:48.915643 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b2kmw" Jul 10 17:17:49.064974 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:49.064951 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-b2kmw"] Jul 10 17:17:49.067118 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:17:49.067090 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd6d0d14_eced_40cb_bd67_12b7fbd9e539.slice/crio-b6413f2fa813338a307acea743706a7ec2bebe84b91520fecef15e4819359147 WatchSource:0}: Error finding container b6413f2fa813338a307acea743706a7ec2bebe84b91520fecef15e4819359147: Status 404 returned error can't find the container with id b6413f2fa813338a307acea743706a7ec2bebe84b91520fecef15e4819359147 Jul 10 17:17:50.011279 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:50.011184 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b2kmw" event={"ID":"fd6d0d14-eced-40cb-bd67-12b7fbd9e539","Type":"ContainerStarted","Data":"b6413f2fa813338a307acea743706a7ec2bebe84b91520fecef15e4819359147"} Jul 10 17:17:52.022817 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:52.022783 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b2kmw" event={"ID":"fd6d0d14-eced-40cb-bd67-12b7fbd9e539","Type":"ContainerStarted","Data":"e8b7e6bb430811a56a73e140cda706253646557d6c0751ab41ed8f17614a0e62"} Jul 10 17:17:52.023210 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:52.022859 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b2kmw" Jul 10 17:17:52.039903 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:52.039860 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b2kmw" podStartSLOduration=2.034521237 podStartE2EDuration="4.039845383s" totalImagesPullingTime="2.005324146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:17:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:17:52.03781826 +0000 UTC m=+404.365515798" watchObservedRunningTime="2026-07-10 17:17:52.039845383 +0000 UTC m=+404.367542922" Jul 10 17:17:53.604842 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:53.604809 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wvk2x"] Jul 10 17:17:53.608194 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:53.608178 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wvk2x" Jul 10 17:17:53.611131 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:53.611114 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-q78b9\"" Jul 10 17:17:53.620765 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:53.620741 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wvk2x"] Jul 10 17:17:53.670844 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:53.670811 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b592ade4-d6db-49b5-b5c5-3742e172990d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wvk2x\" (UID: \"b592ade4-d6db-49b5-b5c5-3742e172990d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wvk2x" Jul 10 17:17:53.670961 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:53.670873 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8kfs\" (UniqueName: \"kubernetes.io/projected/b592ade4-d6db-49b5-b5c5-3742e172990d-kube-api-access-x8kfs\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wvk2x\" (UID: \"b592ade4-d6db-49b5-b5c5-3742e172990d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wvk2x" Jul 10 17:17:53.771249 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:53.771218 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b592ade4-d6db-49b5-b5c5-3742e172990d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wvk2x\" (UID: \"b592ade4-d6db-49b5-b5c5-3742e172990d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wvk2x" Jul 10 17:17:53.771450 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:53.771281 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8kfs\" (UniqueName: \"kubernetes.io/projected/b592ade4-d6db-49b5-b5c5-3742e172990d-kube-api-access-x8kfs\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wvk2x\" (UID: \"b592ade4-d6db-49b5-b5c5-3742e172990d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wvk2x" Jul 10 17:17:53.771604 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:53.771584 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b592ade4-d6db-49b5-b5c5-3742e172990d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wvk2x\" (UID: \"b592ade4-d6db-49b5-b5c5-3742e172990d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wvk2x" Jul 10 17:17:53.788902 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:53.788879 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8kfs\" (UniqueName: \"kubernetes.io/projected/b592ade4-d6db-49b5-b5c5-3742e172990d-kube-api-access-x8kfs\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wvk2x\" (UID: \"b592ade4-d6db-49b5-b5c5-3742e172990d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wvk2x" Jul 10 17:17:53.918091 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:53.918062 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wvk2x" Jul 10 17:17:54.057719 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:54.057693 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wvk2x"] Jul 10 17:17:54.059486 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:17:54.059447 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb592ade4_d6db_49b5_b5c5_3742e172990d.slice/crio-e4b5d1b8658fc7833c886bc1f32e1f436b10382303af60ed8c4990deb800dd3c WatchSource:0}: Error finding container e4b5d1b8658fc7833c886bc1f32e1f436b10382303af60ed8c4990deb800dd3c: Status 404 returned error can't find the container with id e4b5d1b8658fc7833c886bc1f32e1f436b10382303af60ed8c4990deb800dd3c Jul 10 17:17:55.037862 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:55.037831 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wvk2x" event={"ID":"b592ade4-d6db-49b5-b5c5-3742e172990d","Type":"ContainerStarted","Data":"e4b5d1b8658fc7833c886bc1f32e1f436b10382303af60ed8c4990deb800dd3c"} Jul 10 17:17:57.314071 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:57.314041 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-vzrr7"] Jul 10 17:17:57.317258 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:57.317243 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vzrr7" Jul 10 17:17:57.319821 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:57.319797 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-db6ql\"" Jul 10 17:17:57.327476 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:57.327454 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-vzrr7"] Jul 10 17:17:57.400577 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:57.400537 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwjqr\" (UniqueName: \"kubernetes.io/projected/c7b4a154-f747-4cea-889b-cdb3adeb7b13-kube-api-access-cwjqr\") pod \"authorino-operator-75cc8d5cb7-vzrr7\" (UID: \"c7b4a154-f747-4cea-889b-cdb3adeb7b13\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-vzrr7" Jul 10 17:17:57.501925 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:57.501889 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwjqr\" (UniqueName: \"kubernetes.io/projected/c7b4a154-f747-4cea-889b-cdb3adeb7b13-kube-api-access-cwjqr\") pod \"authorino-operator-75cc8d5cb7-vzrr7\" (UID: \"c7b4a154-f747-4cea-889b-cdb3adeb7b13\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-vzrr7" Jul 10 17:17:57.513084 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:57.513056 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwjqr\" (UniqueName: \"kubernetes.io/projected/c7b4a154-f747-4cea-889b-cdb3adeb7b13-kube-api-access-cwjqr\") pod \"authorino-operator-75cc8d5cb7-vzrr7\" (UID: \"c7b4a154-f747-4cea-889b-cdb3adeb7b13\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-vzrr7" Jul 10 17:17:57.627669 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:57.627584 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vzrr7" Jul 10 17:17:57.768501 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:57.768462 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-vzrr7"] Jul 10 17:17:57.770461 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:17:57.770433 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b4a154_f747_4cea_889b_cdb3adeb7b13.slice/crio-d513ddbbbecdfc28ed81cb91d196359d9c44444d687e66bb2c53f1edc16eb32e WatchSource:0}: Error finding container d513ddbbbecdfc28ed81cb91d196359d9c44444d687e66bb2c53f1edc16eb32e: Status 404 returned error can't find the container with id d513ddbbbecdfc28ed81cb91d196359d9c44444d687e66bb2c53f1edc16eb32e Jul 10 17:17:58.054826 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:58.054792 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vzrr7" event={"ID":"c7b4a154-f747-4cea-889b-cdb3adeb7b13","Type":"ContainerStarted","Data":"d513ddbbbecdfc28ed81cb91d196359d9c44444d687e66bb2c53f1edc16eb32e"} Jul 10 17:17:59.006024 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:17:59.005884 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-99lnw" Jul 10 17:18:02.079430 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:02.079366 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wvk2x" event={"ID":"b592ade4-d6db-49b5-b5c5-3742e172990d","Type":"ContainerStarted","Data":"9561d4f41475946b67afb433f7802ab0b95256745c7e9374b3018d3b66886995"} Jul 10 17:18:02.079906 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:02.079501 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wvk2x" Jul 10 17:18:02.080882 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:02.080859 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vzrr7" event={"ID":"c7b4a154-f747-4cea-889b-cdb3adeb7b13","Type":"ContainerStarted","Data":"67bbf37b8e3c3f8c8247ca27154ba3c6535061458de496633dfefa951fa46f75"} Jul 10 17:18:02.080988 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:02.080978 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vzrr7" Jul 10 17:18:02.101284 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:02.101234 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wvk2x" podStartSLOduration=1.964973101 podStartE2EDuration="9.101217453s" totalImagesPullingTime="7.136244352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:17:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:18:02.099066585 +0000 UTC m=+414.426764123" watchObservedRunningTime="2026-07-10 17:18:02.101217453 +0000 UTC m=+414.428914992" Jul 10 17:18:02.119931 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:02.119874 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vzrr7" podStartSLOduration=1.654513404 podStartE2EDuration="5.119857773s" totalImagesPullingTime="3.465344369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:17:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:18:02.118511121 +0000 UTC m=+414.446208662" watchObservedRunningTime="2026-07-10 17:18:02.119857773 +0000 UTC m=+414.447555311" Jul 10 17:18:03.031030 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:03.030992 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-b2kmw" Jul 10 17:18:13.089681 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:13.089651 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vzrr7" Jul 10 17:18:13.090204 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:13.089817 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wvk2x" Jul 10 17:18:14.194443 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:14.194384 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2kmlh"] Jul 10 17:18:14.199637 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:14.199615 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2kmlh" Jul 10 17:18:14.210567 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:14.210535 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2kmlh"] Jul 10 17:18:14.247992 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:14.247956 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt8cg\" (UniqueName: \"kubernetes.io/projected/d79407f5-3290-4983-b14d-9a8998652457-kube-api-access-jt8cg\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2kmlh\" (UID: \"d79407f5-3290-4983-b14d-9a8998652457\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2kmlh" Jul 10 17:18:14.248184 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:14.248040 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d79407f5-3290-4983-b14d-9a8998652457-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2kmlh\" (UID: \"d79407f5-3290-4983-b14d-9a8998652457\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2kmlh" Jul 10 17:18:14.301494 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:14.301464 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2kmlh"] Jul 10 17:18:14.301729 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:18:14.301710 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-jt8cg], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2kmlh" podUID="d79407f5-3290-4983-b14d-9a8998652457" Jul 10 17:18:14.349423 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:14.349373 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d79407f5-3290-4983-b14d-9a8998652457-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2kmlh\" (UID: \"d79407f5-3290-4983-b14d-9a8998652457\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2kmlh" Jul 10 17:18:14.349620 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:14.349464 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt8cg\" (UniqueName: \"kubernetes.io/projected/d79407f5-3290-4983-b14d-9a8998652457-kube-api-access-jt8cg\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2kmlh\" (UID: \"d79407f5-3290-4983-b14d-9a8998652457\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2kmlh" Jul 10 17:18:14.349790 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:14.349770 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d79407f5-3290-4983-b14d-9a8998652457-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2kmlh\" (UID: \"d79407f5-3290-4983-b14d-9a8998652457\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2kmlh" Jul 10 17:18:14.365633 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:14.365599 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt8cg\" (UniqueName: \"kubernetes.io/projected/d79407f5-3290-4983-b14d-9a8998652457-kube-api-access-jt8cg\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2kmlh\" (UID: \"d79407f5-3290-4983-b14d-9a8998652457\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2kmlh" Jul 10 17:18:14.765459 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:14.765414 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2kmlh"] Jul 10 17:18:14.773325 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:14.773296 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2kmlh"] Jul 10 17:18:14.784074 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:14.784045 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wvk2x"] Jul 10 17:18:14.784282 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:14.784237 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wvk2x" podUID="b592ade4-d6db-49b5-b5c5-3742e172990d" containerName="manager" containerID="cri-o://9561d4f41475946b67afb433f7802ab0b95256745c7e9374b3018d3b66886995" gracePeriod=2 Jul 10 17:18:14.797476 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:14.797440 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wvk2x"] Jul 10 17:18:14.808491 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:18:14.808445 2583 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"b592ade4-d6db-49b5-b5c5-3742e172990d\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-wvk2x\": pods \"kuadrant-operator-controller-manager-8684c9ff49-wvk2x\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wvk2x" Jul 10 17:18:14.810626 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:14.810601 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgsqf"] Jul 10 17:18:14.810929 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:14.810917 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b592ade4-d6db-49b5-b5c5-3742e172990d" containerName="manager" Jul 10 17:18:14.810929 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:14.810930 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b592ade4-d6db-49b5-b5c5-3742e172990d" containerName="manager" Jul 10 17:18:14.811028 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:14.810981 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b592ade4-d6db-49b5-b5c5-3742e172990d" containerName="manager" Jul 10 17:18:14.813961 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:14.813944 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgsqf" Jul 10 17:18:14.815326 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:14.815301 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-99lnw"] Jul 10 17:18:14.815529 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:14.815506 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-99lnw" podUID="ca7980e3-3509-4c5d-b71d-41d763d0cee7" containerName="manager" containerID="cri-o://52af461080b97c7c87e972583784502135f67ec91e105206db295d2e9b96858d" gracePeriod=2 Jul 10 17:18:14.816033 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:18:14.816012 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wvk2x\" is forbidden: User \"system:node:ip-10-0-134-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-65.ec2.internal' and this object" podUID="b592ade4-d6db-49b5-b5c5-3742e172990d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wvk2x" Jul 10 17:18:14.822409 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:14.822360 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-99lnw"] Jul 10 17:18:14.840665 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:14.840634 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgsqf"] Jul 10 17:18:14.848735 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:14.848705 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gskkj"] Jul 10 17:18:14.849015 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:14.849000 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca7980e3-3509-4c5d-b71d-41d763d0cee7" containerName="manager" Jul 10 17:18:14.849015 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:14.849014 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca7980e3-3509-4c5d-b71d-41d763d0cee7" containerName="manager" Jul 10 17:18:14.849164 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:14.849076 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca7980e3-3509-4c5d-b71d-41d763d0cee7" containerName="manager" Jul 10 17:18:14.852275 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:14.852252 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gskkj" Jul 10 17:18:14.856350 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:18:14.856308 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-99lnw\" is forbidden: User \"system:node:ip-10-0-134-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-65.ec2.internal' and this object" podUID="ca7980e3-3509-4c5d-b71d-41d763d0cee7" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-99lnw" Jul 10 17:18:14.864020 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:14.863995 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gskkj"] Jul 10 17:18:14.879476 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:18:14.879442 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wvk2x\" is forbidden: User \"system:node:ip-10-0-134-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-65.ec2.internal' and this object" podUID="b592ade4-d6db-49b5-b5c5-3742e172990d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wvk2x" Jul 10 17:18:14.953575 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:14.953536 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79hdp\" (UniqueName: \"kubernetes.io/projected/723f1d41-6b52-402f-830b-b82b9f08260f-kube-api-access-79hdp\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sgsqf\" (UID: \"723f1d41-6b52-402f-830b-b82b9f08260f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgsqf" Jul 10 17:18:14.953750 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:14.953620 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/723f1d41-6b52-402f-830b-b82b9f08260f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sgsqf\" (UID: \"723f1d41-6b52-402f-830b-b82b9f08260f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgsqf" Jul 10 17:18:14.953750 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:14.953703 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8csjc\" (UniqueName: \"kubernetes.io/projected/79ebdd43-2bce-43f5-a14f-1c3090448a38-kube-api-access-8csjc\") pod \"limitador-operator-controller-manager-859d7b78bd-gskkj\" (UID: \"79ebdd43-2bce-43f5-a14f-1c3090448a38\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gskkj" Jul 10 17:18:15.053973 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.053946 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wvk2x" Jul 10 17:18:15.054319 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.054299 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79hdp\" (UniqueName: \"kubernetes.io/projected/723f1d41-6b52-402f-830b-b82b9f08260f-kube-api-access-79hdp\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sgsqf\" (UID: \"723f1d41-6b52-402f-830b-b82b9f08260f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgsqf" Jul 10 17:18:15.054412 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.054345 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/723f1d41-6b52-402f-830b-b82b9f08260f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sgsqf\" (UID: \"723f1d41-6b52-402f-830b-b82b9f08260f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgsqf" Jul 10 17:18:15.054412 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.054377 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8csjc\" (UniqueName: \"kubernetes.io/projected/79ebdd43-2bce-43f5-a14f-1c3090448a38-kube-api-access-8csjc\") pod \"limitador-operator-controller-manager-859d7b78bd-gskkj\" (UID: \"79ebdd43-2bce-43f5-a14f-1c3090448a38\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gskkj" Jul 10 17:18:15.054747 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.054729 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/723f1d41-6b52-402f-830b-b82b9f08260f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sgsqf\" (UID: \"723f1d41-6b52-402f-830b-b82b9f08260f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgsqf" Jul 10 17:18:15.056039 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:18:15.056015 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-99lnw\" is forbidden: User \"system:node:ip-10-0-134-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-65.ec2.internal' and this object" podUID="ca7980e3-3509-4c5d-b71d-41d763d0cee7" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-99lnw" Jul 10 17:18:15.057768 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:18:15.057733 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wvk2x\" is forbidden: User \"system:node:ip-10-0-134-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-65.ec2.internal' and this object" podUID="b592ade4-d6db-49b5-b5c5-3742e172990d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wvk2x" Jul 10 17:18:15.063247 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.063225 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8csjc\" (UniqueName: \"kubernetes.io/projected/79ebdd43-2bce-43f5-a14f-1c3090448a38-kube-api-access-8csjc\") pod \"limitador-operator-controller-manager-859d7b78bd-gskkj\" (UID: \"79ebdd43-2bce-43f5-a14f-1c3090448a38\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gskkj" Jul 10 17:18:15.063542 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.063523 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79hdp\" (UniqueName: \"kubernetes.io/projected/723f1d41-6b52-402f-830b-b82b9f08260f-kube-api-access-79hdp\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sgsqf\" (UID: \"723f1d41-6b52-402f-830b-b82b9f08260f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgsqf" Jul 10 17:18:15.065168 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.065150 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-99lnw" Jul 10 17:18:15.067043 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:18:15.067012 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wvk2x\" is forbidden: User \"system:node:ip-10-0-134-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-65.ec2.internal' and this object" podUID="b592ade4-d6db-49b5-b5c5-3742e172990d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wvk2x" Jul 10 17:18:15.068635 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:18:15.068600 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-99lnw\" is forbidden: User \"system:node:ip-10-0-134-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-65.ec2.internal' and this object" podUID="ca7980e3-3509-4c5d-b71d-41d763d0cee7" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-99lnw" Jul 10 17:18:15.154764 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.154719 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p49jz\" (UniqueName: \"kubernetes.io/projected/ca7980e3-3509-4c5d-b71d-41d763d0cee7-kube-api-access-p49jz\") pod \"ca7980e3-3509-4c5d-b71d-41d763d0cee7\" (UID: \"ca7980e3-3509-4c5d-b71d-41d763d0cee7\") " Jul 10 17:18:15.154946 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.154801 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8kfs\" (UniqueName: \"kubernetes.io/projected/b592ade4-d6db-49b5-b5c5-3742e172990d-kube-api-access-x8kfs\") pod \"b592ade4-d6db-49b5-b5c5-3742e172990d\" (UID: \"b592ade4-d6db-49b5-b5c5-3742e172990d\") " Jul 10 17:18:15.154946 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.154893 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b592ade4-d6db-49b5-b5c5-3742e172990d-extensions-socket-volume\") pod \"b592ade4-d6db-49b5-b5c5-3742e172990d\" (UID: \"b592ade4-d6db-49b5-b5c5-3742e172990d\") " Jul 10 17:18:15.155323 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.155289 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b592ade4-d6db-49b5-b5c5-3742e172990d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b592ade4-d6db-49b5-b5c5-3742e172990d" (UID: "b592ade4-d6db-49b5-b5c5-3742e172990d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:18:15.156995 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.156967 2583 generic.go:356] "Generic (PLEG): container finished" podID="ca7980e3-3509-4c5d-b71d-41d763d0cee7" containerID="52af461080b97c7c87e972583784502135f67ec91e105206db295d2e9b96858d" exitCode=0 Jul 10 17:18:15.157102 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.157036 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-99lnw" Jul 10 17:18:15.157102 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.157069 2583 scope.go:117] "RemoveContainer" containerID="52af461080b97c7c87e972583784502135f67ec91e105206db295d2e9b96858d" Jul 10 17:18:15.157284 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.157256 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b592ade4-d6db-49b5-b5c5-3742e172990d-kube-api-access-x8kfs" (OuterVolumeSpecName: "kube-api-access-x8kfs") pod "b592ade4-d6db-49b5-b5c5-3742e172990d" (UID: "b592ade4-d6db-49b5-b5c5-3742e172990d"). InnerVolumeSpecName "kube-api-access-x8kfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:18:15.157458 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.157295 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca7980e3-3509-4c5d-b71d-41d763d0cee7-kube-api-access-p49jz" (OuterVolumeSpecName: "kube-api-access-p49jz") pod "ca7980e3-3509-4c5d-b71d-41d763d0cee7" (UID: "ca7980e3-3509-4c5d-b71d-41d763d0cee7"). InnerVolumeSpecName "kube-api-access-p49jz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:18:15.158467 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.158444 2583 generic.go:356] "Generic (PLEG): container finished" podID="b592ade4-d6db-49b5-b5c5-3742e172990d" containerID="9561d4f41475946b67afb433f7802ab0b95256745c7e9374b3018d3b66886995" exitCode=0 Jul 10 17:18:15.158556 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.158490 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wvk2x" Jul 10 17:18:15.158673 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.158660 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2kmlh" Jul 10 17:18:15.159177 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:18:15.159157 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wvk2x\" is forbidden: User \"system:node:ip-10-0-134-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-65.ec2.internal' and this object" podUID="b592ade4-d6db-49b5-b5c5-3742e172990d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wvk2x" Jul 10 17:18:15.160863 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:18:15.160841 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-99lnw\" is forbidden: User \"system:node:ip-10-0-134-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-65.ec2.internal' and this object" podUID="ca7980e3-3509-4c5d-b71d-41d763d0cee7" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-99lnw" Jul 10 17:18:15.163313 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:18:15.163288 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-99lnw\" is forbidden: User \"system:node:ip-10-0-134-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-65.ec2.internal' and this object" podUID="ca7980e3-3509-4c5d-b71d-41d763d0cee7" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-99lnw" Jul 10 17:18:15.165130 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:18:15.165099 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-2kmlh\" is forbidden: User \"system:node:ip-10-0-134-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-65.ec2.internal' and this object" podUID="d79407f5-3290-4983-b14d-9a8998652457" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2kmlh" Jul 10 17:18:15.165773 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.165759 2583 scope.go:117] "RemoveContainer" containerID="52af461080b97c7c87e972583784502135f67ec91e105206db295d2e9b96858d" Jul 10 17:18:15.166132 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:18:15.166106 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52af461080b97c7c87e972583784502135f67ec91e105206db295d2e9b96858d\": container with ID starting with 52af461080b97c7c87e972583784502135f67ec91e105206db295d2e9b96858d not found: ID does not exist" containerID="52af461080b97c7c87e972583784502135f67ec91e105206db295d2e9b96858d" Jul 10 17:18:15.166200 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.166144 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52af461080b97c7c87e972583784502135f67ec91e105206db295d2e9b96858d"} err="failed to get container status \"52af461080b97c7c87e972583784502135f67ec91e105206db295d2e9b96858d\": rpc error: code = NotFound desc = could not find container \"52af461080b97c7c87e972583784502135f67ec91e105206db295d2e9b96858d\": container with ID starting with 52af461080b97c7c87e972583784502135f67ec91e105206db295d2e9b96858d not found: ID does not exist" Jul 10 17:18:15.166200 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.166167 2583 scope.go:117] "RemoveContainer" containerID="9561d4f41475946b67afb433f7802ab0b95256745c7e9374b3018d3b66886995" Jul 10 17:18:15.166746 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:18:15.166720 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wvk2x\" is forbidden: User \"system:node:ip-10-0-134-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-65.ec2.internal' and this object" podUID="b592ade4-d6db-49b5-b5c5-3742e172990d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wvk2x" Jul 10 17:18:15.167003 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.166985 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2kmlh" Jul 10 17:18:15.168300 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:18:15.168283 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wvk2x\" is forbidden: User \"system:node:ip-10-0-134-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-65.ec2.internal' and this object" podUID="b592ade4-d6db-49b5-b5c5-3742e172990d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wvk2x" Jul 10 17:18:15.169898 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:18:15.169872 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-99lnw\" is forbidden: User \"system:node:ip-10-0-134-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-65.ec2.internal' and this object" podUID="ca7980e3-3509-4c5d-b71d-41d763d0cee7" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-99lnw" Jul 10 17:18:15.171581 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:18:15.171561 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-2kmlh\" is forbidden: User \"system:node:ip-10-0-134-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-65.ec2.internal' and this object" podUID="d79407f5-3290-4983-b14d-9a8998652457" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2kmlh" Jul 10 17:18:15.173427 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:18:15.173409 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wvk2x\" is forbidden: User \"system:node:ip-10-0-134-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-65.ec2.internal' and this object" podUID="b592ade4-d6db-49b5-b5c5-3742e172990d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wvk2x" Jul 10 17:18:15.173994 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.173975 2583 scope.go:117] "RemoveContainer" containerID="9561d4f41475946b67afb433f7802ab0b95256745c7e9374b3018d3b66886995" Jul 10 17:18:15.174238 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:18:15.174210 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9561d4f41475946b67afb433f7802ab0b95256745c7e9374b3018d3b66886995\": container with ID starting with 9561d4f41475946b67afb433f7802ab0b95256745c7e9374b3018d3b66886995 not found: ID does not exist" containerID="9561d4f41475946b67afb433f7802ab0b95256745c7e9374b3018d3b66886995" Jul 10 17:18:15.174310 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.174237 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9561d4f41475946b67afb433f7802ab0b95256745c7e9374b3018d3b66886995"} err="failed to get container status \"9561d4f41475946b67afb433f7802ab0b95256745c7e9374b3018d3b66886995\": rpc error: code = NotFound desc = could not find container \"9561d4f41475946b67afb433f7802ab0b95256745c7e9374b3018d3b66886995\": container with ID starting with 9561d4f41475946b67afb433f7802ab0b95256745c7e9374b3018d3b66886995 not found: ID does not exist" Jul 10 17:18:15.175098 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:18:15.175068 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-99lnw\" is forbidden: User \"system:node:ip-10-0-134-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-65.ec2.internal' and this object" podUID="ca7980e3-3509-4c5d-b71d-41d763d0cee7" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-99lnw" Jul 10 17:18:15.176667 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:18:15.176652 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-2kmlh\" is forbidden: User \"system:node:ip-10-0-134-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-65.ec2.internal' and this object" podUID="d79407f5-3290-4983-b14d-9a8998652457" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2kmlh" Jul 10 17:18:15.233140 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.233112 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgsqf" Jul 10 17:18:15.242531 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.242506 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gskkj" Jul 10 17:18:15.255570 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.255545 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jt8cg\" (UniqueName: \"kubernetes.io/projected/d79407f5-3290-4983-b14d-9a8998652457-kube-api-access-jt8cg\") pod \"d79407f5-3290-4983-b14d-9a8998652457\" (UID: \"d79407f5-3290-4983-b14d-9a8998652457\") " Jul 10 17:18:15.255671 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.255588 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d79407f5-3290-4983-b14d-9a8998652457-extensions-socket-volume\") pod \"d79407f5-3290-4983-b14d-9a8998652457\" (UID: \"d79407f5-3290-4983-b14d-9a8998652457\") " Jul 10 17:18:15.255866 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.255847 2583 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b592ade4-d6db-49b5-b5c5-3742e172990d-extensions-socket-volume\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:18:15.255970 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.255871 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p49jz\" (UniqueName: \"kubernetes.io/projected/ca7980e3-3509-4c5d-b71d-41d763d0cee7-kube-api-access-p49jz\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:18:15.255970 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.255885 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8kfs\" (UniqueName: \"kubernetes.io/projected/b592ade4-d6db-49b5-b5c5-3742e172990d-kube-api-access-x8kfs\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:18:15.255970 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.255936 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d79407f5-3290-4983-b14d-9a8998652457-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d79407f5-3290-4983-b14d-9a8998652457" (UID: "d79407f5-3290-4983-b14d-9a8998652457"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:18:15.258585 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.258562 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d79407f5-3290-4983-b14d-9a8998652457-kube-api-access-jt8cg" (OuterVolumeSpecName: "kube-api-access-jt8cg") pod "d79407f5-3290-4983-b14d-9a8998652457" (UID: "d79407f5-3290-4983-b14d-9a8998652457"). InnerVolumeSpecName "kube-api-access-jt8cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:18:15.357238 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.357214 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt8cg\" (UniqueName: \"kubernetes.io/projected/d79407f5-3290-4983-b14d-9a8998652457-kube-api-access-jt8cg\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:18:15.357346 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.357243 2583 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d79407f5-3290-4983-b14d-9a8998652457-extensions-socket-volume\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:18:15.392623 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.392597 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgsqf"] Jul 10 17:18:15.395789 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:18:15.395758 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod723f1d41_6b52_402f_830b_b82b9f08260f.slice/crio-a1999834cc895fea449fd5f9ec3daf454a6c04da9ed009eeb5aa8ea38301712a WatchSource:0}: Error finding container a1999834cc895fea449fd5f9ec3daf454a6c04da9ed009eeb5aa8ea38301712a: Status 404 returned error can't find the container with id a1999834cc895fea449fd5f9ec3daf454a6c04da9ed009eeb5aa8ea38301712a Jul 10 17:18:15.414892 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:15.414856 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gskkj"] Jul 10 17:18:15.416007 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:18:15.415982 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ebdd43_2bce_43f5_a14f_1c3090448a38.slice/crio-ff6ec39f673c1c4906922a1efa522fbe61f1a92b60d58361301d6b2da6c69a60 WatchSource:0}: Error finding container ff6ec39f673c1c4906922a1efa522fbe61f1a92b60d58361301d6b2da6c69a60: Status 404 returned error can't find the container with id ff6ec39f673c1c4906922a1efa522fbe61f1a92b60d58361301d6b2da6c69a60 Jul 10 17:18:15.468142 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:18:15.468112 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wvk2x\" is forbidden: User \"system:node:ip-10-0-134-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-65.ec2.internal' and this object" podUID="b592ade4-d6db-49b5-b5c5-3742e172990d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wvk2x" Jul 10 17:18:15.469821 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:18:15.469792 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-99lnw\" is forbidden: User \"system:node:ip-10-0-134-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-65.ec2.internal' and this object" podUID="ca7980e3-3509-4c5d-b71d-41d763d0cee7" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-99lnw" Jul 10 17:18:15.471575 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:18:15.471551 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-2kmlh\" is forbidden: User \"system:node:ip-10-0-134-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-65.ec2.internal' and this object" podUID="d79407f5-3290-4983-b14d-9a8998652457" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2kmlh" Jul 10 17:18:16.167763 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:16.167717 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gskkj" event={"ID":"79ebdd43-2bce-43f5-a14f-1c3090448a38","Type":"ContainerStarted","Data":"ca2710d599fa4ef344506987f27632dc0fdbe1e407768c3c295c2f4319015fb4"} Jul 10 17:18:16.167962 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:16.167904 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gskkj" Jul 10 17:18:16.167962 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:16.167931 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gskkj" event={"ID":"79ebdd43-2bce-43f5-a14f-1c3090448a38","Type":"ContainerStarted","Data":"ff6ec39f673c1c4906922a1efa522fbe61f1a92b60d58361301d6b2da6c69a60"} Jul 10 17:18:16.169356 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:16.169325 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgsqf" event={"ID":"723f1d41-6b52-402f-830b-b82b9f08260f","Type":"ContainerStarted","Data":"284e746d826a153f907e946ece3fa29c9756686b26c0fc82def2a9c73005fb0f"} Jul 10 17:18:16.169593 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:16.169384 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgsqf" event={"ID":"723f1d41-6b52-402f-830b-b82b9f08260f","Type":"ContainerStarted","Data":"a1999834cc895fea449fd5f9ec3daf454a6c04da9ed009eeb5aa8ea38301712a"} Jul 10 17:18:16.169593 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:16.169339 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2kmlh" Jul 10 17:18:16.169593 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:16.169431 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgsqf" Jul 10 17:18:16.169956 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:18:16.169927 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wvk2x\" is forbidden: User \"system:node:ip-10-0-134-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-65.ec2.internal' and this object" podUID="b592ade4-d6db-49b5-b5c5-3742e172990d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wvk2x" Jul 10 17:18:16.171713 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:18:16.171684 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-99lnw\" is forbidden: User \"system:node:ip-10-0-134-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-65.ec2.internal' and this object" podUID="ca7980e3-3509-4c5d-b71d-41d763d0cee7" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-99lnw" Jul 10 17:18:16.173378 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:18:16.173353 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-2kmlh\" is forbidden: User \"system:node:ip-10-0-134-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-65.ec2.internal' and this object" podUID="d79407f5-3290-4983-b14d-9a8998652457" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2kmlh" Jul 10 17:18:16.193365 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:18:16.193322 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wvk2x\" is forbidden: User \"system:node:ip-10-0-134-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-65.ec2.internal' and this object" podUID="b592ade4-d6db-49b5-b5c5-3742e172990d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wvk2x" Jul 10 17:18:16.194016 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:16.193970 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gskkj" podStartSLOduration=2.193955028 podStartE2EDuration="2.193955028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:18:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:18:16.191116699 +0000 UTC m=+428.518814240" watchObservedRunningTime="2026-07-10 17:18:16.193955028 +0000 UTC m=+428.521652567" Jul 10 17:18:16.206141 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:16.206112 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b592ade4-d6db-49b5-b5c5-3742e172990d" path="/var/lib/kubelet/pods/b592ade4-d6db-49b5-b5c5-3742e172990d/volumes" Jul 10 17:18:16.206446 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:16.206433 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca7980e3-3509-4c5d-b71d-41d763d0cee7" path="/var/lib/kubelet/pods/ca7980e3-3509-4c5d-b71d-41d763d0cee7/volumes" Jul 10 17:18:16.206745 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:16.206734 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d79407f5-3290-4983-b14d-9a8998652457" path="/var/lib/kubelet/pods/d79407f5-3290-4983-b14d-9a8998652457/volumes" Jul 10 17:18:16.212915 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:16.212871 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgsqf" podStartSLOduration=2.212858546 podStartE2EDuration="2.212858546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:18:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:18:16.21204171 +0000 UTC m=+428.539739248" watchObservedRunningTime="2026-07-10 17:18:16.212858546 +0000 UTC m=+428.540556085" Jul 10 17:18:16.214151 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:18:16.214128 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-99lnw\" is forbidden: User \"system:node:ip-10-0-134-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-65.ec2.internal' and this object" podUID="ca7980e3-3509-4c5d-b71d-41d763d0cee7" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-99lnw" Jul 10 17:18:16.215770 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:18:16.215748 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-2kmlh\" is forbidden: User \"system:node:ip-10-0-134-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-65.ec2.internal' and this object" podUID="d79407f5-3290-4983-b14d-9a8998652457" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2kmlh" Jul 10 17:18:27.178213 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:27.178184 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gskkj" Jul 10 17:18:27.178691 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:27.178252 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgsqf" Jul 10 17:18:47.588608 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:47.588573 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgsqf"] Jul 10 17:18:47.589071 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:47.588869 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgsqf" podUID="723f1d41-6b52-402f-830b-b82b9f08260f" containerName="manager" containerID="cri-o://284e746d826a153f907e946ece3fa29c9756686b26c0fc82def2a9c73005fb0f" gracePeriod=10 Jul 10 17:18:47.842502 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:47.842436 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwzhn"] Jul 10 17:18:47.844412 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:47.844378 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgsqf" Jul 10 17:18:47.844849 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:47.844835 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwzhn" Jul 10 17:18:47.859641 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:47.859608 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwzhn"] Jul 10 17:18:47.928227 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:47.928192 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79hdp\" (UniqueName: \"kubernetes.io/projected/723f1d41-6b52-402f-830b-b82b9f08260f-kube-api-access-79hdp\") pod \"723f1d41-6b52-402f-830b-b82b9f08260f\" (UID: \"723f1d41-6b52-402f-830b-b82b9f08260f\") " Jul 10 17:18:47.928227 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:47.928228 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/723f1d41-6b52-402f-830b-b82b9f08260f-extensions-socket-volume\") pod \"723f1d41-6b52-402f-830b-b82b9f08260f\" (UID: \"723f1d41-6b52-402f-830b-b82b9f08260f\") " Jul 10 17:18:47.928516 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:47.928276 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e9508f1-f98f-43c9-bd18-efdce98e6308-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-cwzhn\" (UID: \"7e9508f1-f98f-43c9-bd18-efdce98e6308\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwzhn" Jul 10 17:18:47.928516 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:47.928325 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m6rf\" (UniqueName: \"kubernetes.io/projected/7e9508f1-f98f-43c9-bd18-efdce98e6308-kube-api-access-5m6rf\") pod \"kuadrant-operator-controller-manager-86878c769f-cwzhn\" (UID: \"7e9508f1-f98f-43c9-bd18-efdce98e6308\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwzhn" Jul 10 17:18:47.928750 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:47.928727 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723f1d41-6b52-402f-830b-b82b9f08260f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "723f1d41-6b52-402f-830b-b82b9f08260f" (UID: "723f1d41-6b52-402f-830b-b82b9f08260f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:18:47.930464 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:47.930442 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/723f1d41-6b52-402f-830b-b82b9f08260f-kube-api-access-79hdp" (OuterVolumeSpecName: "kube-api-access-79hdp") pod "723f1d41-6b52-402f-830b-b82b9f08260f" (UID: "723f1d41-6b52-402f-830b-b82b9f08260f"). InnerVolumeSpecName "kube-api-access-79hdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:18:48.028668 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:48.028622 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m6rf\" (UniqueName: \"kubernetes.io/projected/7e9508f1-f98f-43c9-bd18-efdce98e6308-kube-api-access-5m6rf\") pod \"kuadrant-operator-controller-manager-86878c769f-cwzhn\" (UID: \"7e9508f1-f98f-43c9-bd18-efdce98e6308\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwzhn" Jul 10 17:18:48.028668 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:48.028681 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e9508f1-f98f-43c9-bd18-efdce98e6308-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-cwzhn\" (UID: \"7e9508f1-f98f-43c9-bd18-efdce98e6308\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwzhn" Jul 10 17:18:48.028902 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:48.028748 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79hdp\" (UniqueName: \"kubernetes.io/projected/723f1d41-6b52-402f-830b-b82b9f08260f-kube-api-access-79hdp\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:18:48.028902 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:48.028759 2583 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/723f1d41-6b52-402f-830b-b82b9f08260f-extensions-socket-volume\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:18:48.029078 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:48.029055 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e9508f1-f98f-43c9-bd18-efdce98e6308-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-cwzhn\" (UID: \"7e9508f1-f98f-43c9-bd18-efdce98e6308\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwzhn" Jul 10 17:18:48.046899 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:48.046861 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m6rf\" (UniqueName: \"kubernetes.io/projected/7e9508f1-f98f-43c9-bd18-efdce98e6308-kube-api-access-5m6rf\") pod \"kuadrant-operator-controller-manager-86878c769f-cwzhn\" (UID: \"7e9508f1-f98f-43c9-bd18-efdce98e6308\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwzhn" Jul 10 17:18:48.154587 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:48.154483 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwzhn" Jul 10 17:18:48.296925 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:48.296898 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwzhn"] Jul 10 17:18:48.298803 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:18:48.298768 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e9508f1_f98f_43c9_bd18_efdce98e6308.slice/crio-31ce59fa509e81d4c7338b4a53f3e1fb35080b8757c0f1a34c3c679ae37f65a9 WatchSource:0}: Error finding container 31ce59fa509e81d4c7338b4a53f3e1fb35080b8757c0f1a34c3c679ae37f65a9: Status 404 returned error can't find the container with id 31ce59fa509e81d4c7338b4a53f3e1fb35080b8757c0f1a34c3c679ae37f65a9 Jul 10 17:18:48.342344 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:48.342305 2583 generic.go:356] "Generic (PLEG): container finished" podID="723f1d41-6b52-402f-830b-b82b9f08260f" containerID="284e746d826a153f907e946ece3fa29c9756686b26c0fc82def2a9c73005fb0f" exitCode=0 Jul 10 17:18:48.342500 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:48.342350 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgsqf" Jul 10 17:18:48.342500 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:48.342375 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgsqf" event={"ID":"723f1d41-6b52-402f-830b-b82b9f08260f","Type":"ContainerDied","Data":"284e746d826a153f907e946ece3fa29c9756686b26c0fc82def2a9c73005fb0f"} Jul 10 17:18:48.342500 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:48.342467 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgsqf" event={"ID":"723f1d41-6b52-402f-830b-b82b9f08260f","Type":"ContainerDied","Data":"a1999834cc895fea449fd5f9ec3daf454a6c04da9ed009eeb5aa8ea38301712a"} Jul 10 17:18:48.342500 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:48.342497 2583 scope.go:117] "RemoveContainer" containerID="284e746d826a153f907e946ece3fa29c9756686b26c0fc82def2a9c73005fb0f" Jul 10 17:18:48.343809 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:48.343787 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwzhn" event={"ID":"7e9508f1-f98f-43c9-bd18-efdce98e6308","Type":"ContainerStarted","Data":"31ce59fa509e81d4c7338b4a53f3e1fb35080b8757c0f1a34c3c679ae37f65a9"} Jul 10 17:18:48.351360 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:48.351275 2583 scope.go:117] "RemoveContainer" containerID="284e746d826a153f907e946ece3fa29c9756686b26c0fc82def2a9c73005fb0f" Jul 10 17:18:48.351673 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:18:48.351653 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"284e746d826a153f907e946ece3fa29c9756686b26c0fc82def2a9c73005fb0f\": container with ID starting with 284e746d826a153f907e946ece3fa29c9756686b26c0fc82def2a9c73005fb0f not found: ID does not exist" containerID="284e746d826a153f907e946ece3fa29c9756686b26c0fc82def2a9c73005fb0f" Jul 10 17:18:48.351743 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:48.351680 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"284e746d826a153f907e946ece3fa29c9756686b26c0fc82def2a9c73005fb0f"} err="failed to get container status \"284e746d826a153f907e946ece3fa29c9756686b26c0fc82def2a9c73005fb0f\": rpc error: code = NotFound desc = could not find container \"284e746d826a153f907e946ece3fa29c9756686b26c0fc82def2a9c73005fb0f\": container with ID starting with 284e746d826a153f907e946ece3fa29c9756686b26c0fc82def2a9c73005fb0f not found: ID does not exist" Jul 10 17:18:48.358899 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:48.358876 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgsqf"] Jul 10 17:18:48.366466 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:48.366441 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sgsqf"] Jul 10 17:18:49.351466 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:49.351433 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwzhn" event={"ID":"7e9508f1-f98f-43c9-bd18-efdce98e6308","Type":"ContainerStarted","Data":"49cb3131cd19227043c72d24ad23f83b63a4c84e87e2bd4d24db42e4b7323b2d"} Jul 10 17:18:49.351903 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:49.351487 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwzhn" Jul 10 17:18:49.374493 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:49.374440 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwzhn" podStartSLOduration=2.374424784 podStartE2EDuration="2.374424784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:18:49.372135708 +0000 UTC m=+461.699833248" watchObservedRunningTime="2026-07-10 17:18:49.374424784 +0000 UTC m=+461.702122320" Jul 10 17:18:50.205366 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:18:50.205333 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="723f1d41-6b52-402f-830b-b82b9f08260f" path="/var/lib/kubelet/pods/723f1d41-6b52-402f-830b-b82b9f08260f/volumes" Jul 10 17:19:00.358764 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:00.358731 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwzhn" Jul 10 17:19:03.824215 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.824182 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6"] Jul 10 17:19:03.825237 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.825214 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="723f1d41-6b52-402f-830b-b82b9f08260f" containerName="manager" Jul 10 17:19:03.825237 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.825239 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="723f1d41-6b52-402f-830b-b82b9f08260f" containerName="manager" Jul 10 17:19:03.825373 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.825341 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="723f1d41-6b52-402f-830b-b82b9f08260f" containerName="manager" Jul 10 17:19:03.827615 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.827595 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:19:03.829898 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.829878 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-cztmv\"" Jul 10 17:19:03.830002 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.829930 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 17:19:03.838653 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.838630 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6"] Jul 10 17:19:03.842846 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.842824 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhrgp\" (UniqueName: \"kubernetes.io/projected/553d3bf3-9e01-4b25-9ac2-01b0da4faf79-kube-api-access-fhrgp\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mlrv6\" (UID: \"553d3bf3-9e01-4b25-9ac2-01b0da4faf79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:19:03.842976 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.842867 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/553d3bf3-9e01-4b25-9ac2-01b0da4faf79-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mlrv6\" (UID: \"553d3bf3-9e01-4b25-9ac2-01b0da4faf79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:19:03.842976 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.842889 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/553d3bf3-9e01-4b25-9ac2-01b0da4faf79-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mlrv6\" (UID: \"553d3bf3-9e01-4b25-9ac2-01b0da4faf79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:19:03.842976 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.842904 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/553d3bf3-9e01-4b25-9ac2-01b0da4faf79-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mlrv6\" (UID: \"553d3bf3-9e01-4b25-9ac2-01b0da4faf79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:19:03.843133 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.842972 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/553d3bf3-9e01-4b25-9ac2-01b0da4faf79-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mlrv6\" (UID: \"553d3bf3-9e01-4b25-9ac2-01b0da4faf79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:19:03.843133 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.843004 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/553d3bf3-9e01-4b25-9ac2-01b0da4faf79-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mlrv6\" (UID: \"553d3bf3-9e01-4b25-9ac2-01b0da4faf79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:19:03.843133 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.843048 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/553d3bf3-9e01-4b25-9ac2-01b0da4faf79-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mlrv6\" (UID: \"553d3bf3-9e01-4b25-9ac2-01b0da4faf79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:19:03.843133 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.843082 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/553d3bf3-9e01-4b25-9ac2-01b0da4faf79-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mlrv6\" (UID: \"553d3bf3-9e01-4b25-9ac2-01b0da4faf79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:19:03.843133 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.843128 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/553d3bf3-9e01-4b25-9ac2-01b0da4faf79-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mlrv6\" (UID: \"553d3bf3-9e01-4b25-9ac2-01b0da4faf79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:19:03.943581 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.943534 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/553d3bf3-9e01-4b25-9ac2-01b0da4faf79-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mlrv6\" (UID: \"553d3bf3-9e01-4b25-9ac2-01b0da4faf79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:19:03.943581 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.943596 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhrgp\" (UniqueName: \"kubernetes.io/projected/553d3bf3-9e01-4b25-9ac2-01b0da4faf79-kube-api-access-fhrgp\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mlrv6\" (UID: \"553d3bf3-9e01-4b25-9ac2-01b0da4faf79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:19:03.943847 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.943655 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/553d3bf3-9e01-4b25-9ac2-01b0da4faf79-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mlrv6\" (UID: \"553d3bf3-9e01-4b25-9ac2-01b0da4faf79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:19:03.943847 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.943688 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/553d3bf3-9e01-4b25-9ac2-01b0da4faf79-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mlrv6\" (UID: \"553d3bf3-9e01-4b25-9ac2-01b0da4faf79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:19:03.943847 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.943709 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/553d3bf3-9e01-4b25-9ac2-01b0da4faf79-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mlrv6\" (UID: \"553d3bf3-9e01-4b25-9ac2-01b0da4faf79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:19:03.944338 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.943853 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/553d3bf3-9e01-4b25-9ac2-01b0da4faf79-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mlrv6\" (UID: \"553d3bf3-9e01-4b25-9ac2-01b0da4faf79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:19:03.944338 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.943918 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/553d3bf3-9e01-4b25-9ac2-01b0da4faf79-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mlrv6\" (UID: \"553d3bf3-9e01-4b25-9ac2-01b0da4faf79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:19:03.944338 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.943965 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/553d3bf3-9e01-4b25-9ac2-01b0da4faf79-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mlrv6\" (UID: \"553d3bf3-9e01-4b25-9ac2-01b0da4faf79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:19:03.944338 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.944001 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/553d3bf3-9e01-4b25-9ac2-01b0da4faf79-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mlrv6\" (UID: \"553d3bf3-9e01-4b25-9ac2-01b0da4faf79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:19:03.944338 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.944070 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/553d3bf3-9e01-4b25-9ac2-01b0da4faf79-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mlrv6\" (UID: \"553d3bf3-9e01-4b25-9ac2-01b0da4faf79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:19:03.944338 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.944266 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/553d3bf3-9e01-4b25-9ac2-01b0da4faf79-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mlrv6\" (UID: \"553d3bf3-9e01-4b25-9ac2-01b0da4faf79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:19:03.944663 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.944438 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/553d3bf3-9e01-4b25-9ac2-01b0da4faf79-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mlrv6\" (UID: \"553d3bf3-9e01-4b25-9ac2-01b0da4faf79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:19:03.944663 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.944487 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/553d3bf3-9e01-4b25-9ac2-01b0da4faf79-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mlrv6\" (UID: \"553d3bf3-9e01-4b25-9ac2-01b0da4faf79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:19:03.944663 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.944558 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/553d3bf3-9e01-4b25-9ac2-01b0da4faf79-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mlrv6\" (UID: \"553d3bf3-9e01-4b25-9ac2-01b0da4faf79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:19:03.946211 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.946180 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/553d3bf3-9e01-4b25-9ac2-01b0da4faf79-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mlrv6\" (UID: \"553d3bf3-9e01-4b25-9ac2-01b0da4faf79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:19:03.946600 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.946576 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/553d3bf3-9e01-4b25-9ac2-01b0da4faf79-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mlrv6\" (UID: \"553d3bf3-9e01-4b25-9ac2-01b0da4faf79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:19:03.951576 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.951546 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/553d3bf3-9e01-4b25-9ac2-01b0da4faf79-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mlrv6\" (UID: \"553d3bf3-9e01-4b25-9ac2-01b0da4faf79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:19:03.951752 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:03.951732 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhrgp\" (UniqueName: \"kubernetes.io/projected/553d3bf3-9e01-4b25-9ac2-01b0da4faf79-kube-api-access-fhrgp\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mlrv6\" (UID: \"553d3bf3-9e01-4b25-9ac2-01b0da4faf79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:19:04.139725 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:04.139630 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:19:04.284201 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:04.284168 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6"] Jul 10 17:19:04.287996 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:19:04.287968 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod553d3bf3_9e01_4b25_9ac2_01b0da4faf79.slice/crio-7d6e97a80698eb898a1ec42cef93005a8c0e23d97e3748c22a5bf92f04633d0f WatchSource:0}: Error finding container 7d6e97a80698eb898a1ec42cef93005a8c0e23d97e3748c22a5bf92f04633d0f: Status 404 returned error can't find the container with id 7d6e97a80698eb898a1ec42cef93005a8c0e23d97e3748c22a5bf92f04633d0f Jul 10 17:19:04.426300 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:04.426263 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" event={"ID":"553d3bf3-9e01-4b25-9ac2-01b0da4faf79","Type":"ContainerStarted","Data":"7d6e97a80698eb898a1ec42cef93005a8c0e23d97e3748c22a5bf92f04633d0f"} Jul 10 17:19:06.827152 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:06.827119 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:19:06.827445 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:06.827210 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:19:06.827445 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:06.827239 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:19:07.447425 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:07.447378 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" event={"ID":"553d3bf3-9e01-4b25-9ac2-01b0da4faf79","Type":"ContainerStarted","Data":"b01c5f6748242f1391e0709566d530c1c4e159c80a416f417e9bdd765c4c0dc7"} Jul 10 17:19:07.470131 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:07.470084 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" podStartSLOduration=1.933245975 podStartE2EDuration="4.470066917s" totalImagesPullingTime="2.536820942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:19:07.468864219 +0000 UTC m=+479.796561752" watchObservedRunningTime="2026-07-10 17:19:07.470066917 +0000 UTC m=+479.797764456" Jul 10 17:19:08.140699 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:08.140664 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:19:08.145417 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:08.145366 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:19:08.423452 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:08.423412 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ns9tv"] Jul 10 17:19:08.426685 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:08.426660 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-ns9tv" Jul 10 17:19:08.429014 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:08.428982 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2qjm5\"" Jul 10 17:19:08.429127 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:08.429027 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 10 17:19:08.434371 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:08.434339 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ns9tv"] Jul 10 17:19:08.452896 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:08.452866 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:19:08.454082 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:08.454062 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:19:08.486194 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:08.486161 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0592fba6-1412-4443-85c2-cce7c5b1fe59-config-file\") pod \"limitador-limitador-64c94fd64f-ns9tv\" (UID: \"0592fba6-1412-4443-85c2-cce7c5b1fe59\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ns9tv" Jul 10 17:19:08.486512 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:08.486489 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dws7\" (UniqueName: \"kubernetes.io/projected/0592fba6-1412-4443-85c2-cce7c5b1fe59-kube-api-access-5dws7\") pod \"limitador-limitador-64c94fd64f-ns9tv\" (UID: \"0592fba6-1412-4443-85c2-cce7c5b1fe59\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ns9tv" Jul 10 17:19:08.526491 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:08.526459 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ns9tv"] Jul 10 17:19:08.587095 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:08.587057 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dws7\" (UniqueName: \"kubernetes.io/projected/0592fba6-1412-4443-85c2-cce7c5b1fe59-kube-api-access-5dws7\") pod \"limitador-limitador-64c94fd64f-ns9tv\" (UID: \"0592fba6-1412-4443-85c2-cce7c5b1fe59\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ns9tv" Jul 10 17:19:08.587277 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:08.587141 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0592fba6-1412-4443-85c2-cce7c5b1fe59-config-file\") pod \"limitador-limitador-64c94fd64f-ns9tv\" (UID: \"0592fba6-1412-4443-85c2-cce7c5b1fe59\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ns9tv" Jul 10 17:19:08.587759 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:08.587741 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0592fba6-1412-4443-85c2-cce7c5b1fe59-config-file\") pod \"limitador-limitador-64c94fd64f-ns9tv\" (UID: \"0592fba6-1412-4443-85c2-cce7c5b1fe59\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ns9tv" Jul 10 17:19:08.595611 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:08.595584 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dws7\" (UniqueName: \"kubernetes.io/projected/0592fba6-1412-4443-85c2-cce7c5b1fe59-kube-api-access-5dws7\") pod \"limitador-limitador-64c94fd64f-ns9tv\" (UID: \"0592fba6-1412-4443-85c2-cce7c5b1fe59\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ns9tv" Jul 10 17:19:08.740824 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:08.740732 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-ns9tv" Jul 10 17:19:08.885635 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:08.885608 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ns9tv"] Jul 10 17:19:09.213312 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:09.213267 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-dm99c"] Jul 10 17:19:09.216513 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:09.216483 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-dm99c" Jul 10 17:19:09.218917 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:09.218897 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ftsm9\"" Jul 10 17:19:09.224294 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:09.224269 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-dm99c"] Jul 10 17:19:09.294582 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:09.294545 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78s54\" (UniqueName: \"kubernetes.io/projected/f40d1d5e-1ef2-4228-a151-5e36de4a6e01-kube-api-access-78s54\") pod \"authorino-58cd955675-dm99c\" (UID: \"f40d1d5e-1ef2-4228-a151-5e36de4a6e01\") " pod="kuadrant-system/authorino-58cd955675-dm99c" Jul 10 17:19:09.395852 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:09.395809 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78s54\" (UniqueName: \"kubernetes.io/projected/f40d1d5e-1ef2-4228-a151-5e36de4a6e01-kube-api-access-78s54\") pod \"authorino-58cd955675-dm99c\" (UID: \"f40d1d5e-1ef2-4228-a151-5e36de4a6e01\") " pod="kuadrant-system/authorino-58cd955675-dm99c" Jul 10 17:19:09.404135 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:09.404101 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78s54\" (UniqueName: \"kubernetes.io/projected/f40d1d5e-1ef2-4228-a151-5e36de4a6e01-kube-api-access-78s54\") pod \"authorino-58cd955675-dm99c\" (UID: \"f40d1d5e-1ef2-4228-a151-5e36de4a6e01\") " pod="kuadrant-system/authorino-58cd955675-dm99c" Jul 10 17:19:09.459216 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:09.459181 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-ns9tv" event={"ID":"0592fba6-1412-4443-85c2-cce7c5b1fe59","Type":"ContainerStarted","Data":"02cd4a541ef7f51cecc0d4359bd3d0414cc90290885dcd195d36086e3afd3d28"} Jul 10 17:19:09.528053 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:09.527964 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-dm99c" Jul 10 17:19:09.687344 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:09.687313 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-dm99c"] Jul 10 17:19:10.473604 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:10.473563 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-dm99c" event={"ID":"f40d1d5e-1ef2-4228-a151-5e36de4a6e01","Type":"ContainerStarted","Data":"a81d1b4d7e631a2f9a93a961dd00f5a2d242116e4970de80a399c7cfded182cf"} Jul 10 17:19:11.983682 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:19:11.983644 2583 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: reading blob sha256:55c0205b422beeca9ab8ece9c61b1e34f31686b8a7adf249272ac75b4dd57e4d: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image" image="quay.io/kuadrant/authorino:v0.24.0" Jul 10 17:19:11.984070 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:19:11.983758 2583 kuberuntime_manager.go:1459] "Unhandled Error" err="container authorino start failed in pod authorino-58cd955675-dm99c_kuadrant-system(f40d1d5e-1ef2-4228-a151-5e36de4a6e01): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: reading blob sha256:55c0205b422beeca9ab8ece9c61b1e34f31686b8a7adf249272ac75b4dd57e4d: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image" logger="UnhandledError" Jul 10 17:19:11.984070 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:19:11.983791 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"authorino\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: reading blob sha256:55c0205b422beeca9ab8ece9c61b1e34f31686b8a7adf249272ac75b4dd57e4d: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image\"" pod="kuadrant-system/authorino-58cd955675-dm99c" podUID="f40d1d5e-1ef2-4228-a151-5e36de4a6e01" Jul 10 17:19:12.486061 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:12.486022 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-ns9tv" event={"ID":"0592fba6-1412-4443-85c2-cce7c5b1fe59","Type":"ContainerStarted","Data":"35c3267916d9a6d9ae8a4ccf5cf58f6081e8e7f8083be4e088d519e164f5c65e"} Jul 10 17:19:12.486270 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:12.486140 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-ns9tv" Jul 10 17:19:12.486590 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:19:12.486568 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"authorino\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/kuadrant/authorino:v0.24.0\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: reading blob sha256:55c0205b422beeca9ab8ece9c61b1e34f31686b8a7adf249272ac75b4dd57e4d: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image\"" pod="kuadrant-system/authorino-58cd955675-dm99c" podUID="f40d1d5e-1ef2-4228-a151-5e36de4a6e01" Jul 10 17:19:12.503000 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:12.502945 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-ns9tv" podStartSLOduration=1.617902 podStartE2EDuration="4.502929212s" totalImagesPullingTime="2.885027212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:19:12.500262447 +0000 UTC m=+484.827959987" watchObservedRunningTime="2026-07-10 17:19:12.502929212 +0000 UTC m=+484.830626750" Jul 10 17:19:14.778865 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:14.778829 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-dm99c"] Jul 10 17:19:14.914518 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:14.914495 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-dm99c" Jul 10 17:19:15.048455 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:15.048356 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78s54\" (UniqueName: \"kubernetes.io/projected/f40d1d5e-1ef2-4228-a151-5e36de4a6e01-kube-api-access-78s54\") pod \"f40d1d5e-1ef2-4228-a151-5e36de4a6e01\" (UID: \"f40d1d5e-1ef2-4228-a151-5e36de4a6e01\") " Jul 10 17:19:15.050541 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:15.050517 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40d1d5e-1ef2-4228-a151-5e36de4a6e01-kube-api-access-78s54" (OuterVolumeSpecName: "kube-api-access-78s54") pod "f40d1d5e-1ef2-4228-a151-5e36de4a6e01" (UID: "f40d1d5e-1ef2-4228-a151-5e36de4a6e01"). InnerVolumeSpecName "kube-api-access-78s54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:19:15.149619 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:15.149587 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78s54\" (UniqueName: \"kubernetes.io/projected/f40d1d5e-1ef2-4228-a151-5e36de4a6e01-kube-api-access-78s54\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:19:15.501573 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:15.501538 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-dm99c" event={"ID":"f40d1d5e-1ef2-4228-a151-5e36de4a6e01","Type":"ContainerDied","Data":"a81d1b4d7e631a2f9a93a961dd00f5a2d242116e4970de80a399c7cfded182cf"} Jul 10 17:19:15.501776 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:15.501549 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-dm99c" Jul 10 17:19:15.535902 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:15.535873 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-dm99c"] Jul 10 17:19:15.540080 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:15.540053 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-dm99c"] Jul 10 17:19:16.206027 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:16.205993 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f40d1d5e-1ef2-4228-a151-5e36de4a6e01" path="/var/lib/kubelet/pods/f40d1d5e-1ef2-4228-a151-5e36de4a6e01/volumes" Jul 10 17:19:23.492278 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:23.492248 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-ns9tv" Jul 10 17:19:25.772044 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:25.772007 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ns9tv"] Jul 10 17:19:25.772476 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:25.772180 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-ns9tv" podUID="0592fba6-1412-4443-85c2-cce7c5b1fe59" containerName="limitador" containerID="cri-o://35c3267916d9a6d9ae8a4ccf5cf58f6081e8e7f8083be4e088d519e164f5c65e" gracePeriod=30 Jul 10 17:19:26.556441 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:26.556384 2583 generic.go:356] "Generic (PLEG): container finished" podID="0592fba6-1412-4443-85c2-cce7c5b1fe59" containerID="35c3267916d9a6d9ae8a4ccf5cf58f6081e8e7f8083be4e088d519e164f5c65e" exitCode=0 Jul 10 17:19:26.556608 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:26.556458 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-ns9tv" event={"ID":"0592fba6-1412-4443-85c2-cce7c5b1fe59","Type":"ContainerDied","Data":"35c3267916d9a6d9ae8a4ccf5cf58f6081e8e7f8083be4e088d519e164f5c65e"} Jul 10 17:19:26.728219 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:26.728196 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-ns9tv" Jul 10 17:19:26.856048 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:26.855961 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0592fba6-1412-4443-85c2-cce7c5b1fe59-config-file\") pod \"0592fba6-1412-4443-85c2-cce7c5b1fe59\" (UID: \"0592fba6-1412-4443-85c2-cce7c5b1fe59\") " Jul 10 17:19:26.856048 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:26.856002 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dws7\" (UniqueName: \"kubernetes.io/projected/0592fba6-1412-4443-85c2-cce7c5b1fe59-kube-api-access-5dws7\") pod \"0592fba6-1412-4443-85c2-cce7c5b1fe59\" (UID: \"0592fba6-1412-4443-85c2-cce7c5b1fe59\") " Jul 10 17:19:26.856587 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:26.856430 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0592fba6-1412-4443-85c2-cce7c5b1fe59-config-file" (OuterVolumeSpecName: "config-file") pod "0592fba6-1412-4443-85c2-cce7c5b1fe59" (UID: "0592fba6-1412-4443-85c2-cce7c5b1fe59"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:19:26.858366 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:26.858341 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0592fba6-1412-4443-85c2-cce7c5b1fe59-kube-api-access-5dws7" (OuterVolumeSpecName: "kube-api-access-5dws7") pod "0592fba6-1412-4443-85c2-cce7c5b1fe59" (UID: "0592fba6-1412-4443-85c2-cce7c5b1fe59"). InnerVolumeSpecName "kube-api-access-5dws7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:19:26.957042 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:26.956999 2583 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0592fba6-1412-4443-85c2-cce7c5b1fe59-config-file\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:19:26.957042 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:26.957038 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dws7\" (UniqueName: \"kubernetes.io/projected/0592fba6-1412-4443-85c2-cce7c5b1fe59-kube-api-access-5dws7\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:19:27.562372 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:27.562343 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-ns9tv" Jul 10 17:19:27.562585 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:27.562409 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-ns9tv" event={"ID":"0592fba6-1412-4443-85c2-cce7c5b1fe59","Type":"ContainerDied","Data":"02cd4a541ef7f51cecc0d4359bd3d0414cc90290885dcd195d36086e3afd3d28"} Jul 10 17:19:27.562585 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:27.562469 2583 scope.go:117] "RemoveContainer" containerID="35c3267916d9a6d9ae8a4ccf5cf58f6081e8e7f8083be4e088d519e164f5c65e" Jul 10 17:19:27.583497 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:27.583473 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ns9tv"] Jul 10 17:19:27.585254 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:27.585235 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ns9tv"] Jul 10 17:19:28.206837 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:28.206803 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0592fba6-1412-4443-85c2-cce7c5b1fe59" path="/var/lib/kubelet/pods/0592fba6-1412-4443-85c2-cce7c5b1fe59/volumes" Jul 10 17:19:30.030229 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:30.030194 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-hvqc8"] Jul 10 17:19:30.030818 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:30.030797 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0592fba6-1412-4443-85c2-cce7c5b1fe59" containerName="limitador" Jul 10 17:19:30.030895 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:30.030821 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0592fba6-1412-4443-85c2-cce7c5b1fe59" containerName="limitador" Jul 10 17:19:30.030951 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:30.030943 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0592fba6-1412-4443-85c2-cce7c5b1fe59" containerName="limitador" Jul 10 17:19:30.032860 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:30.032840 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hvqc8" Jul 10 17:19:30.035900 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:30.035872 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 17:19:30.036022 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:30.035872 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 10 17:19:30.036022 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:30.035875 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-7hqg6\"" Jul 10 17:19:30.036519 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:30.036502 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 17:19:30.041867 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:30.041846 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-hvqc8"] Jul 10 17:19:30.184834 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:30.184803 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n7q9\" (UniqueName: \"kubernetes.io/projected/89e2948a-bd5e-4f18-a615-5ba8aa318e63-kube-api-access-4n7q9\") pod \"postgres-69fb4594cf-hvqc8\" (UID: \"89e2948a-bd5e-4f18-a615-5ba8aa318e63\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hvqc8" Jul 10 17:19:30.185013 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:30.184890 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/89e2948a-bd5e-4f18-a615-5ba8aa318e63-data\") pod \"postgres-69fb4594cf-hvqc8\" (UID: \"89e2948a-bd5e-4f18-a615-5ba8aa318e63\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hvqc8" Jul 10 17:19:30.286084 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:30.285994 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/89e2948a-bd5e-4f18-a615-5ba8aa318e63-data\") pod \"postgres-69fb4594cf-hvqc8\" (UID: \"89e2948a-bd5e-4f18-a615-5ba8aa318e63\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hvqc8" Jul 10 17:19:30.286084 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:30.286076 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n7q9\" (UniqueName: \"kubernetes.io/projected/89e2948a-bd5e-4f18-a615-5ba8aa318e63-kube-api-access-4n7q9\") pod \"postgres-69fb4594cf-hvqc8\" (UID: \"89e2948a-bd5e-4f18-a615-5ba8aa318e63\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hvqc8" Jul 10 17:19:30.286480 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:30.286461 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/89e2948a-bd5e-4f18-a615-5ba8aa318e63-data\") pod \"postgres-69fb4594cf-hvqc8\" (UID: \"89e2948a-bd5e-4f18-a615-5ba8aa318e63\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hvqc8" Jul 10 17:19:30.294557 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:30.294527 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n7q9\" (UniqueName: \"kubernetes.io/projected/89e2948a-bd5e-4f18-a615-5ba8aa318e63-kube-api-access-4n7q9\") pod \"postgres-69fb4594cf-hvqc8\" (UID: \"89e2948a-bd5e-4f18-a615-5ba8aa318e63\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hvqc8" Jul 10 17:19:30.344558 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:30.344521 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hvqc8" Jul 10 17:19:30.480307 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:30.480276 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-hvqc8"] Jul 10 17:19:30.483874 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:19:30.483844 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e2948a_bd5e_4f18_a615_5ba8aa318e63.slice/crio-73296bf735c97805a84699ecf2f777dc58133938f4e38cd8d3538e6d55259754 WatchSource:0}: Error finding container 73296bf735c97805a84699ecf2f777dc58133938f4e38cd8d3538e6d55259754: Status 404 returned error can't find the container with id 73296bf735c97805a84699ecf2f777dc58133938f4e38cd8d3538e6d55259754 Jul 10 17:19:30.581606 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:30.581517 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hvqc8" event={"ID":"89e2948a-bd5e-4f18-a615-5ba8aa318e63","Type":"ContainerStarted","Data":"73296bf735c97805a84699ecf2f777dc58133938f4e38cd8d3538e6d55259754"} Jul 10 17:19:35.611432 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:35.611316 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hvqc8" event={"ID":"89e2948a-bd5e-4f18-a615-5ba8aa318e63","Type":"ContainerStarted","Data":"636fda582ddb0daf2cb2acf35ac47f92ad52d7ca30190f853e80ad725bbfe515"} Jul 10 17:19:35.611837 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:35.611462 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hvqc8" Jul 10 17:19:35.627477 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:35.627427 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hvqc8" podStartSLOduration=0.80977262 podStartE2EDuration="5.627382738s" totalImagesPullingTime="4.817610118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:19:35.626054045 +0000 UTC m=+507.953751585" watchObservedRunningTime="2026-07-10 17:19:35.627382738 +0000 UTC m=+507.955080276" Jul 10 17:19:41.645527 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:41.645498 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hvqc8" Jul 10 17:19:48.028858 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:48.028825 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7d9b574468-8f4bs"] Jul 10 17:19:48.045883 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:48.045845 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d9b574468-8f4bs"] Jul 10 17:19:48.046061 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:48.045980 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d9b574468-8f4bs" Jul 10 17:19:48.048699 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:48.048671 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5p2bc\"" Jul 10 17:19:48.048699 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:48.048686 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 10 17:19:48.048989 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:48.048975 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 10 17:19:48.139859 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:48.139813 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcnm2\" (UniqueName: \"kubernetes.io/projected/65bd85d0-1590-4120-bf5b-73c3318f50ab-kube-api-access-dcnm2\") pod \"maas-controller-7d9b574468-8f4bs\" (UID: \"65bd85d0-1590-4120-bf5b-73c3318f50ab\") " pod="opendatahub/maas-controller-7d9b574468-8f4bs" Jul 10 17:19:48.140071 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:48.139922 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65bd85d0-1590-4120-bf5b-73c3318f50ab-webhook-cert\") pod \"maas-controller-7d9b574468-8f4bs\" (UID: \"65bd85d0-1590-4120-bf5b-73c3318f50ab\") " pod="opendatahub/maas-controller-7d9b574468-8f4bs" Jul 10 17:19:48.198525 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:48.198490 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b4655bb55-rwrpx"] Jul 10 17:19:48.202829 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:48.202799 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b4655bb55-rwrpx" Jul 10 17:19:48.211833 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:48.211805 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b4655bb55-rwrpx"] Jul 10 17:19:48.240355 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:48.240327 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdf62\" (UniqueName: \"kubernetes.io/projected/d131fb34-27ae-45de-b636-3c1b6131b57b-kube-api-access-tdf62\") pod \"maas-controller-5b4655bb55-rwrpx\" (UID: \"d131fb34-27ae-45de-b636-3c1b6131b57b\") " pod="opendatahub/maas-controller-5b4655bb55-rwrpx" Jul 10 17:19:48.240355 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:48.240367 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65bd85d0-1590-4120-bf5b-73c3318f50ab-webhook-cert\") pod \"maas-controller-7d9b574468-8f4bs\" (UID: \"65bd85d0-1590-4120-bf5b-73c3318f50ab\") " pod="opendatahub/maas-controller-7d9b574468-8f4bs" Jul 10 17:19:48.240640 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:48.240468 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcnm2\" (UniqueName: \"kubernetes.io/projected/65bd85d0-1590-4120-bf5b-73c3318f50ab-kube-api-access-dcnm2\") pod \"maas-controller-7d9b574468-8f4bs\" (UID: \"65bd85d0-1590-4120-bf5b-73c3318f50ab\") " pod="opendatahub/maas-controller-7d9b574468-8f4bs" Jul 10 17:19:48.240640 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:19:48.240508 2583 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 10 17:19:48.240640 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:48.240511 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d131fb34-27ae-45de-b636-3c1b6131b57b-webhook-cert\") pod \"maas-controller-5b4655bb55-rwrpx\" (UID: \"d131fb34-27ae-45de-b636-3c1b6131b57b\") " pod="opendatahub/maas-controller-5b4655bb55-rwrpx" Jul 10 17:19:48.240640 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:19:48.240578 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65bd85d0-1590-4120-bf5b-73c3318f50ab-webhook-cert podName:65bd85d0-1590-4120-bf5b-73c3318f50ab nodeName:}" failed. No retries permitted until 2026-07-10 17:19:48.74055497 +0000 UTC m=+521.068252488 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/65bd85d0-1590-4120-bf5b-73c3318f50ab-webhook-cert") pod "maas-controller-7d9b574468-8f4bs" (UID: "65bd85d0-1590-4120-bf5b-73c3318f50ab") : secret "maas-controller-webhook-cert" not found Jul 10 17:19:48.249892 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:48.249859 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcnm2\" (UniqueName: \"kubernetes.io/projected/65bd85d0-1590-4120-bf5b-73c3318f50ab-kube-api-access-dcnm2\") pod \"maas-controller-7d9b574468-8f4bs\" (UID: \"65bd85d0-1590-4120-bf5b-73c3318f50ab\") " pod="opendatahub/maas-controller-7d9b574468-8f4bs" Jul 10 17:19:48.341162 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:48.341065 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdf62\" (UniqueName: \"kubernetes.io/projected/d131fb34-27ae-45de-b636-3c1b6131b57b-kube-api-access-tdf62\") pod \"maas-controller-5b4655bb55-rwrpx\" (UID: \"d131fb34-27ae-45de-b636-3c1b6131b57b\") " pod="opendatahub/maas-controller-5b4655bb55-rwrpx" Jul 10 17:19:48.341162 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:48.341138 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d131fb34-27ae-45de-b636-3c1b6131b57b-webhook-cert\") pod \"maas-controller-5b4655bb55-rwrpx\" (UID: \"d131fb34-27ae-45de-b636-3c1b6131b57b\") " pod="opendatahub/maas-controller-5b4655bb55-rwrpx" Jul 10 17:19:48.343876 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:48.343848 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d131fb34-27ae-45de-b636-3c1b6131b57b-webhook-cert\") pod \"maas-controller-5b4655bb55-rwrpx\" (UID: \"d131fb34-27ae-45de-b636-3c1b6131b57b\") " pod="opendatahub/maas-controller-5b4655bb55-rwrpx" Jul 10 17:19:48.349246 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:48.349222 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdf62\" (UniqueName: \"kubernetes.io/projected/d131fb34-27ae-45de-b636-3c1b6131b57b-kube-api-access-tdf62\") pod \"maas-controller-5b4655bb55-rwrpx\" (UID: \"d131fb34-27ae-45de-b636-3c1b6131b57b\") " pod="opendatahub/maas-controller-5b4655bb55-rwrpx" Jul 10 17:19:48.514834 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:48.514792 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b4655bb55-rwrpx" Jul 10 17:19:48.652159 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:48.652047 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b4655bb55-rwrpx"] Jul 10 17:19:48.654843 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:19:48.654815 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd131fb34_27ae_45de_b636_3c1b6131b57b.slice/crio-c7bccb324f44fa54dab54727595ba5ef42952a6f676e2eebcfcc35189e1e6e65 WatchSource:0}: Error finding container c7bccb324f44fa54dab54727595ba5ef42952a6f676e2eebcfcc35189e1e6e65: Status 404 returned error can't find the container with id c7bccb324f44fa54dab54727595ba5ef42952a6f676e2eebcfcc35189e1e6e65 Jul 10 17:19:48.675248 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:48.675210 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b4655bb55-rwrpx" event={"ID":"d131fb34-27ae-45de-b636-3c1b6131b57b","Type":"ContainerStarted","Data":"c7bccb324f44fa54dab54727595ba5ef42952a6f676e2eebcfcc35189e1e6e65"} Jul 10 17:19:48.745016 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:48.744979 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65bd85d0-1590-4120-bf5b-73c3318f50ab-webhook-cert\") pod \"maas-controller-7d9b574468-8f4bs\" (UID: \"65bd85d0-1590-4120-bf5b-73c3318f50ab\") " pod="opendatahub/maas-controller-7d9b574468-8f4bs" Jul 10 17:19:48.747556 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:48.747527 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65bd85d0-1590-4120-bf5b-73c3318f50ab-webhook-cert\") pod \"maas-controller-7d9b574468-8f4bs\" (UID: \"65bd85d0-1590-4120-bf5b-73c3318f50ab\") " pod="opendatahub/maas-controller-7d9b574468-8f4bs" Jul 10 17:19:48.960066 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:48.960033 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d9b574468-8f4bs" Jul 10 17:19:49.094849 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:49.094822 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d9b574468-8f4bs"] Jul 10 17:19:49.096874 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:19:49.096845 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65bd85d0_1590_4120_bf5b_73c3318f50ab.slice/crio-78ba482b0697d0a4f8a685e98b842ca03b99c052e0b4b1e631d61829c6d49a84 WatchSource:0}: Error finding container 78ba482b0697d0a4f8a685e98b842ca03b99c052e0b4b1e631d61829c6d49a84: Status 404 returned error can't find the container with id 78ba482b0697d0a4f8a685e98b842ca03b99c052e0b4b1e631d61829c6d49a84 Jul 10 17:19:49.683097 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:49.683052 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d9b574468-8f4bs" event={"ID":"65bd85d0-1590-4120-bf5b-73c3318f50ab","Type":"ContainerStarted","Data":"78ba482b0697d0a4f8a685e98b842ca03b99c052e0b4b1e631d61829c6d49a84"} Jul 10 17:19:51.696561 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:51.696519 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b4655bb55-rwrpx" event={"ID":"d131fb34-27ae-45de-b636-3c1b6131b57b","Type":"ContainerStarted","Data":"98f0afe894459aa037c3d99310b3991559ebf0f7ce8d8809a9e2507e26550ab9"} Jul 10 17:19:51.697022 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:51.696701 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b4655bb55-rwrpx" Jul 10 17:19:51.698191 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:51.698161 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d9b574468-8f4bs" event={"ID":"65bd85d0-1590-4120-bf5b-73c3318f50ab","Type":"ContainerStarted","Data":"ed0797751afa0265a2bb7dc46779040cf34a2040991df469d68d3e9eab788ff4"} Jul 10 17:19:51.698327 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:51.698274 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7d9b574468-8f4bs" Jul 10 17:19:51.734923 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:51.734874 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5b4655bb55-rwrpx" podStartSLOduration=1.399625184 podStartE2EDuration="3.734857708s" totalImagesPullingTime="2.335232524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:19:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:19:51.733529107 +0000 UTC m=+524.061226646" watchObservedRunningTime="2026-07-10 17:19:51.734857708 +0000 UTC m=+524.062555249" Jul 10 17:19:51.780667 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:19:51.780622 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7d9b574468-8f4bs" podStartSLOduration=1.897926982 podStartE2EDuration="3.780606336s" totalImagesPullingTime="1.882679354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:19:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:19:51.778280451 +0000 UTC m=+524.105977990" watchObservedRunningTime="2026-07-10 17:19:51.780606336 +0000 UTC m=+524.108303874" Jul 10 17:20:02.706665 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:02.706589 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b4655bb55-rwrpx" Jul 10 17:20:02.707049 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:02.706675 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7d9b574468-8f4bs" Jul 10 17:20:02.765012 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:02.764982 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d9b574468-8f4bs"] Jul 10 17:20:02.765196 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:02.765165 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7d9b574468-8f4bs" podUID="65bd85d0-1590-4120-bf5b-73c3318f50ab" containerName="manager" containerID="cri-o://ed0797751afa0265a2bb7dc46779040cf34a2040991df469d68d3e9eab788ff4" gracePeriod=30 Jul 10 17:20:03.019994 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:03.019968 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d9b574468-8f4bs" Jul 10 17:20:03.070040 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:03.070004 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcnm2\" (UniqueName: \"kubernetes.io/projected/65bd85d0-1590-4120-bf5b-73c3318f50ab-kube-api-access-dcnm2\") pod \"65bd85d0-1590-4120-bf5b-73c3318f50ab\" (UID: \"65bd85d0-1590-4120-bf5b-73c3318f50ab\") " Jul 10 17:20:03.070239 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:03.070049 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65bd85d0-1590-4120-bf5b-73c3318f50ab-webhook-cert\") pod \"65bd85d0-1590-4120-bf5b-73c3318f50ab\" (UID: \"65bd85d0-1590-4120-bf5b-73c3318f50ab\") " Jul 10 17:20:03.072466 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:03.072439 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65bd85d0-1590-4120-bf5b-73c3318f50ab-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "65bd85d0-1590-4120-bf5b-73c3318f50ab" (UID: "65bd85d0-1590-4120-bf5b-73c3318f50ab"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:20:03.072568 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:03.072441 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65bd85d0-1590-4120-bf5b-73c3318f50ab-kube-api-access-dcnm2" (OuterVolumeSpecName: "kube-api-access-dcnm2") pod "65bd85d0-1590-4120-bf5b-73c3318f50ab" (UID: "65bd85d0-1590-4120-bf5b-73c3318f50ab"). InnerVolumeSpecName "kube-api-access-dcnm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:20:03.171541 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:03.171500 2583 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65bd85d0-1590-4120-bf5b-73c3318f50ab-webhook-cert\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:20:03.171541 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:03.171532 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcnm2\" (UniqueName: \"kubernetes.io/projected/65bd85d0-1590-4120-bf5b-73c3318f50ab-kube-api-access-dcnm2\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:20:03.761956 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:03.761921 2583 generic.go:356] "Generic (PLEG): container finished" podID="65bd85d0-1590-4120-bf5b-73c3318f50ab" containerID="ed0797751afa0265a2bb7dc46779040cf34a2040991df469d68d3e9eab788ff4" exitCode=0 Jul 10 17:20:03.762361 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:03.761966 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d9b574468-8f4bs" Jul 10 17:20:03.762361 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:03.761997 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d9b574468-8f4bs" event={"ID":"65bd85d0-1590-4120-bf5b-73c3318f50ab","Type":"ContainerDied","Data":"ed0797751afa0265a2bb7dc46779040cf34a2040991df469d68d3e9eab788ff4"} Jul 10 17:20:03.762361 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:03.762051 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d9b574468-8f4bs" event={"ID":"65bd85d0-1590-4120-bf5b-73c3318f50ab","Type":"ContainerDied","Data":"78ba482b0697d0a4f8a685e98b842ca03b99c052e0b4b1e631d61829c6d49a84"} Jul 10 17:20:03.762361 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:03.762074 2583 scope.go:117] "RemoveContainer" containerID="ed0797751afa0265a2bb7dc46779040cf34a2040991df469d68d3e9eab788ff4" Jul 10 17:20:03.771462 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:03.771266 2583 scope.go:117] "RemoveContainer" containerID="ed0797751afa0265a2bb7dc46779040cf34a2040991df469d68d3e9eab788ff4" Jul 10 17:20:03.771596 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:20:03.771578 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed0797751afa0265a2bb7dc46779040cf34a2040991df469d68d3e9eab788ff4\": container with ID starting with ed0797751afa0265a2bb7dc46779040cf34a2040991df469d68d3e9eab788ff4 not found: ID does not exist" containerID="ed0797751afa0265a2bb7dc46779040cf34a2040991df469d68d3e9eab788ff4" Jul 10 17:20:03.771651 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:03.771608 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed0797751afa0265a2bb7dc46779040cf34a2040991df469d68d3e9eab788ff4"} err="failed to get container status \"ed0797751afa0265a2bb7dc46779040cf34a2040991df469d68d3e9eab788ff4\": rpc error: code = NotFound desc = could not find container \"ed0797751afa0265a2bb7dc46779040cf34a2040991df469d68d3e9eab788ff4\": container with ID starting with ed0797751afa0265a2bb7dc46779040cf34a2040991df469d68d3e9eab788ff4 not found: ID does not exist" Jul 10 17:20:03.784877 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:03.784842 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d9b574468-8f4bs"] Jul 10 17:20:03.786986 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:03.786959 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7d9b574468-8f4bs"] Jul 10 17:20:04.206255 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:04.206221 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65bd85d0-1590-4120-bf5b-73c3318f50ab" path="/var/lib/kubelet/pods/65bd85d0-1590-4120-bf5b-73c3318f50ab/volumes" Jul 10 17:20:23.798067 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:23.798028 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-6bjhw"] Jul 10 17:20:23.798531 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:23.798377 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65bd85d0-1590-4120-bf5b-73c3318f50ab" containerName="manager" Jul 10 17:20:23.798531 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:23.798402 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="65bd85d0-1590-4120-bf5b-73c3318f50ab" containerName="manager" Jul 10 17:20:23.798531 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:23.798478 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="65bd85d0-1590-4120-bf5b-73c3318f50ab" containerName="manager" Jul 10 17:20:23.810760 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:23.810736 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-6bjhw" Jul 10 17:20:23.814925 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:23.814898 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 10 17:20:23.815221 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:23.815196 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-jjckd\"" Jul 10 17:20:23.816593 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:23.816571 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-6bjhw"] Jul 10 17:20:23.841009 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:23.840981 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-qrscc"] Jul 10 17:20:23.843724 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:23.843704 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-qrscc" Jul 10 17:20:23.852874 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:23.852850 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-qrscc"] Jul 10 17:20:23.948869 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:23.948828 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5cf1f6-d727-424a-82b4-4581245dba8c-plugins-config-volume\") pod \"payload-pre-processing-7964457696-qrscc\" (UID: \"ee5cf1f6-d727-424a-82b4-4581245dba8c\") " pod="openshift-ingress/payload-pre-processing-7964457696-qrscc" Jul 10 17:20:23.949070 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:23.948943 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c69c3ec1-fa81-4c47-ac65-c478feee0a14-plugins-config-volume\") pod \"payload-processing-58f86c7798-6bjhw\" (UID: \"c69c3ec1-fa81-4c47-ac65-c478feee0a14\") " pod="openshift-ingress/payload-processing-58f86c7798-6bjhw" Jul 10 17:20:23.949070 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:23.949008 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9r74\" (UniqueName: \"kubernetes.io/projected/c69c3ec1-fa81-4c47-ac65-c478feee0a14-kube-api-access-w9r74\") pod \"payload-processing-58f86c7798-6bjhw\" (UID: \"c69c3ec1-fa81-4c47-ac65-c478feee0a14\") " pod="openshift-ingress/payload-processing-58f86c7798-6bjhw" Jul 10 17:20:23.949070 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:23.949035 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5l6c\" (UniqueName: \"kubernetes.io/projected/ee5cf1f6-d727-424a-82b4-4581245dba8c-kube-api-access-l5l6c\") pod \"payload-pre-processing-7964457696-qrscc\" (UID: \"ee5cf1f6-d727-424a-82b4-4581245dba8c\") " pod="openshift-ingress/payload-pre-processing-7964457696-qrscc" Jul 10 17:20:24.049512 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:24.049409 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5l6c\" (UniqueName: \"kubernetes.io/projected/ee5cf1f6-d727-424a-82b4-4581245dba8c-kube-api-access-l5l6c\") pod \"payload-pre-processing-7964457696-qrscc\" (UID: \"ee5cf1f6-d727-424a-82b4-4581245dba8c\") " pod="openshift-ingress/payload-pre-processing-7964457696-qrscc" Jul 10 17:20:24.049512 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:24.049456 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5cf1f6-d727-424a-82b4-4581245dba8c-plugins-config-volume\") pod \"payload-pre-processing-7964457696-qrscc\" (UID: \"ee5cf1f6-d727-424a-82b4-4581245dba8c\") " pod="openshift-ingress/payload-pre-processing-7964457696-qrscc" Jul 10 17:20:24.049512 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:24.049516 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c69c3ec1-fa81-4c47-ac65-c478feee0a14-plugins-config-volume\") pod \"payload-processing-58f86c7798-6bjhw\" (UID: \"c69c3ec1-fa81-4c47-ac65-c478feee0a14\") " pod="openshift-ingress/payload-processing-58f86c7798-6bjhw" Jul 10 17:20:24.049807 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:24.049559 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9r74\" (UniqueName: \"kubernetes.io/projected/c69c3ec1-fa81-4c47-ac65-c478feee0a14-kube-api-access-w9r74\") pod \"payload-processing-58f86c7798-6bjhw\" (UID: \"c69c3ec1-fa81-4c47-ac65-c478feee0a14\") " pod="openshift-ingress/payload-processing-58f86c7798-6bjhw" Jul 10 17:20:24.050257 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:24.050236 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5cf1f6-d727-424a-82b4-4581245dba8c-plugins-config-volume\") pod \"payload-pre-processing-7964457696-qrscc\" (UID: \"ee5cf1f6-d727-424a-82b4-4581245dba8c\") " pod="openshift-ingress/payload-pre-processing-7964457696-qrscc" Jul 10 17:20:24.050303 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:24.050251 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c69c3ec1-fa81-4c47-ac65-c478feee0a14-plugins-config-volume\") pod \"payload-processing-58f86c7798-6bjhw\" (UID: \"c69c3ec1-fa81-4c47-ac65-c478feee0a14\") " pod="openshift-ingress/payload-processing-58f86c7798-6bjhw" Jul 10 17:20:24.059105 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:24.059069 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9r74\" (UniqueName: \"kubernetes.io/projected/c69c3ec1-fa81-4c47-ac65-c478feee0a14-kube-api-access-w9r74\") pod \"payload-processing-58f86c7798-6bjhw\" (UID: \"c69c3ec1-fa81-4c47-ac65-c478feee0a14\") " pod="openshift-ingress/payload-processing-58f86c7798-6bjhw" Jul 10 17:20:24.059105 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:24.059099 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5l6c\" (UniqueName: \"kubernetes.io/projected/ee5cf1f6-d727-424a-82b4-4581245dba8c-kube-api-access-l5l6c\") pod \"payload-pre-processing-7964457696-qrscc\" (UID: \"ee5cf1f6-d727-424a-82b4-4581245dba8c\") " pod="openshift-ingress/payload-pre-processing-7964457696-qrscc" Jul 10 17:20:24.122916 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:24.122835 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-6bjhw" Jul 10 17:20:24.155298 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:24.155263 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-qrscc" Jul 10 17:20:24.285640 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:24.285599 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-6bjhw"] Jul 10 17:20:24.287215 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:20:24.287184 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc69c3ec1_fa81_4c47_ac65_c478feee0a14.slice/crio-f9f3e1df7a6d8ed182894b80c63f24084ffd567acc2fe1d0268b3844c48d5908 WatchSource:0}: Error finding container f9f3e1df7a6d8ed182894b80c63f24084ffd567acc2fe1d0268b3844c48d5908: Status 404 returned error can't find the container with id f9f3e1df7a6d8ed182894b80c63f24084ffd567acc2fe1d0268b3844c48d5908 Jul 10 17:20:24.316184 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:24.316147 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-qrscc"] Jul 10 17:20:24.318241 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:20:24.318211 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee5cf1f6_d727_424a_82b4_4581245dba8c.slice/crio-0b54ae0a686c4d3626e0dd4f3545f4d148587d71dfd429ad6c5c5cb0e1f3bc07 WatchSource:0}: Error finding container 0b54ae0a686c4d3626e0dd4f3545f4d148587d71dfd429ad6c5c5cb0e1f3bc07: Status 404 returned error can't find the container with id 0b54ae0a686c4d3626e0dd4f3545f4d148587d71dfd429ad6c5c5cb0e1f3bc07 Jul 10 17:20:24.873263 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:24.873224 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-qrscc" event={"ID":"ee5cf1f6-d727-424a-82b4-4581245dba8c","Type":"ContainerStarted","Data":"0b54ae0a686c4d3626e0dd4f3545f4d148587d71dfd429ad6c5c5cb0e1f3bc07"} Jul 10 17:20:24.874825 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:24.874797 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-6bjhw" event={"ID":"c69c3ec1-fa81-4c47-ac65-c478feee0a14","Type":"ContainerStarted","Data":"f9f3e1df7a6d8ed182894b80c63f24084ffd567acc2fe1d0268b3844c48d5908"} Jul 10 17:20:26.887524 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:26.887487 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-6bjhw" event={"ID":"c69c3ec1-fa81-4c47-ac65-c478feee0a14","Type":"ContainerStarted","Data":"1354902fbd3f13b4564585935e822d31c023acacc4909a71c158f1e88aeaf6a0"} Jul 10 17:20:26.888018 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:26.887578 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-6bjhw" Jul 10 17:20:26.889044 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:26.889020 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-qrscc" event={"ID":"ee5cf1f6-d727-424a-82b4-4581245dba8c","Type":"ContainerStarted","Data":"c737fa9c36a71cb4f4bd340ce9f50e128c8f5b7cb1a7d6e5a9a903649d5f46cc"} Jul 10 17:20:26.889142 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:26.889080 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-qrscc" Jul 10 17:20:26.906352 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:26.906289 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-6bjhw" podStartSLOduration=1.876393235 podStartE2EDuration="3.906265378s" totalImagesPullingTime="2.029872143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:20:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:20:26.901215406 +0000 UTC m=+559.228912946" watchObservedRunningTime="2026-07-10 17:20:26.906265378 +0000 UTC m=+559.233962918" Jul 10 17:20:26.919171 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:26.919104 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-qrscc" podStartSLOduration=1.921674952 podStartE2EDuration="3.91906725s" totalImagesPullingTime="1.997392298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:20:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:20:26.91645038 +0000 UTC m=+559.244147921" watchObservedRunningTime="2026-07-10 17:20:26.91906725 +0000 UTC m=+559.246764790" Jul 10 17:20:31.921882 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:31.921846 2583 generic.go:356] "Generic (PLEG): container finished" podID="553d3bf3-9e01-4b25-9ac2-01b0da4faf79" containerID="b01c5f6748242f1391e0709566d530c1c4e159c80a416f417e9bdd765c4c0dc7" exitCode=0 Jul 10 17:20:31.922310 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:31.921917 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" event={"ID":"553d3bf3-9e01-4b25-9ac2-01b0da4faf79","Type":"ContainerDied","Data":"b01c5f6748242f1391e0709566d530c1c4e159c80a416f417e9bdd765c4c0dc7"} Jul 10 17:20:31.922310 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:31.922293 2583 scope.go:117] "RemoveContainer" containerID="b01c5f6748242f1391e0709566d530c1c4e159c80a416f417e9bdd765c4c0dc7" Jul 10 17:20:31.923258 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:31.923231 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:20:31.923355 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:31.923295 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:20:31.923355 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:31.923331 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:20:32.929069 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:32.929036 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" event={"ID":"553d3bf3-9e01-4b25-9ac2-01b0da4faf79","Type":"ContainerStarted","Data":"260ad8dcaf50ea959b24a356bd6b2b8d04792b11d6081df525ac74d025789431"} Jul 10 17:20:33.140470 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:33.140431 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:20:33.140672 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:33.140648 2583 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-mlrv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 10 17:20:33.140732 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:33.140708 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" podUID="553d3bf3-9e01-4b25-9ac2-01b0da4faf79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 10 17:20:34.140303 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:34.140267 2583 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-mlrv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 10 17:20:34.140754 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:34.140323 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" podUID="553d3bf3-9e01-4b25-9ac2-01b0da4faf79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 10 17:20:35.140760 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:35.140724 2583 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-mlrv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 10 17:20:35.141138 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:35.140781 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" podUID="553d3bf3-9e01-4b25-9ac2-01b0da4faf79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 10 17:20:36.140706 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:36.140668 2583 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-mlrv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 10 17:20:36.140885 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:36.140724 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" podUID="553d3bf3-9e01-4b25-9ac2-01b0da4faf79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 10 17:20:37.144260 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:37.144231 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:20:37.144769 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:37.144640 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:20:37.145419 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:37.145384 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mlrv6" Jul 10 17:20:37.898572 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:37.898535 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-6bjhw" Jul 10 17:20:37.898902 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:37.898886 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-qrscc" Jul 10 17:20:44.143643 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:44.143610 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl"] Jul 10 17:20:44.146417 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:44.146374 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl" Jul 10 17:20:44.148803 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:44.148770 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 17:20:44.148803 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:44.148792 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 17:20:44.149670 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:44.149652 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5vqx5\"" Jul 10 17:20:44.149788 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:44.149720 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 10 17:20:44.162170 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:44.162138 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl"] Jul 10 17:20:44.331634 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:44.331595 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl\" (UID: \"eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl" Jul 10 17:20:44.331785 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:44.331645 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl\" (UID: \"eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl" Jul 10 17:20:44.331785 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:44.331672 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf5g4\" (UniqueName: \"kubernetes.io/projected/eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb-kube-api-access-mf5g4\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl\" (UID: \"eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl" Jul 10 17:20:44.331785 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:44.331711 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl\" (UID: \"eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl" Jul 10 17:20:44.331785 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:44.331728 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl\" (UID: \"eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl" Jul 10 17:20:44.331920 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:44.331805 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl\" (UID: \"eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl" Jul 10 17:20:44.433066 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:44.433029 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl\" (UID: \"eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl" Jul 10 17:20:44.433228 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:44.433082 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl\" (UID: \"eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl" Jul 10 17:20:44.433228 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:44.433109 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf5g4\" (UniqueName: \"kubernetes.io/projected/eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb-kube-api-access-mf5g4\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl\" (UID: \"eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl" Jul 10 17:20:44.433228 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:44.433151 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl\" (UID: \"eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl" Jul 10 17:20:44.433354 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:44.433289 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl\" (UID: \"eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl" Jul 10 17:20:44.433354 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:44.433330 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl\" (UID: \"eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl" Jul 10 17:20:44.433539 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:44.433518 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl\" (UID: \"eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl" Jul 10 17:20:44.433602 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:44.433590 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl\" (UID: \"eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl" Jul 10 17:20:44.433701 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:44.433680 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl\" (UID: \"eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl" Jul 10 17:20:44.435570 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:44.435549 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl\" (UID: \"eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl" Jul 10 17:20:44.435844 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:44.435823 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl\" (UID: \"eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl" Jul 10 17:20:44.440728 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:44.440705 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf5g4\" (UniqueName: \"kubernetes.io/projected/eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb-kube-api-access-mf5g4\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl\" (UID: \"eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl" Jul 10 17:20:44.463810 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:44.463780 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl" Jul 10 17:20:44.600896 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:44.600853 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl"] Jul 10 17:20:44.603064 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:20:44.603031 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb1e1e1c_a404_4c9a_a017_3c9ec4d0fdbb.slice/crio-8e4a173bfeced3fe18496bf0edc173478bfee13af9287977730f440feb003275 WatchSource:0}: Error finding container 8e4a173bfeced3fe18496bf0edc173478bfee13af9287977730f440feb003275: Status 404 returned error can't find the container with id 8e4a173bfeced3fe18496bf0edc173478bfee13af9287977730f440feb003275 Jul 10 17:20:44.995886 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:44.995804 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl" event={"ID":"eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb","Type":"ContainerStarted","Data":"8e4a173bfeced3fe18496bf0edc173478bfee13af9287977730f440feb003275"} Jul 10 17:20:46.004777 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:46.004741 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl" event={"ID":"eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb","Type":"ContainerStarted","Data":"d97d1189d0e2ce2e5358229c4a8dd6f5d00ba04608609a2640f53ea261992065"} Jul 10 17:20:46.022416 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:46.022337 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl" podStartSLOduration=0.846941332 podStartE2EDuration="2.022319981s" totalImagesPullingTime="1.175378649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:20:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:20:46.020108075 +0000 UTC m=+578.347805614" watchObservedRunningTime="2026-07-10 17:20:46.022319981 +0000 UTC m=+578.350017521" Jul 10 17:20:54.464765 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:54.464722 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl" Jul 10 17:20:54.465203 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:54.464785 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl" Jul 10 17:20:54.477348 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:54.477322 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl" Jul 10 17:20:55.061736 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:20:55.061708 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl" Jul 10 17:21:08.100131 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:21:08.100099 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js69w_e41baaf7-4171-4df1-8bdd-d0697a61fdce/ovn-acl-logging/0.log" Jul 10 17:21:08.100898 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:21:08.100878 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js69w_e41baaf7-4171-4df1-8bdd-d0697a61fdce/ovn-acl-logging/0.log" Jul 10 17:21:17.359946 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:21:17.359913 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv"] Jul 10 17:21:17.364331 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:21:17.364309 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv" Jul 10 17:21:17.368101 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:21:17.368077 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 10 17:21:17.376671 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:21:17.376640 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv"] Jul 10 17:21:17.525944 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:21:17.525909 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a6ae435-53b4-49d0-9baa-72d31ed73529-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv\" (UID: \"5a6ae435-53b4-49d0-9baa-72d31ed73529\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv" Jul 10 17:21:17.526164 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:21:17.525950 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a6ae435-53b4-49d0-9baa-72d31ed73529-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv\" (UID: \"5a6ae435-53b4-49d0-9baa-72d31ed73529\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv" Jul 10 17:21:17.526164 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:21:17.526026 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a6ae435-53b4-49d0-9baa-72d31ed73529-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv\" (UID: \"5a6ae435-53b4-49d0-9baa-72d31ed73529\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv" Jul 10 17:21:17.526164 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:21:17.526113 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a6ae435-53b4-49d0-9baa-72d31ed73529-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv\" (UID: \"5a6ae435-53b4-49d0-9baa-72d31ed73529\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv" Jul 10 17:21:17.526164 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:21:17.526144 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a6ae435-53b4-49d0-9baa-72d31ed73529-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv\" (UID: \"5a6ae435-53b4-49d0-9baa-72d31ed73529\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv" Jul 10 17:21:17.526331 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:21:17.526188 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46cz6\" (UniqueName: \"kubernetes.io/projected/5a6ae435-53b4-49d0-9baa-72d31ed73529-kube-api-access-46cz6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv\" (UID: \"5a6ae435-53b4-49d0-9baa-72d31ed73529\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv" Jul 10 17:21:17.627281 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:21:17.627177 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a6ae435-53b4-49d0-9baa-72d31ed73529-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv\" (UID: \"5a6ae435-53b4-49d0-9baa-72d31ed73529\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv" Jul 10 17:21:17.627281 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:21:17.627224 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a6ae435-53b4-49d0-9baa-72d31ed73529-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv\" (UID: \"5a6ae435-53b4-49d0-9baa-72d31ed73529\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv" Jul 10 17:21:17.627488 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:21:17.627357 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46cz6\" (UniqueName: \"kubernetes.io/projected/5a6ae435-53b4-49d0-9baa-72d31ed73529-kube-api-access-46cz6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv\" (UID: \"5a6ae435-53b4-49d0-9baa-72d31ed73529\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv" Jul 10 17:21:17.627488 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:21:17.627467 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a6ae435-53b4-49d0-9baa-72d31ed73529-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv\" (UID: \"5a6ae435-53b4-49d0-9baa-72d31ed73529\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv" Jul 10 17:21:17.627568 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:21:17.627502 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a6ae435-53b4-49d0-9baa-72d31ed73529-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv\" (UID: \"5a6ae435-53b4-49d0-9baa-72d31ed73529\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv" Jul 10 17:21:17.627609 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:21:17.627566 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a6ae435-53b4-49d0-9baa-72d31ed73529-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv\" (UID: \"5a6ae435-53b4-49d0-9baa-72d31ed73529\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv" Jul 10 17:21:17.627859 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:21:17.627829 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a6ae435-53b4-49d0-9baa-72d31ed73529-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv\" (UID: \"5a6ae435-53b4-49d0-9baa-72d31ed73529\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv" Jul 10 17:21:17.627958 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:21:17.627886 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a6ae435-53b4-49d0-9baa-72d31ed73529-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv\" (UID: \"5a6ae435-53b4-49d0-9baa-72d31ed73529\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv" Jul 10 17:21:17.627958 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:21:17.627933 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a6ae435-53b4-49d0-9baa-72d31ed73529-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv\" (UID: \"5a6ae435-53b4-49d0-9baa-72d31ed73529\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv" Jul 10 17:21:17.629827 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:21:17.629802 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a6ae435-53b4-49d0-9baa-72d31ed73529-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv\" (UID: \"5a6ae435-53b4-49d0-9baa-72d31ed73529\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv" Jul 10 17:21:17.629953 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:21:17.629937 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a6ae435-53b4-49d0-9baa-72d31ed73529-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv\" (UID: \"5a6ae435-53b4-49d0-9baa-72d31ed73529\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv" Jul 10 17:21:17.635976 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:21:17.635950 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46cz6\" (UniqueName: \"kubernetes.io/projected/5a6ae435-53b4-49d0-9baa-72d31ed73529-kube-api-access-46cz6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv\" (UID: \"5a6ae435-53b4-49d0-9baa-72d31ed73529\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv" Jul 10 17:21:17.676636 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:21:17.676592 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv" Jul 10 17:21:17.819414 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:21:17.819368 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv"] Jul 10 17:21:17.821258 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:21:17.821229 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a6ae435_53b4_49d0_9baa_72d31ed73529.slice/crio-939d98af81e251d07e46a4bb1e386859dce4f11ee080ee210ada155c31edadcf WatchSource:0}: Error finding container 939d98af81e251d07e46a4bb1e386859dce4f11ee080ee210ada155c31edadcf: Status 404 returned error can't find the container with id 939d98af81e251d07e46a4bb1e386859dce4f11ee080ee210ada155c31edadcf Jul 10 17:21:18.178281 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:21:18.178241 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv" event={"ID":"5a6ae435-53b4-49d0-9baa-72d31ed73529","Type":"ContainerStarted","Data":"c5605abcaaf42d8392b0e760a4c40f7cc708fb673e811c08c204ae2403f9e46c"} Jul 10 17:21:18.178504 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:21:18.178304 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv" event={"ID":"5a6ae435-53b4-49d0-9baa-72d31ed73529","Type":"ContainerStarted","Data":"939d98af81e251d07e46a4bb1e386859dce4f11ee080ee210ada155c31edadcf"} Jul 10 17:21:18.202903 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:21:18.200553 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv" podStartSLOduration=1.01985779 podStartE2EDuration="1.200527722s" totalImagesPullingTime="180.669932ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:21:18.194562931 +0000 UTC m=+610.522260489" watchObservedRunningTime="2026-07-10 17:21:18.200527722 +0000 UTC m=+610.528225262" Jul 10 17:21:27.676955 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:21:27.676916 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv" Jul 10 17:21:27.677355 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:21:27.676987 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv" Jul 10 17:21:27.689592 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:21:27.689561 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv" Jul 10 17:21:28.240916 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:21:28.240889 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv" Jul 10 17:23:40.168453 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:40.168412 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b4655bb55-rwrpx"] Jul 10 17:23:40.169041 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:40.168695 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5b4655bb55-rwrpx" podUID="d131fb34-27ae-45de-b636-3c1b6131b57b" containerName="manager" containerID="cri-o://98f0afe894459aa037c3d99310b3991559ebf0f7ce8d8809a9e2507e26550ab9" gracePeriod=30 Jul 10 17:23:40.422567 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:40.422504 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b4655bb55-rwrpx" Jul 10 17:23:40.515674 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:40.515637 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d131fb34-27ae-45de-b636-3c1b6131b57b-webhook-cert\") pod \"d131fb34-27ae-45de-b636-3c1b6131b57b\" (UID: \"d131fb34-27ae-45de-b636-3c1b6131b57b\") " Jul 10 17:23:40.515838 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:40.515715 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdf62\" (UniqueName: \"kubernetes.io/projected/d131fb34-27ae-45de-b636-3c1b6131b57b-kube-api-access-tdf62\") pod \"d131fb34-27ae-45de-b636-3c1b6131b57b\" (UID: \"d131fb34-27ae-45de-b636-3c1b6131b57b\") " Jul 10 17:23:40.517904 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:40.517869 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d131fb34-27ae-45de-b636-3c1b6131b57b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d131fb34-27ae-45de-b636-3c1b6131b57b" (UID: "d131fb34-27ae-45de-b636-3c1b6131b57b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:23:40.517904 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:40.517863 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d131fb34-27ae-45de-b636-3c1b6131b57b-kube-api-access-tdf62" (OuterVolumeSpecName: "kube-api-access-tdf62") pod "d131fb34-27ae-45de-b636-3c1b6131b57b" (UID: "d131fb34-27ae-45de-b636-3c1b6131b57b"). InnerVolumeSpecName "kube-api-access-tdf62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:23:40.616862 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:40.616824 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdf62\" (UniqueName: \"kubernetes.io/projected/d131fb34-27ae-45de-b636-3c1b6131b57b-kube-api-access-tdf62\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:23:40.616862 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:40.616858 2583 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d131fb34-27ae-45de-b636-3c1b6131b57b-webhook-cert\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:23:40.911980 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:40.911892 2583 generic.go:356] "Generic (PLEG): container finished" podID="d131fb34-27ae-45de-b636-3c1b6131b57b" containerID="98f0afe894459aa037c3d99310b3991559ebf0f7ce8d8809a9e2507e26550ab9" exitCode=0 Jul 10 17:23:40.911980 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:40.911939 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b4655bb55-rwrpx" Jul 10 17:23:40.911980 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:40.911971 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b4655bb55-rwrpx" event={"ID":"d131fb34-27ae-45de-b636-3c1b6131b57b","Type":"ContainerDied","Data":"98f0afe894459aa037c3d99310b3991559ebf0f7ce8d8809a9e2507e26550ab9"} Jul 10 17:23:40.912211 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:40.912031 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b4655bb55-rwrpx" event={"ID":"d131fb34-27ae-45de-b636-3c1b6131b57b","Type":"ContainerDied","Data":"c7bccb324f44fa54dab54727595ba5ef42952a6f676e2eebcfcc35189e1e6e65"} Jul 10 17:23:40.912211 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:40.912053 2583 scope.go:117] "RemoveContainer" containerID="98f0afe894459aa037c3d99310b3991559ebf0f7ce8d8809a9e2507e26550ab9" Jul 10 17:23:40.921109 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:40.921092 2583 scope.go:117] "RemoveContainer" containerID="98f0afe894459aa037c3d99310b3991559ebf0f7ce8d8809a9e2507e26550ab9" Jul 10 17:23:40.921401 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:23:40.921367 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98f0afe894459aa037c3d99310b3991559ebf0f7ce8d8809a9e2507e26550ab9\": container with ID starting with 98f0afe894459aa037c3d99310b3991559ebf0f7ce8d8809a9e2507e26550ab9 not found: ID does not exist" containerID="98f0afe894459aa037c3d99310b3991559ebf0f7ce8d8809a9e2507e26550ab9" Jul 10 17:23:40.921455 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:40.921410 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98f0afe894459aa037c3d99310b3991559ebf0f7ce8d8809a9e2507e26550ab9"} err="failed to get container status \"98f0afe894459aa037c3d99310b3991559ebf0f7ce8d8809a9e2507e26550ab9\": rpc error: code = NotFound desc = could not find container \"98f0afe894459aa037c3d99310b3991559ebf0f7ce8d8809a9e2507e26550ab9\": container with ID starting with 98f0afe894459aa037c3d99310b3991559ebf0f7ce8d8809a9e2507e26550ab9 not found: ID does not exist" Jul 10 17:23:40.933539 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:40.933514 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b4655bb55-rwrpx"] Jul 10 17:23:40.935499 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:40.935479 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5b4655bb55-rwrpx"] Jul 10 17:23:41.671275 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:41.671243 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b4655bb55-h9s24"] Jul 10 17:23:41.671685 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:41.671643 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d131fb34-27ae-45de-b636-3c1b6131b57b" containerName="manager" Jul 10 17:23:41.671685 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:41.671654 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d131fb34-27ae-45de-b636-3c1b6131b57b" containerName="manager" Jul 10 17:23:41.671751 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:41.671728 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d131fb34-27ae-45de-b636-3c1b6131b57b" containerName="manager" Jul 10 17:23:41.675822 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:41.675804 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b4655bb55-h9s24" Jul 10 17:23:41.679564 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:41.679539 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 10 17:23:41.679718 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:41.679604 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 10 17:23:41.679718 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:41.679615 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5p2bc\"" Jul 10 17:23:41.685534 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:41.685502 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b4655bb55-h9s24"] Jul 10 17:23:41.827764 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:41.827731 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49640ea0-1609-4683-bf14-f14f0bdb5882-webhook-cert\") pod \"maas-controller-5b4655bb55-h9s24\" (UID: \"49640ea0-1609-4683-bf14-f14f0bdb5882\") " pod="opendatahub/maas-controller-5b4655bb55-h9s24" Jul 10 17:23:41.827937 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:41.827785 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq2j8\" (UniqueName: \"kubernetes.io/projected/49640ea0-1609-4683-bf14-f14f0bdb5882-kube-api-access-zq2j8\") pod \"maas-controller-5b4655bb55-h9s24\" (UID: \"49640ea0-1609-4683-bf14-f14f0bdb5882\") " pod="opendatahub/maas-controller-5b4655bb55-h9s24" Jul 10 17:23:41.928628 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:41.928537 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq2j8\" (UniqueName: \"kubernetes.io/projected/49640ea0-1609-4683-bf14-f14f0bdb5882-kube-api-access-zq2j8\") pod \"maas-controller-5b4655bb55-h9s24\" (UID: \"49640ea0-1609-4683-bf14-f14f0bdb5882\") " pod="opendatahub/maas-controller-5b4655bb55-h9s24" Jul 10 17:23:41.928791 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:41.928651 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49640ea0-1609-4683-bf14-f14f0bdb5882-webhook-cert\") pod \"maas-controller-5b4655bb55-h9s24\" (UID: \"49640ea0-1609-4683-bf14-f14f0bdb5882\") " pod="opendatahub/maas-controller-5b4655bb55-h9s24" Jul 10 17:23:41.931281 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:41.931255 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49640ea0-1609-4683-bf14-f14f0bdb5882-webhook-cert\") pod \"maas-controller-5b4655bb55-h9s24\" (UID: \"49640ea0-1609-4683-bf14-f14f0bdb5882\") " pod="opendatahub/maas-controller-5b4655bb55-h9s24" Jul 10 17:23:41.936643 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:41.936615 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq2j8\" (UniqueName: \"kubernetes.io/projected/49640ea0-1609-4683-bf14-f14f0bdb5882-kube-api-access-zq2j8\") pod \"maas-controller-5b4655bb55-h9s24\" (UID: \"49640ea0-1609-4683-bf14-f14f0bdb5882\") " pod="opendatahub/maas-controller-5b4655bb55-h9s24" Jul 10 17:23:41.986793 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:41.986748 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b4655bb55-h9s24" Jul 10 17:23:42.133517 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:42.133490 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b4655bb55-h9s24"] Jul 10 17:23:42.135750 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:23:42.135719 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49640ea0_1609_4683_bf14_f14f0bdb5882.slice/crio-4487f94965234faa7043662e218e141589df6827129d23af663b5040a9fcf4f4 WatchSource:0}: Error finding container 4487f94965234faa7043662e218e141589df6827129d23af663b5040a9fcf4f4: Status 404 returned error can't find the container with id 4487f94965234faa7043662e218e141589df6827129d23af663b5040a9fcf4f4 Jul 10 17:23:42.137171 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:42.137149 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 17:23:42.205891 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:42.205809 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d131fb34-27ae-45de-b636-3c1b6131b57b" path="/var/lib/kubelet/pods/d131fb34-27ae-45de-b636-3c1b6131b57b/volumes" Jul 10 17:23:42.924935 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:42.924908 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b4655bb55-h9s24" event={"ID":"49640ea0-1609-4683-bf14-f14f0bdb5882","Type":"ContainerStarted","Data":"31fa2197907babfb7c9563557b36a58d895eff4285b316bfe91f28518360f89a"} Jul 10 17:23:42.925341 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:42.924957 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b4655bb55-h9s24" event={"ID":"49640ea0-1609-4683-bf14-f14f0bdb5882","Type":"ContainerStarted","Data":"4487f94965234faa7043662e218e141589df6827129d23af663b5040a9fcf4f4"} Jul 10 17:23:42.925341 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:42.925023 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b4655bb55-h9s24" Jul 10 17:23:42.941285 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:42.941234 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5b4655bb55-h9s24" podStartSLOduration=1.4607694549999999 podStartE2EDuration="1.941219324s" totalImagesPullingTime="480.449869ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:23:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:23:42.940570298 +0000 UTC m=+755.268267838" watchObservedRunningTime="2026-07-10 17:23:42.941219324 +0000 UTC m=+755.268916867" Jul 10 17:23:53.933407 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:23:53.933355 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b4655bb55-h9s24" Jul 10 17:26:08.134511 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:26:08.134430 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js69w_e41baaf7-4171-4df1-8bdd-d0697a61fdce/ovn-acl-logging/0.log" Jul 10 17:26:08.136853 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:26:08.135926 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js69w_e41baaf7-4171-4df1-8bdd-d0697a61fdce/ovn-acl-logging/0.log" Jul 10 17:28:26.015836 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:28:26.015799 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwzhn"] Jul 10 17:28:26.016380 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:28:26.016012 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwzhn" podUID="7e9508f1-f98f-43c9-bd18-efdce98e6308" containerName="manager" containerID="cri-o://49cb3131cd19227043c72d24ad23f83b63a4c84e87e2bd4d24db42e4b7323b2d" gracePeriod=10 Jul 10 17:28:28.166783 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:28:28.166758 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwzhn" Jul 10 17:28:28.229859 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:28:28.229824 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e9508f1-f98f-43c9-bd18-efdce98e6308-extensions-socket-volume\") pod \"7e9508f1-f98f-43c9-bd18-efdce98e6308\" (UID: \"7e9508f1-f98f-43c9-bd18-efdce98e6308\") " Jul 10 17:28:28.230045 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:28:28.229932 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5m6rf\" (UniqueName: \"kubernetes.io/projected/7e9508f1-f98f-43c9-bd18-efdce98e6308-kube-api-access-5m6rf\") pod \"7e9508f1-f98f-43c9-bd18-efdce98e6308\" (UID: \"7e9508f1-f98f-43c9-bd18-efdce98e6308\") " Jul 10 17:28:28.230256 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:28:28.230231 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9508f1-f98f-43c9-bd18-efdce98e6308-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7e9508f1-f98f-43c9-bd18-efdce98e6308" (UID: "7e9508f1-f98f-43c9-bd18-efdce98e6308"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:28:28.232254 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:28:28.232214 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e9508f1-f98f-43c9-bd18-efdce98e6308-kube-api-access-5m6rf" (OuterVolumeSpecName: "kube-api-access-5m6rf") pod "7e9508f1-f98f-43c9-bd18-efdce98e6308" (UID: "7e9508f1-f98f-43c9-bd18-efdce98e6308"). InnerVolumeSpecName "kube-api-access-5m6rf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:28:28.330850 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:28:28.330758 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m6rf\" (UniqueName: \"kubernetes.io/projected/7e9508f1-f98f-43c9-bd18-efdce98e6308-kube-api-access-5m6rf\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:28:28.330850 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:28:28.330789 2583 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e9508f1-f98f-43c9-bd18-efdce98e6308-extensions-socket-volume\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:28:28.439668 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:28:28.439636 2583 generic.go:356] "Generic (PLEG): container finished" podID="7e9508f1-f98f-43c9-bd18-efdce98e6308" containerID="49cb3131cd19227043c72d24ad23f83b63a4c84e87e2bd4d24db42e4b7323b2d" exitCode=0 Jul 10 17:28:28.439852 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:28:28.439691 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwzhn" Jul 10 17:28:28.439852 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:28:28.439697 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwzhn" event={"ID":"7e9508f1-f98f-43c9-bd18-efdce98e6308","Type":"ContainerDied","Data":"49cb3131cd19227043c72d24ad23f83b63a4c84e87e2bd4d24db42e4b7323b2d"} Jul 10 17:28:28.439852 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:28:28.439739 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwzhn" event={"ID":"7e9508f1-f98f-43c9-bd18-efdce98e6308","Type":"ContainerDied","Data":"31ce59fa509e81d4c7338b4a53f3e1fb35080b8757c0f1a34c3c679ae37f65a9"} Jul 10 17:28:28.439852 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:28:28.439760 2583 scope.go:117] "RemoveContainer" containerID="49cb3131cd19227043c72d24ad23f83b63a4c84e87e2bd4d24db42e4b7323b2d" Jul 10 17:28:28.449899 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:28:28.449880 2583 scope.go:117] "RemoveContainer" containerID="49cb3131cd19227043c72d24ad23f83b63a4c84e87e2bd4d24db42e4b7323b2d" Jul 10 17:28:28.450200 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:28:28.450181 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49cb3131cd19227043c72d24ad23f83b63a4c84e87e2bd4d24db42e4b7323b2d\": container with ID starting with 49cb3131cd19227043c72d24ad23f83b63a4c84e87e2bd4d24db42e4b7323b2d not found: ID does not exist" containerID="49cb3131cd19227043c72d24ad23f83b63a4c84e87e2bd4d24db42e4b7323b2d" Jul 10 17:28:28.450265 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:28:28.450212 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49cb3131cd19227043c72d24ad23f83b63a4c84e87e2bd4d24db42e4b7323b2d"} err="failed to get container status \"49cb3131cd19227043c72d24ad23f83b63a4c84e87e2bd4d24db42e4b7323b2d\": rpc error: code = NotFound desc = could not find container \"49cb3131cd19227043c72d24ad23f83b63a4c84e87e2bd4d24db42e4b7323b2d\": container with ID starting with 49cb3131cd19227043c72d24ad23f83b63a4c84e87e2bd4d24db42e4b7323b2d not found: ID does not exist" Jul 10 17:28:28.461046 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:28:28.461021 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwzhn"] Jul 10 17:28:28.464816 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:28:28.464791 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwzhn"] Jul 10 17:28:30.206027 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:28:30.205993 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e9508f1-f98f-43c9-bd18-efdce98e6308" path="/var/lib/kubelet/pods/7e9508f1-f98f-43c9-bd18-efdce98e6308/volumes" Jul 10 17:29:27.106642 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:29:27.106610 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-db25c"] Jul 10 17:29:27.107157 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:29:27.107033 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e9508f1-f98f-43c9-bd18-efdce98e6308" containerName="manager" Jul 10 17:29:27.107157 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:29:27.107045 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e9508f1-f98f-43c9-bd18-efdce98e6308" containerName="manager" Jul 10 17:29:27.107157 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:29:27.107101 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e9508f1-f98f-43c9-bd18-efdce98e6308" containerName="manager" Jul 10 17:29:27.111433 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:29:27.111410 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-db25c" Jul 10 17:29:27.114520 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:29:27.114495 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-q78b9\"" Jul 10 17:29:27.122064 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:29:27.122037 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-db25c"] Jul 10 17:29:27.145241 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:29:27.145198 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0279bca-c281-4a5e-ab40-f74c854cde9c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-db25c\" (UID: \"e0279bca-c281-4a5e-ab40-f74c854cde9c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-db25c" Jul 10 17:29:27.145468 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:29:27.145323 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntd86\" (UniqueName: \"kubernetes.io/projected/e0279bca-c281-4a5e-ab40-f74c854cde9c-kube-api-access-ntd86\") pod \"kuadrant-operator-controller-manager-86878c769f-db25c\" (UID: \"e0279bca-c281-4a5e-ab40-f74c854cde9c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-db25c" Jul 10 17:29:27.246281 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:29:27.246239 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntd86\" (UniqueName: \"kubernetes.io/projected/e0279bca-c281-4a5e-ab40-f74c854cde9c-kube-api-access-ntd86\") pod \"kuadrant-operator-controller-manager-86878c769f-db25c\" (UID: \"e0279bca-c281-4a5e-ab40-f74c854cde9c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-db25c" Jul 10 17:29:27.246526 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:29:27.246305 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0279bca-c281-4a5e-ab40-f74c854cde9c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-db25c\" (UID: \"e0279bca-c281-4a5e-ab40-f74c854cde9c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-db25c" Jul 10 17:29:27.246684 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:29:27.246666 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0279bca-c281-4a5e-ab40-f74c854cde9c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-db25c\" (UID: \"e0279bca-c281-4a5e-ab40-f74c854cde9c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-db25c" Jul 10 17:29:27.255508 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:29:27.255480 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntd86\" (UniqueName: \"kubernetes.io/projected/e0279bca-c281-4a5e-ab40-f74c854cde9c-kube-api-access-ntd86\") pod \"kuadrant-operator-controller-manager-86878c769f-db25c\" (UID: \"e0279bca-c281-4a5e-ab40-f74c854cde9c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-db25c" Jul 10 17:29:27.422451 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:29:27.422385 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-db25c" Jul 10 17:29:27.574133 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:29:27.574107 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-db25c"] Jul 10 17:29:27.575773 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:29:27.575748 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0279bca_c281_4a5e_ab40_f74c854cde9c.slice/crio-b5838a69180576b4c079bb70577e5be570f6ef855c936cf227106486dad3c027 WatchSource:0}: Error finding container b5838a69180576b4c079bb70577e5be570f6ef855c936cf227106486dad3c027: Status 404 returned error can't find the container with id b5838a69180576b4c079bb70577e5be570f6ef855c936cf227106486dad3c027 Jul 10 17:29:27.758911 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:29:27.758800 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-db25c" event={"ID":"e0279bca-c281-4a5e-ab40-f74c854cde9c","Type":"ContainerStarted","Data":"9053f40b25b6c667d6326d5ebfb345d07a79195bbd3cfaedde140b6847ba88f7"} Jul 10 17:29:27.758911 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:29:27.758871 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-db25c" event={"ID":"e0279bca-c281-4a5e-ab40-f74c854cde9c","Type":"ContainerStarted","Data":"b5838a69180576b4c079bb70577e5be570f6ef855c936cf227106486dad3c027"} Jul 10 17:29:27.758911 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:29:27.758901 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-db25c" Jul 10 17:29:27.780602 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:29:27.780543 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-db25c" podStartSLOduration=0.780520219 podStartE2EDuration="780.520219ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:29:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:29:27.779171991 +0000 UTC m=+1100.106869531" watchObservedRunningTime="2026-07-10 17:29:27.780520219 +0000 UTC m=+1100.108217760" Jul 10 17:29:38.766136 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:29:38.766108 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-db25c" Jul 10 17:31:08.166627 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:31:08.166601 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js69w_e41baaf7-4171-4df1-8bdd-d0697a61fdce/ovn-acl-logging/0.log" Jul 10 17:31:08.168824 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:31:08.168427 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js69w_e41baaf7-4171-4df1-8bdd-d0697a61fdce/ovn-acl-logging/0.log" Jul 10 17:35:47.215308 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:35:47.215262 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dddaaba5-pbxjc"] Jul 10 17:35:47.219321 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:35:47.219298 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dddaaba5-pbxjc" Jul 10 17:35:47.221975 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:35:47.221946 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m7h25\"" Jul 10 17:35:47.225587 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:35:47.225557 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dddaaba5-pbxjc"] Jul 10 17:35:47.368258 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:35:47.368213 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/342379b1-29b3-470a-aade-18e6a468989c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-dddaaba5-pbxjc\" (UID: \"342379b1-29b3-470a-aade-18e6a468989c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dddaaba5-pbxjc" Jul 10 17:35:47.468985 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:35:47.468874 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/342379b1-29b3-470a-aade-18e6a468989c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-dddaaba5-pbxjc\" (UID: \"342379b1-29b3-470a-aade-18e6a468989c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dddaaba5-pbxjc" Jul 10 17:35:47.469615 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:35:47.469590 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/342379b1-29b3-470a-aade-18e6a468989c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-dddaaba5-pbxjc\" (UID: \"342379b1-29b3-470a-aade-18e6a468989c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dddaaba5-pbxjc" Jul 10 17:35:47.531770 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:35:47.531721 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dddaaba5-pbxjc" Jul 10 17:35:47.675856 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:35:47.675816 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dddaaba5-pbxjc"] Jul 10 17:35:47.688659 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:35:47.688631 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 17:35:47.749517 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:35:47.749416 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dddaaba5-pbxjc" event={"ID":"342379b1-29b3-470a-aade-18e6a468989c","Type":"ContainerStarted","Data":"999a52acb771e6ab200f97ca9cc21107e67a78b03fe69a878f1517f4e4c6972c"} Jul 10 17:35:50.766599 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:35:50.766559 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dddaaba5-pbxjc" event={"ID":"342379b1-29b3-470a-aade-18e6a468989c","Type":"ContainerStarted","Data":"329c54a6e03d9b6b0f177ca0ae02d3db8348c38426ed243c932bf2de183f2a0d"} Jul 10 17:35:50.780856 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:35:50.780801 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dddaaba5-pbxjc" podStartSLOduration=1.458103168 podStartE2EDuration="3.780784326s" totalImagesPullingTime="2.322681158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:35:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:35:50.780521741 +0000 UTC m=+1483.108219280" watchObservedRunningTime="2026-07-10 17:35:50.780784326 +0000 UTC m=+1483.108481866" Jul 10 17:35:51.773792 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:35:51.773753 2583 generic.go:356] "Generic (PLEG): container finished" podID="342379b1-29b3-470a-aade-18e6a468989c" containerID="329c54a6e03d9b6b0f177ca0ae02d3db8348c38426ed243c932bf2de183f2a0d" exitCode=0 Jul 10 17:35:51.774179 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:35:51.773818 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dddaaba5-pbxjc" event={"ID":"342379b1-29b3-470a-aade-18e6a468989c","Type":"ContainerDied","Data":"329c54a6e03d9b6b0f177ca0ae02d3db8348c38426ed243c932bf2de183f2a0d"} Jul 10 17:35:52.931137 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:35:52.931110 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dddaaba5-pbxjc" Jul 10 17:35:53.014969 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:35:53.014936 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/342379b1-29b3-470a-aade-18e6a468989c-maas-api-service-ca\") pod \"342379b1-29b3-470a-aade-18e6a468989c\" (UID: \"342379b1-29b3-470a-aade-18e6a468989c\") " Jul 10 17:35:53.015294 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:35:53.015269 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/342379b1-29b3-470a-aade-18e6a468989c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "342379b1-29b3-470a-aade-18e6a468989c" (UID: "342379b1-29b3-470a-aade-18e6a468989c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:35:53.116100 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:35:53.116006 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/342379b1-29b3-470a-aade-18e6a468989c-maas-api-service-ca\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:35:53.792477 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:35:53.792430 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dddaaba5-pbxjc" event={"ID":"342379b1-29b3-470a-aade-18e6a468989c","Type":"ContainerDied","Data":"999a52acb771e6ab200f97ca9cc21107e67a78b03fe69a878f1517f4e4c6972c"} Jul 10 17:35:53.792663 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:35:53.792498 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="999a52acb771e6ab200f97ca9cc21107e67a78b03fe69a878f1517f4e4c6972c" Jul 10 17:35:53.792663 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:35:53.792450 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dddaaba5-pbxjc" Jul 10 17:36:08.200557 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:08.200440 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js69w_e41baaf7-4171-4df1-8bdd-d0697a61fdce/ovn-acl-logging/0.log" Jul 10 17:36:08.218626 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:08.204017 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js69w_e41baaf7-4171-4df1-8bdd-d0697a61fdce/ovn-acl-logging/0.log" Jul 10 17:36:12.760176 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:12.760139 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp"] Jul 10 17:36:12.760743 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:12.760722 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="342379b1-29b3-470a-aade-18e6a468989c" containerName="revoke-keys" Jul 10 17:36:12.760810 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:12.760748 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="342379b1-29b3-470a-aade-18e6a468989c" containerName="revoke-keys" Jul 10 17:36:12.760883 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:12.760871 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="342379b1-29b3-470a-aade-18e6a468989c" containerName="revoke-keys" Jul 10 17:36:12.764098 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:12.764075 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:12.766562 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:12.766523 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1ed84dc4-openshift-default-dockercfg-9p42c\"" Jul 10 17:36:12.773276 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:12.773252 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp"] Jul 10 17:36:12.882331 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:12.882298 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa161c63-784e-471d-92b1-75d6e61c60a3-istio-envoy\") pod \"e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:12.882331 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:12.882345 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa161c63-784e-471d-92b1-75d6e61c60a3-istio-podinfo\") pod \"e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:12.882587 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:12.882435 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa161c63-784e-471d-92b1-75d6e61c60a3-credential-socket\") pod \"e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:12.882587 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:12.882485 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n9fx\" (UniqueName: \"kubernetes.io/projected/fa161c63-784e-471d-92b1-75d6e61c60a3-kube-api-access-6n9fx\") pod \"e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:12.882587 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:12.882505 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa161c63-784e-471d-92b1-75d6e61c60a3-workload-socket\") pod \"e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:12.882587 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:12.882551 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa161c63-784e-471d-92b1-75d6e61c60a3-istio-token\") pod \"e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:12.882734 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:12.882635 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa161c63-784e-471d-92b1-75d6e61c60a3-workload-certs\") pod \"e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:12.882734 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:12.882678 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa161c63-784e-471d-92b1-75d6e61c60a3-istio-data\") pod \"e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:12.882734 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:12.882698 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa161c63-784e-471d-92b1-75d6e61c60a3-istiod-ca-cert\") pod \"e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:12.983285 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:12.983247 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa161c63-784e-471d-92b1-75d6e61c60a3-workload-socket\") pod \"e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:12.983285 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:12.983291 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa161c63-784e-471d-92b1-75d6e61c60a3-istio-token\") pod \"e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:12.983577 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:12.983347 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa161c63-784e-471d-92b1-75d6e61c60a3-workload-certs\") pod \"e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:12.983577 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:12.983384 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa161c63-784e-471d-92b1-75d6e61c60a3-istio-data\") pod \"e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:12.983677 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:12.983582 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa161c63-784e-471d-92b1-75d6e61c60a3-istiod-ca-cert\") pod \"e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:12.983677 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:12.983624 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa161c63-784e-471d-92b1-75d6e61c60a3-istio-envoy\") pod \"e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:12.983677 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:12.983674 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa161c63-784e-471d-92b1-75d6e61c60a3-istio-podinfo\") pod \"e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:12.983833 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:12.983711 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa161c63-784e-471d-92b1-75d6e61c60a3-credential-socket\") pod \"e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:12.983833 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:12.983738 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n9fx\" (UniqueName: \"kubernetes.io/projected/fa161c63-784e-471d-92b1-75d6e61c60a3-kube-api-access-6n9fx\") pod \"e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:12.983833 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:12.983751 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa161c63-784e-471d-92b1-75d6e61c60a3-workload-socket\") pod \"e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:12.983833 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:12.983782 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa161c63-784e-471d-92b1-75d6e61c60a3-istio-data\") pod \"e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:12.984209 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:12.984184 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa161c63-784e-471d-92b1-75d6e61c60a3-credential-socket\") pod \"e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:12.984209 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:12.984198 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa161c63-784e-471d-92b1-75d6e61c60a3-workload-certs\") pod \"e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:12.984566 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:12.984546 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa161c63-784e-471d-92b1-75d6e61c60a3-istiod-ca-cert\") pod \"e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:12.986200 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:12.986182 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa161c63-784e-471d-92b1-75d6e61c60a3-istio-envoy\") pod \"e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:12.986259 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:12.986242 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa161c63-784e-471d-92b1-75d6e61c60a3-istio-podinfo\") pod \"e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:12.991209 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:12.991185 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa161c63-784e-471d-92b1-75d6e61c60a3-istio-token\") pod \"e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:12.991553 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:12.991523 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n9fx\" (UniqueName: \"kubernetes.io/projected/fa161c63-784e-471d-92b1-75d6e61c60a3-kube-api-access-6n9fx\") pod \"e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:13.075937 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:13.075852 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:13.223765 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:13.223738 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp"] Jul 10 17:36:13.224901 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:36:13.224872 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa161c63_784e_471d_92b1_75d6e61c60a3.slice/crio-9a29f8128af03a8e7e50747570159d763e2da71670110eef78dc4fad00808dbf WatchSource:0}: Error finding container 9a29f8128af03a8e7e50747570159d763e2da71670110eef78dc4fad00808dbf: Status 404 returned error can't find the container with id 9a29f8128af03a8e7e50747570159d763e2da71670110eef78dc4fad00808dbf Jul 10 17:36:13.227110 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:13.227080 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:36:13.227185 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:13.227163 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:36:13.227223 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:13.227191 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:36:13.899159 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:13.899119 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" event={"ID":"fa161c63-784e-471d-92b1-75d6e61c60a3","Type":"ContainerStarted","Data":"249723b461e9badd2df7af318ca5e418ba1054867a5eab64177a51c2b521ef10"} Jul 10 17:36:13.899658 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:13.899183 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" event={"ID":"fa161c63-784e-471d-92b1-75d6e61c60a3","Type":"ContainerStarted","Data":"9a29f8128af03a8e7e50747570159d763e2da71670110eef78dc4fad00808dbf"} Jul 10 17:36:13.921266 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:13.921211 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" podStartSLOduration=1.921191272 podStartE2EDuration="1.921191272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:36:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:36:13.918151632 +0000 UTC m=+1506.245849170" watchObservedRunningTime="2026-07-10 17:36:13.921191272 +0000 UTC m=+1506.248888810" Jul 10 17:36:14.076249 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:14.076210 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:14.081940 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:14.081910 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:14.906586 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:14.906557 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:14.907466 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:14.907447 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:24.966661 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:24.966572 2583 generic.go:356] "Generic (PLEG): container finished" podID="fa161c63-784e-471d-92b1-75d6e61c60a3" containerID="249723b461e9badd2df7af318ca5e418ba1054867a5eab64177a51c2b521ef10" exitCode=0 Jul 10 17:36:24.966661 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:24.966648 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" event={"ID":"fa161c63-784e-471d-92b1-75d6e61c60a3","Type":"ContainerDied","Data":"249723b461e9badd2df7af318ca5e418ba1054867a5eab64177a51c2b521ef10"} Jul 10 17:36:24.967164 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:24.967146 2583 scope.go:117] "RemoveContainer" containerID="249723b461e9badd2df7af318ca5e418ba1054867a5eab64177a51c2b521ef10" Jul 10 17:36:24.968289 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:24.968256 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:36:24.968411 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:24.968325 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:36:24.968411 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:24.968357 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:36:25.974013 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:25.973969 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" event={"ID":"fa161c63-784e-471d-92b1-75d6e61c60a3","Type":"ContainerStarted","Data":"f547896d069f52e0db69e58f6b950de0c662621f7e0d696f05f22a7c8c1e6451"} Jul 10 17:36:26.076753 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:26.076718 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:26.077909 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:26.077880 2583 patch_prober.go:28] interesting pod/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 10 17:36:26.078003 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:26.077949 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" podUID="fa161c63-784e-471d-92b1-75d6e61c60a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 10 17:36:27.076828 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:27.076792 2583 patch_prober.go:28] interesting pod/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 10 17:36:27.077207 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:27.076844 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" podUID="fa161c63-784e-471d-92b1-75d6e61c60a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 10 17:36:28.077138 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:28.077106 2583 patch_prober.go:28] interesting pod/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 10 17:36:28.077555 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:28.077163 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" podUID="fa161c63-784e-471d-92b1-75d6e61c60a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 10 17:36:29.076709 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:29.076616 2583 patch_prober.go:28] interesting pod/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 10 17:36:29.076709 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:29.076678 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" podUID="fa161c63-784e-471d-92b1-75d6e61c60a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 10 17:36:29.906905 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:29.906860 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:30.080228 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:30.080197 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:36:30.081341 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:36:30.081320 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:37:45.146795 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:45.146751 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp"] Jul 10 17:37:45.147437 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:45.147022 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" podUID="fa161c63-784e-471d-92b1-75d6e61c60a3" containerName="istio-proxy" containerID="cri-o://f547896d069f52e0db69e58f6b950de0c662621f7e0d696f05f22a7c8c1e6451" gracePeriod=30 Jul 10 17:37:50.416966 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.416934 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:37:50.434506 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.434473 2583 generic.go:356] "Generic (PLEG): container finished" podID="fa161c63-784e-471d-92b1-75d6e61c60a3" containerID="f547896d069f52e0db69e58f6b950de0c662621f7e0d696f05f22a7c8c1e6451" exitCode=0 Jul 10 17:37:50.434676 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.434517 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" Jul 10 17:37:50.434676 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.434528 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" event={"ID":"fa161c63-784e-471d-92b1-75d6e61c60a3","Type":"ContainerDied","Data":"f547896d069f52e0db69e58f6b950de0c662621f7e0d696f05f22a7c8c1e6451"} Jul 10 17:37:50.434676 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.434586 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp" event={"ID":"fa161c63-784e-471d-92b1-75d6e61c60a3","Type":"ContainerDied","Data":"9a29f8128af03a8e7e50747570159d763e2da71670110eef78dc4fad00808dbf"} Jul 10 17:37:50.434676 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.434616 2583 scope.go:117] "RemoveContainer" containerID="f547896d069f52e0db69e58f6b950de0c662621f7e0d696f05f22a7c8c1e6451" Jul 10 17:37:50.451382 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.451355 2583 scope.go:117] "RemoveContainer" containerID="249723b461e9badd2df7af318ca5e418ba1054867a5eab64177a51c2b521ef10" Jul 10 17:37:50.460567 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.460538 2583 scope.go:117] "RemoveContainer" containerID="f547896d069f52e0db69e58f6b950de0c662621f7e0d696f05f22a7c8c1e6451" Jul 10 17:37:50.460901 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:37:50.460868 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f547896d069f52e0db69e58f6b950de0c662621f7e0d696f05f22a7c8c1e6451\": container with ID starting with f547896d069f52e0db69e58f6b950de0c662621f7e0d696f05f22a7c8c1e6451 not found: ID does not exist" containerID="f547896d069f52e0db69e58f6b950de0c662621f7e0d696f05f22a7c8c1e6451" Jul 10 17:37:50.460999 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.460906 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f547896d069f52e0db69e58f6b950de0c662621f7e0d696f05f22a7c8c1e6451"} err="failed to get container status \"f547896d069f52e0db69e58f6b950de0c662621f7e0d696f05f22a7c8c1e6451\": rpc error: code = NotFound desc = could not find container \"f547896d069f52e0db69e58f6b950de0c662621f7e0d696f05f22a7c8c1e6451\": container with ID starting with f547896d069f52e0db69e58f6b950de0c662621f7e0d696f05f22a7c8c1e6451 not found: ID does not exist" Jul 10 17:37:50.460999 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.460934 2583 scope.go:117] "RemoveContainer" containerID="249723b461e9badd2df7af318ca5e418ba1054867a5eab64177a51c2b521ef10" Jul 10 17:37:50.461234 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:37:50.461213 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"249723b461e9badd2df7af318ca5e418ba1054867a5eab64177a51c2b521ef10\": container with ID starting with 249723b461e9badd2df7af318ca5e418ba1054867a5eab64177a51c2b521ef10 not found: ID does not exist" containerID="249723b461e9badd2df7af318ca5e418ba1054867a5eab64177a51c2b521ef10" Jul 10 17:37:50.461299 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.461243 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"249723b461e9badd2df7af318ca5e418ba1054867a5eab64177a51c2b521ef10"} err="failed to get container status \"249723b461e9badd2df7af318ca5e418ba1054867a5eab64177a51c2b521ef10\": rpc error: code = NotFound desc = could not find container \"249723b461e9badd2df7af318ca5e418ba1054867a5eab64177a51c2b521ef10\": container with ID starting with 249723b461e9badd2df7af318ca5e418ba1054867a5eab64177a51c2b521ef10 not found: ID does not exist" Jul 10 17:37:50.552553 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.552525 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa161c63-784e-471d-92b1-75d6e61c60a3-istio-envoy\") pod \"fa161c63-784e-471d-92b1-75d6e61c60a3\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " Jul 10 17:37:50.552734 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.552561 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa161c63-784e-471d-92b1-75d6e61c60a3-workload-socket\") pod \"fa161c63-784e-471d-92b1-75d6e61c60a3\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " Jul 10 17:37:50.552734 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.552583 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa161c63-784e-471d-92b1-75d6e61c60a3-credential-socket\") pod \"fa161c63-784e-471d-92b1-75d6e61c60a3\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " Jul 10 17:37:50.552734 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.552634 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa161c63-784e-471d-92b1-75d6e61c60a3-istio-podinfo\") pod \"fa161c63-784e-471d-92b1-75d6e61c60a3\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " Jul 10 17:37:50.552734 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.552655 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa161c63-784e-471d-92b1-75d6e61c60a3-istiod-ca-cert\") pod \"fa161c63-784e-471d-92b1-75d6e61c60a3\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " Jul 10 17:37:50.552734 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.552706 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa161c63-784e-471d-92b1-75d6e61c60a3-istio-token\") pod \"fa161c63-784e-471d-92b1-75d6e61c60a3\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " Jul 10 17:37:50.552734 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.552728 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa161c63-784e-471d-92b1-75d6e61c60a3-istio-data\") pod \"fa161c63-784e-471d-92b1-75d6e61c60a3\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " Jul 10 17:37:50.553049 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.552754 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa161c63-784e-471d-92b1-75d6e61c60a3-workload-certs\") pod \"fa161c63-784e-471d-92b1-75d6e61c60a3\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " Jul 10 17:37:50.553049 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.552791 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6n9fx\" (UniqueName: \"kubernetes.io/projected/fa161c63-784e-471d-92b1-75d6e61c60a3-kube-api-access-6n9fx\") pod \"fa161c63-784e-471d-92b1-75d6e61c60a3\" (UID: \"fa161c63-784e-471d-92b1-75d6e61c60a3\") " Jul 10 17:37:50.553049 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.552868 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa161c63-784e-471d-92b1-75d6e61c60a3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fa161c63-784e-471d-92b1-75d6e61c60a3" (UID: "fa161c63-784e-471d-92b1-75d6e61c60a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:37:50.553049 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.553002 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa161c63-784e-471d-92b1-75d6e61c60a3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fa161c63-784e-471d-92b1-75d6e61c60a3" (UID: "fa161c63-784e-471d-92b1-75d6e61c60a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:37:50.553274 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.553119 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa161c63-784e-471d-92b1-75d6e61c60a3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fa161c63-784e-471d-92b1-75d6e61c60a3" (UID: "fa161c63-784e-471d-92b1-75d6e61c60a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:37:50.553274 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.553127 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa161c63-784e-471d-92b1-75d6e61c60a3-workload-socket\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:37:50.553381 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.553278 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa161c63-784e-471d-92b1-75d6e61c60a3-credential-socket\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:37:50.553381 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.553274 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa161c63-784e-471d-92b1-75d6e61c60a3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fa161c63-784e-471d-92b1-75d6e61c60a3" (UID: "fa161c63-784e-471d-92b1-75d6e61c60a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:37:50.554059 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.554030 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa161c63-784e-471d-92b1-75d6e61c60a3-istio-data" (OuterVolumeSpecName: "istio-data") pod "fa161c63-784e-471d-92b1-75d6e61c60a3" (UID: "fa161c63-784e-471d-92b1-75d6e61c60a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:37:50.555085 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.555057 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa161c63-784e-471d-92b1-75d6e61c60a3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fa161c63-784e-471d-92b1-75d6e61c60a3" (UID: "fa161c63-784e-471d-92b1-75d6e61c60a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:37:50.555085 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.555069 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa161c63-784e-471d-92b1-75d6e61c60a3-istio-token" (OuterVolumeSpecName: "istio-token") pod "fa161c63-784e-471d-92b1-75d6e61c60a3" (UID: "fa161c63-784e-471d-92b1-75d6e61c60a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:37:50.555214 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.555096 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa161c63-784e-471d-92b1-75d6e61c60a3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fa161c63-784e-471d-92b1-75d6e61c60a3" (UID: "fa161c63-784e-471d-92b1-75d6e61c60a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 17:37:50.555214 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.555161 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa161c63-784e-471d-92b1-75d6e61c60a3-kube-api-access-6n9fx" (OuterVolumeSpecName: "kube-api-access-6n9fx") pod "fa161c63-784e-471d-92b1-75d6e61c60a3" (UID: "fa161c63-784e-471d-92b1-75d6e61c60a3"). InnerVolumeSpecName "kube-api-access-6n9fx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:37:50.653884 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.653840 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa161c63-784e-471d-92b1-75d6e61c60a3-istio-token\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:37:50.653884 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.653876 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa161c63-784e-471d-92b1-75d6e61c60a3-istio-data\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:37:50.653884 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.653890 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa161c63-784e-471d-92b1-75d6e61c60a3-workload-certs\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:37:50.654127 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.653901 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n9fx\" (UniqueName: \"kubernetes.io/projected/fa161c63-784e-471d-92b1-75d6e61c60a3-kube-api-access-6n9fx\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:37:50.654127 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.653915 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa161c63-784e-471d-92b1-75d6e61c60a3-istio-envoy\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:37:50.654127 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.653931 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa161c63-784e-471d-92b1-75d6e61c60a3-istio-podinfo\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:37:50.654127 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.653940 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa161c63-784e-471d-92b1-75d6e61c60a3-istiod-ca-cert\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:37:50.756902 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.756866 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp"] Jul 10 17:37:50.762747 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.762725 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1ed84dc4-openshift-default-574d749cb5-f26xp"] Jul 10 17:37:50.979094 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.979063 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-cggpf"] Jul 10 17:37:50.979482 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.979467 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa161c63-784e-471d-92b1-75d6e61c60a3" containerName="istio-proxy" Jul 10 17:37:50.979482 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.979482 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa161c63-784e-471d-92b1-75d6e61c60a3" containerName="istio-proxy" Jul 10 17:37:50.979643 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.979558 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa161c63-784e-471d-92b1-75d6e61c60a3" containerName="istio-proxy" Jul 10 17:37:50.979643 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.979568 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa161c63-784e-471d-92b1-75d6e61c60a3" containerName="istio-proxy" Jul 10 17:37:50.982730 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.982710 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-cggpf" Jul 10 17:37:50.984969 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.984949 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m7h25\"" Jul 10 17:37:50.989324 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:50.989260 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-cggpf"] Jul 10 17:37:51.156468 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:51.156435 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3b09de3-b336-4350-aeb0-beb75379babc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-5d9215e4-cggpf\" (UID: \"e3b09de3-b336-4350-aeb0-beb75379babc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-cggpf" Jul 10 17:37:51.257213 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:51.257135 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3b09de3-b336-4350-aeb0-beb75379babc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-5d9215e4-cggpf\" (UID: \"e3b09de3-b336-4350-aeb0-beb75379babc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-cggpf" Jul 10 17:37:51.257790 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:51.257772 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3b09de3-b336-4350-aeb0-beb75379babc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-5d9215e4-cggpf\" (UID: \"e3b09de3-b336-4350-aeb0-beb75379babc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-cggpf" Jul 10 17:37:51.296135 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:51.296103 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-cggpf" Jul 10 17:37:51.440878 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:51.440779 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-cggpf"] Jul 10 17:37:52.206981 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:52.206940 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa161c63-784e-471d-92b1-75d6e61c60a3" path="/var/lib/kubelet/pods/fa161c63-784e-471d-92b1-75d6e61c60a3/volumes" Jul 10 17:37:52.452361 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:52.452326 2583 generic.go:356] "Generic (PLEG): container finished" podID="e3b09de3-b336-4350-aeb0-beb75379babc" containerID="36cac126dddb7af2673dfc9b3283271061f38a371b5f0d4ba33f201b08089699" exitCode=7 Jul 10 17:37:52.452779 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:52.452425 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-cggpf" event={"ID":"e3b09de3-b336-4350-aeb0-beb75379babc","Type":"ContainerDied","Data":"36cac126dddb7af2673dfc9b3283271061f38a371b5f0d4ba33f201b08089699"} Jul 10 17:37:52.452779 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:52.452476 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-cggpf" event={"ID":"e3b09de3-b336-4350-aeb0-beb75379babc","Type":"ContainerStarted","Data":"9d80828e718c446f787ce3943795645ee97efa8e807540e1925ec2966b9448da"} Jul 10 17:37:52.452779 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:52.452607 2583 scope.go:117] "RemoveContainer" containerID="36cac126dddb7af2673dfc9b3283271061f38a371b5f0d4ba33f201b08089699" Jul 10 17:37:53.460143 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:53.460103 2583 generic.go:356] "Generic (PLEG): container finished" podID="e3b09de3-b336-4350-aeb0-beb75379babc" containerID="86240d8557af789b48cff3fd3f3ad4745971ae604750ea98ef50df9af77f06f3" exitCode=7 Jul 10 17:37:53.460567 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:53.460182 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-cggpf" event={"ID":"e3b09de3-b336-4350-aeb0-beb75379babc","Type":"ContainerDied","Data":"86240d8557af789b48cff3fd3f3ad4745971ae604750ea98ef50df9af77f06f3"} Jul 10 17:37:53.460567 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:53.460278 2583 scope.go:117] "RemoveContainer" containerID="36cac126dddb7af2673dfc9b3283271061f38a371b5f0d4ba33f201b08089699" Jul 10 17:37:53.460567 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:53.460506 2583 scope.go:117] "RemoveContainer" containerID="86240d8557af789b48cff3fd3f3ad4745971ae604750ea98ef50df9af77f06f3" Jul 10 17:37:53.460733 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:37:53.460715 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-5d9215e4-cggpf_odh-ai-gateway-infra(e3b09de3-b336-4350-aeb0-beb75379babc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-cggpf" podUID="e3b09de3-b336-4350-aeb0-beb75379babc" Jul 10 17:37:54.467897 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:37:54.467870 2583 scope.go:117] "RemoveContainer" containerID="86240d8557af789b48cff3fd3f3ad4745971ae604750ea98ef50df9af77f06f3" Jul 10 17:37:54.468356 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:37:54.468081 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-5d9215e4-cggpf_odh-ai-gateway-infra(e3b09de3-b336-4350-aeb0-beb75379babc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-cggpf" podUID="e3b09de3-b336-4350-aeb0-beb75379babc" Jul 10 17:38:09.200778 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:09.200730 2583 scope.go:117] "RemoveContainer" containerID="86240d8557af789b48cff3fd3f3ad4745971ae604750ea98ef50df9af77f06f3" Jul 10 17:38:09.551257 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:09.551226 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-cggpf" event={"ID":"e3b09de3-b336-4350-aeb0-beb75379babc","Type":"ContainerStarted","Data":"b336aeac3c13908bdf4b35f6e1d50a83a865c7c4df64562775ab61823387e485"} Jul 10 17:38:09.568556 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:09.568505 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-cggpf" podStartSLOduration=19.568487268 podStartE2EDuration="19.568487268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:37:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:38:09.56649597 +0000 UTC m=+1621.894193512" watchObservedRunningTime="2026-07-10 17:38:09.568487268 +0000 UTC m=+1621.896184807" Jul 10 17:38:10.558956 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:10.558918 2583 generic.go:356] "Generic (PLEG): container finished" podID="e3b09de3-b336-4350-aeb0-beb75379babc" containerID="b336aeac3c13908bdf4b35f6e1d50a83a865c7c4df64562775ab61823387e485" exitCode=0 Jul 10 17:38:10.559428 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:10.558993 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-cggpf" event={"ID":"e3b09de3-b336-4350-aeb0-beb75379babc","Type":"ContainerDied","Data":"b336aeac3c13908bdf4b35f6e1d50a83a865c7c4df64562775ab61823387e485"} Jul 10 17:38:10.559428 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:10.559102 2583 scope.go:117] "RemoveContainer" containerID="86240d8557af789b48cff3fd3f3ad4745971ae604750ea98ef50df9af77f06f3" Jul 10 17:38:10.576745 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:10.576721 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-cggpf"] Jul 10 17:38:11.705247 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:11.705216 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-cggpf" Jul 10 17:38:11.746146 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:11.746114 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3b09de3-b336-4350-aeb0-beb75379babc-maas-api-service-ca\") pod \"e3b09de3-b336-4350-aeb0-beb75379babc\" (UID: \"e3b09de3-b336-4350-aeb0-beb75379babc\") " Jul 10 17:38:11.746476 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:11.746451 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b09de3-b336-4350-aeb0-beb75379babc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e3b09de3-b336-4350-aeb0-beb75379babc" (UID: "e3b09de3-b336-4350-aeb0-beb75379babc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:38:11.846985 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:11.846887 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3b09de3-b336-4350-aeb0-beb75379babc-maas-api-service-ca\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:38:12.572139 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:12.572058 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-cggpf" Jul 10 17:38:12.572285 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:12.572051 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-cggpf" event={"ID":"e3b09de3-b336-4350-aeb0-beb75379babc","Type":"ContainerDied","Data":"9d80828e718c446f787ce3943795645ee97efa8e807540e1925ec2966b9448da"} Jul 10 17:38:12.572285 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:12.572195 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d80828e718c446f787ce3943795645ee97efa8e807540e1925ec2966b9448da" Jul 10 17:38:12.601083 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:12.601047 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-cggpf"] Jul 10 17:38:12.604464 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:12.604441 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-cggpf"] Jul 10 17:38:14.205481 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:14.205441 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3b09de3-b336-4350-aeb0-beb75379babc" path="/var/lib/kubelet/pods/e3b09de3-b336-4350-aeb0-beb75379babc/volumes" Jul 10 17:38:51.035347 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:51.035302 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-5xjrw"] Jul 10 17:38:51.035910 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:51.035891 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3b09de3-b336-4350-aeb0-beb75379babc" containerName="revoke-keys" Jul 10 17:38:51.035994 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:51.035916 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3b09de3-b336-4350-aeb0-beb75379babc" containerName="revoke-keys" Jul 10 17:38:51.035994 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:51.035929 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3b09de3-b336-4350-aeb0-beb75379babc" containerName="revoke-keys" Jul 10 17:38:51.035994 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:51.035936 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3b09de3-b336-4350-aeb0-beb75379babc" containerName="revoke-keys" Jul 10 17:38:51.035994 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:51.035982 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa161c63-784e-471d-92b1-75d6e61c60a3" containerName="istio-proxy" Jul 10 17:38:51.035994 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:51.035991 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa161c63-784e-471d-92b1-75d6e61c60a3" containerName="istio-proxy" Jul 10 17:38:51.036236 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:51.036090 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3b09de3-b336-4350-aeb0-beb75379babc" containerName="revoke-keys" Jul 10 17:38:51.036236 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:51.036101 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3b09de3-b336-4350-aeb0-beb75379babc" containerName="revoke-keys" Jul 10 17:38:51.039273 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:51.039253 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-5xjrw" Jul 10 17:38:51.041927 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:51.041905 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m7h25\"" Jul 10 17:38:51.046848 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:51.046755 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-5xjrw"] Jul 10 17:38:51.081986 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:51.081940 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d77a1607-4417-46b1-b6e0-0267da87cb51-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-5d9215e4-5xjrw\" (UID: \"d77a1607-4417-46b1-b6e0-0267da87cb51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-5xjrw" Jul 10 17:38:51.182512 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:51.182479 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d77a1607-4417-46b1-b6e0-0267da87cb51-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-5d9215e4-5xjrw\" (UID: \"d77a1607-4417-46b1-b6e0-0267da87cb51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-5xjrw" Jul 10 17:38:51.183115 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:51.183088 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d77a1607-4417-46b1-b6e0-0267da87cb51-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-5d9215e4-5xjrw\" (UID: \"d77a1607-4417-46b1-b6e0-0267da87cb51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-5xjrw" Jul 10 17:38:51.351928 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:51.351831 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-5xjrw" Jul 10 17:38:51.494869 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:51.494844 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-5xjrw"] Jul 10 17:38:51.496508 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:38:51.496481 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd77a1607_4417_46b1_b6e0_0267da87cb51.slice/crio-2b22f8922f08c0c6a258554da692566b07012f5463aeb5245625e73cb08e2437 WatchSource:0}: Error finding container 2b22f8922f08c0c6a258554da692566b07012f5463aeb5245625e73cb08e2437: Status 404 returned error can't find the container with id 2b22f8922f08c0c6a258554da692566b07012f5463aeb5245625e73cb08e2437 Jul 10 17:38:51.779156 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:51.779123 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-5xjrw" event={"ID":"d77a1607-4417-46b1-b6e0-0267da87cb51","Type":"ContainerStarted","Data":"2b22f8922f08c0c6a258554da692566b07012f5463aeb5245625e73cb08e2437"} Jul 10 17:38:52.786466 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:52.786425 2583 generic.go:356] "Generic (PLEG): container finished" podID="d77a1607-4417-46b1-b6e0-0267da87cb51" containerID="807910fcdeb72238c6608ac649516037046f1c00c297875858a67af7c51269b3" exitCode=0 Jul 10 17:38:52.786860 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:52.786505 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-5xjrw" event={"ID":"d77a1607-4417-46b1-b6e0-0267da87cb51","Type":"ContainerDied","Data":"807910fcdeb72238c6608ac649516037046f1c00c297875858a67af7c51269b3"} Jul 10 17:38:53.935023 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:53.934994 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-5xjrw" Jul 10 17:38:54.001666 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:54.001625 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d77a1607-4417-46b1-b6e0-0267da87cb51-maas-api-service-ca\") pod \"d77a1607-4417-46b1-b6e0-0267da87cb51\" (UID: \"d77a1607-4417-46b1-b6e0-0267da87cb51\") " Jul 10 17:38:54.001995 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:54.001970 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d77a1607-4417-46b1-b6e0-0267da87cb51-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d77a1607-4417-46b1-b6e0-0267da87cb51" (UID: "d77a1607-4417-46b1-b6e0-0267da87cb51"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:38:54.102817 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:54.102727 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d77a1607-4417-46b1-b6e0-0267da87cb51-maas-api-service-ca\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:38:54.798420 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:54.798369 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-5xjrw" Jul 10 17:38:54.798420 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:54.798382 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5d9215e4-5xjrw" event={"ID":"d77a1607-4417-46b1-b6e0-0267da87cb51","Type":"ContainerDied","Data":"2b22f8922f08c0c6a258554da692566b07012f5463aeb5245625e73cb08e2437"} Jul 10 17:38:54.798647 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:38:54.798463 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b22f8922f08c0c6a258554da692566b07012f5463aeb5245625e73cb08e2437" Jul 10 17:41:05.287305 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:41:05.287270 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1c07f63-7nbs4"] Jul 10 17:41:05.287919 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:41:05.287899 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d77a1607-4417-46b1-b6e0-0267da87cb51" containerName="revoke-keys" Jul 10 17:41:05.288007 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:41:05.287923 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d77a1607-4417-46b1-b6e0-0267da87cb51" containerName="revoke-keys" Jul 10 17:41:05.288007 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:41:05.287945 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3b09de3-b336-4350-aeb0-beb75379babc" containerName="revoke-keys" Jul 10 17:41:05.288007 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:41:05.287952 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3b09de3-b336-4350-aeb0-beb75379babc" containerName="revoke-keys" Jul 10 17:41:05.288159 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:41:05.288038 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3b09de3-b336-4350-aeb0-beb75379babc" containerName="revoke-keys" Jul 10 17:41:05.288159 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:41:05.288054 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d77a1607-4417-46b1-b6e0-0267da87cb51" containerName="revoke-keys" Jul 10 17:41:05.291033 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:41:05.291012 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1c07f63-7nbs4" Jul 10 17:41:05.293520 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:41:05.293496 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m7h25\"" Jul 10 17:41:05.299962 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:41:05.299886 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1c07f63-7nbs4"] Jul 10 17:41:05.455461 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:41:05.455430 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da2a94db-de75-4bab-a128-eb6b5ce0505e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c1c07f63-7nbs4\" (UID: \"da2a94db-de75-4bab-a128-eb6b5ce0505e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1c07f63-7nbs4" Jul 10 17:41:05.556745 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:41:05.556658 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da2a94db-de75-4bab-a128-eb6b5ce0505e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c1c07f63-7nbs4\" (UID: \"da2a94db-de75-4bab-a128-eb6b5ce0505e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1c07f63-7nbs4" Jul 10 17:41:05.557322 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:41:05.557304 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da2a94db-de75-4bab-a128-eb6b5ce0505e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c1c07f63-7nbs4\" (UID: \"da2a94db-de75-4bab-a128-eb6b5ce0505e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1c07f63-7nbs4" Jul 10 17:41:05.604686 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:41:05.604654 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1c07f63-7nbs4" Jul 10 17:41:05.745121 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:41:05.745092 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1c07f63-7nbs4"] Jul 10 17:41:06.493615 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:41:06.493578 2583 generic.go:356] "Generic (PLEG): container finished" podID="da2a94db-de75-4bab-a128-eb6b5ce0505e" containerID="87f14139288b3a4ea649f603b091fa115f3f9ae0a064e5f57bd9e513a146b68b" exitCode=0 Jul 10 17:41:06.494067 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:41:06.493678 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1c07f63-7nbs4" event={"ID":"da2a94db-de75-4bab-a128-eb6b5ce0505e","Type":"ContainerDied","Data":"87f14139288b3a4ea649f603b091fa115f3f9ae0a064e5f57bd9e513a146b68b"} Jul 10 17:41:06.494067 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:41:06.493761 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1c07f63-7nbs4" event={"ID":"da2a94db-de75-4bab-a128-eb6b5ce0505e","Type":"ContainerStarted","Data":"b51c5dd252a3fea9a68c26c60d8e7107f1c71947179aedd2dd586d1aca034f01"} Jul 10 17:41:07.653877 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:41:07.653852 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1c07f63-7nbs4" Jul 10 17:41:07.777763 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:41:07.777670 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da2a94db-de75-4bab-a128-eb6b5ce0505e-maas-api-service-ca\") pod \"da2a94db-de75-4bab-a128-eb6b5ce0505e\" (UID: \"da2a94db-de75-4bab-a128-eb6b5ce0505e\") " Jul 10 17:41:07.777999 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:41:07.777978 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da2a94db-de75-4bab-a128-eb6b5ce0505e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "da2a94db-de75-4bab-a128-eb6b5ce0505e" (UID: "da2a94db-de75-4bab-a128-eb6b5ce0505e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:41:07.879123 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:41:07.879084 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da2a94db-de75-4bab-a128-eb6b5ce0505e-maas-api-service-ca\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:41:08.236461 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:41:08.236424 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js69w_e41baaf7-4171-4df1-8bdd-d0697a61fdce/ovn-acl-logging/0.log" Jul 10 17:41:08.242027 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:41:08.242005 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js69w_e41baaf7-4171-4df1-8bdd-d0697a61fdce/ovn-acl-logging/0.log" Jul 10 17:41:08.512905 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:41:08.512822 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1c07f63-7nbs4" Jul 10 17:41:08.512905 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:41:08.512857 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1c07f63-7nbs4" event={"ID":"da2a94db-de75-4bab-a128-eb6b5ce0505e","Type":"ContainerDied","Data":"b51c5dd252a3fea9a68c26c60d8e7107f1c71947179aedd2dd586d1aca034f01"} Jul 10 17:41:08.513083 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:41:08.512907 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b51c5dd252a3fea9a68c26c60d8e7107f1c71947179aedd2dd586d1aca034f01" Jul 10 17:42:27.823345 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:42:27.823309 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-039c4f53-brp62"] Jul 10 17:42:27.824236 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:42:27.823912 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da2a94db-de75-4bab-a128-eb6b5ce0505e" containerName="revoke-keys" Jul 10 17:42:27.824236 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:42:27.823933 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="da2a94db-de75-4bab-a128-eb6b5ce0505e" containerName="revoke-keys" Jul 10 17:42:27.824236 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:42:27.824012 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="da2a94db-de75-4bab-a128-eb6b5ce0505e" containerName="revoke-keys" Jul 10 17:42:27.827266 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:42:27.827245 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-039c4f53-brp62" Jul 10 17:42:27.829766 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:42:27.829742 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m7h25\"" Jul 10 17:42:27.833147 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:42:27.833122 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-039c4f53-brp62"] Jul 10 17:42:27.855668 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:42:27.855636 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c092fe77-2742-4f35-b8dc-41e5a5fb9819-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-039c4f53-brp62\" (UID: \"c092fe77-2742-4f35-b8dc-41e5a5fb9819\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-039c4f53-brp62" Jul 10 17:42:27.956795 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:42:27.956763 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c092fe77-2742-4f35-b8dc-41e5a5fb9819-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-039c4f53-brp62\" (UID: \"c092fe77-2742-4f35-b8dc-41e5a5fb9819\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-039c4f53-brp62" Jul 10 17:42:27.957474 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:42:27.957454 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c092fe77-2742-4f35-b8dc-41e5a5fb9819-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-039c4f53-brp62\" (UID: \"c092fe77-2742-4f35-b8dc-41e5a5fb9819\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-039c4f53-brp62" Jul 10 17:42:28.139839 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:42:28.139749 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-039c4f53-brp62" Jul 10 17:42:28.281625 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:42:28.281592 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-039c4f53-brp62"] Jul 10 17:42:28.283205 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:42:28.283172 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc092fe77_2742_4f35_b8dc_41e5a5fb9819.slice/crio-b0be13492558359dc85a1e1da6ba5ad92143dfb8f01d7a826b23578d6f089494 WatchSource:0}: Error finding container b0be13492558359dc85a1e1da6ba5ad92143dfb8f01d7a826b23578d6f089494: Status 404 returned error can't find the container with id b0be13492558359dc85a1e1da6ba5ad92143dfb8f01d7a826b23578d6f089494 Jul 10 17:42:28.946616 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:42:28.946526 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-039c4f53-brp62" event={"ID":"c092fe77-2742-4f35-b8dc-41e5a5fb9819","Type":"ContainerStarted","Data":"0e0684b158567f63d6322892dc24794a3c9f2beb5a2b98e4a8fba9764b3d2a81"} Jul 10 17:42:28.946616 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:42:28.946588 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-039c4f53-brp62" event={"ID":"c092fe77-2742-4f35-b8dc-41e5a5fb9819","Type":"ContainerStarted","Data":"b0be13492558359dc85a1e1da6ba5ad92143dfb8f01d7a826b23578d6f089494"} Jul 10 17:42:28.962706 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:42:28.962661 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-039c4f53-brp62" podStartSLOduration=1.962640828 podStartE2EDuration="1.962640828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:42:28.961371162 +0000 UTC m=+1881.289068701" watchObservedRunningTime="2026-07-10 17:42:28.962640828 +0000 UTC m=+1881.290338367" Jul 10 17:42:29.953194 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:42:29.953160 2583 generic.go:356] "Generic (PLEG): container finished" podID="c092fe77-2742-4f35-b8dc-41e5a5fb9819" containerID="0e0684b158567f63d6322892dc24794a3c9f2beb5a2b98e4a8fba9764b3d2a81" exitCode=0 Jul 10 17:42:29.953597 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:42:29.953202 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-039c4f53-brp62" event={"ID":"c092fe77-2742-4f35-b8dc-41e5a5fb9819","Type":"ContainerDied","Data":"0e0684b158567f63d6322892dc24794a3c9f2beb5a2b98e4a8fba9764b3d2a81"} Jul 10 17:42:31.101169 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:42:31.101125 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-039c4f53-brp62" Jul 10 17:42:31.180150 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:42:31.180117 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c092fe77-2742-4f35-b8dc-41e5a5fb9819-maas-api-service-ca\") pod \"c092fe77-2742-4f35-b8dc-41e5a5fb9819\" (UID: \"c092fe77-2742-4f35-b8dc-41e5a5fb9819\") " Jul 10 17:42:31.180560 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:42:31.180528 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c092fe77-2742-4f35-b8dc-41e5a5fb9819-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c092fe77-2742-4f35-b8dc-41e5a5fb9819" (UID: "c092fe77-2742-4f35-b8dc-41e5a5fb9819"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:42:31.280950 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:42:31.280849 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c092fe77-2742-4f35-b8dc-41e5a5fb9819-maas-api-service-ca\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:42:31.966319 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:42:31.966280 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-039c4f53-brp62" event={"ID":"c092fe77-2742-4f35-b8dc-41e5a5fb9819","Type":"ContainerDied","Data":"b0be13492558359dc85a1e1da6ba5ad92143dfb8f01d7a826b23578d6f089494"} Jul 10 17:42:31.966319 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:42:31.966317 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-039c4f53-brp62" Jul 10 17:42:31.966557 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:42:31.966337 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0be13492558359dc85a1e1da6ba5ad92143dfb8f01d7a826b23578d6f089494" Jul 10 17:44:38.790320 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:38.790283 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq"] Jul 10 17:44:38.791208 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:38.791171 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c092fe77-2742-4f35-b8dc-41e5a5fb9819" containerName="revoke-keys" Jul 10 17:44:38.791326 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:38.791216 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c092fe77-2742-4f35-b8dc-41e5a5fb9819" containerName="revoke-keys" Jul 10 17:44:38.791495 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:38.791478 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c092fe77-2742-4f35-b8dc-41e5a5fb9819" containerName="revoke-keys" Jul 10 17:44:38.794593 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:38.794572 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:44:38.797173 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:38.797151 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-050e5c-openshift-default-dockercfg-js4r8\"" Jul 10 17:44:38.805046 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:38.805021 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq"] Jul 10 17:44:38.864732 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:38.864693 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6292244c-8dd3-4cbd-abfa-f10232c66d22-workload-socket\") pod \"e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:44:38.864732 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:38.864731 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6292244c-8dd3-4cbd-abfa-f10232c66d22-workload-certs\") pod \"e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:44:38.864992 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:38.864752 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6292244c-8dd3-4cbd-abfa-f10232c66d22-istio-data\") pod \"e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:44:38.864992 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:38.864829 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6292244c-8dd3-4cbd-abfa-f10232c66d22-istio-token\") pod \"e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:44:38.864992 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:38.864859 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kq46\" (UniqueName: \"kubernetes.io/projected/6292244c-8dd3-4cbd-abfa-f10232c66d22-kube-api-access-4kq46\") pod \"e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:44:38.864992 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:38.864892 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6292244c-8dd3-4cbd-abfa-f10232c66d22-istio-podinfo\") pod \"e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:44:38.864992 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:38.864957 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6292244c-8dd3-4cbd-abfa-f10232c66d22-istio-envoy\") pod \"e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:44:38.865267 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:38.865016 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6292244c-8dd3-4cbd-abfa-f10232c66d22-credential-socket\") pod \"e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:44:38.865267 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:38.865035 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6292244c-8dd3-4cbd-abfa-f10232c66d22-istiod-ca-cert\") pod \"e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:44:38.965723 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:38.965683 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6292244c-8dd3-4cbd-abfa-f10232c66d22-istio-data\") pod \"e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:44:38.965888 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:38.965732 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6292244c-8dd3-4cbd-abfa-f10232c66d22-istio-token\") pod \"e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:44:38.965888 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:38.965752 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kq46\" (UniqueName: \"kubernetes.io/projected/6292244c-8dd3-4cbd-abfa-f10232c66d22-kube-api-access-4kq46\") pod \"e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:44:38.965888 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:38.965778 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6292244c-8dd3-4cbd-abfa-f10232c66d22-istio-podinfo\") pod \"e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:44:38.965888 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:38.965828 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6292244c-8dd3-4cbd-abfa-f10232c66d22-istio-envoy\") pod \"e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:44:38.966159 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:38.965902 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6292244c-8dd3-4cbd-abfa-f10232c66d22-credential-socket\") pod \"e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:44:38.966159 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:38.965925 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6292244c-8dd3-4cbd-abfa-f10232c66d22-istiod-ca-cert\") pod \"e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:44:38.966159 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:38.965951 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6292244c-8dd3-4cbd-abfa-f10232c66d22-workload-socket\") pod \"e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:44:38.966159 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:38.965978 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6292244c-8dd3-4cbd-abfa-f10232c66d22-workload-certs\") pod \"e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:44:38.966159 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:38.966102 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6292244c-8dd3-4cbd-abfa-f10232c66d22-istio-data\") pod \"e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:44:38.966437 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:38.966284 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6292244c-8dd3-4cbd-abfa-f10232c66d22-credential-socket\") pod \"e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:44:38.966437 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:38.966409 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6292244c-8dd3-4cbd-abfa-f10232c66d22-workload-certs\") pod \"e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:44:38.966533 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:38.966479 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6292244c-8dd3-4cbd-abfa-f10232c66d22-workload-socket\") pod \"e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:44:38.966655 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:38.966638 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6292244c-8dd3-4cbd-abfa-f10232c66d22-istiod-ca-cert\") pod \"e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:44:38.968330 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:38.968308 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6292244c-8dd3-4cbd-abfa-f10232c66d22-istio-envoy\") pod \"e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:44:38.968605 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:38.968589 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6292244c-8dd3-4cbd-abfa-f10232c66d22-istio-podinfo\") pod \"e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:44:38.973652 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:38.973626 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kq46\" (UniqueName: \"kubernetes.io/projected/6292244c-8dd3-4cbd-abfa-f10232c66d22-kube-api-access-4kq46\") pod \"e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:44:38.974229 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:38.974208 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6292244c-8dd3-4cbd-abfa-f10232c66d22-istio-token\") pod \"e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:44:39.106565 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:39.106457 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:44:39.255222 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:39.255192 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq"] Jul 10 17:44:39.257237 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:44:39.257207 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6292244c_8dd3_4cbd_abfa_f10232c66d22.slice/crio-2aa05e6df447a334d61c5b7459c60328caacf4284ca6690276edd8976b25c446 WatchSource:0}: Error finding container 2aa05e6df447a334d61c5b7459c60328caacf4284ca6690276edd8976b25c446: Status 404 returned error can't find the container with id 2aa05e6df447a334d61c5b7459c60328caacf4284ca6690276edd8976b25c446 Jul 10 17:44:39.259830 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:39.259799 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:44:39.259919 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:39.259872 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:44:39.259962 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:39.259920 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:44:39.643027 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:39.642994 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" event={"ID":"6292244c-8dd3-4cbd-abfa-f10232c66d22","Type":"ContainerStarted","Data":"ed984372de4aa1ee556eb3fd82f37cce3cb34e83a509b9bbae3b5c06f208051a"} Jul 10 17:44:39.643193 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:39.643047 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" event={"ID":"6292244c-8dd3-4cbd-abfa-f10232c66d22","Type":"ContainerStarted","Data":"2aa05e6df447a334d61c5b7459c60328caacf4284ca6690276edd8976b25c446"} Jul 10 17:44:39.670951 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:39.670898 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" podStartSLOduration=1.6708799970000001 podStartE2EDuration="1.670879997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:44:39.668096531 +0000 UTC m=+2011.995794072" watchObservedRunningTime="2026-07-10 17:44:39.670879997 +0000 UTC m=+2011.998577535" Jul 10 17:44:40.107076 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:40.107035 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:44:40.112340 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:40.112314 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:44:40.648457 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:40.648417 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:44:40.649443 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:40.649402 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:44:56.689636 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:56.689587 2583 patch_prober.go:28] interesting pod/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 17:44:56.690014 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:56.689672 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" podUID="6292244c-8dd3-4cbd-abfa-f10232c66d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 17:44:57.759362 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:57.759327 2583 generic.go:356] "Generic (PLEG): container finished" podID="6292244c-8dd3-4cbd-abfa-f10232c66d22" containerID="ed984372de4aa1ee556eb3fd82f37cce3cb34e83a509b9bbae3b5c06f208051a" exitCode=0 Jul 10 17:44:57.759803 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:57.759412 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" event={"ID":"6292244c-8dd3-4cbd-abfa-f10232c66d22","Type":"ContainerDied","Data":"ed984372de4aa1ee556eb3fd82f37cce3cb34e83a509b9bbae3b5c06f208051a"} Jul 10 17:44:57.759803 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:57.759792 2583 scope.go:117] "RemoveContainer" containerID="ed984372de4aa1ee556eb3fd82f37cce3cb34e83a509b9bbae3b5c06f208051a" Jul 10 17:44:57.760811 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:57.760781 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:44:57.760873 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:57.760856 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:44:57.760910 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:57.760892 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:44:58.767240 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:58.767207 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" event={"ID":"6292244c-8dd3-4cbd-abfa-f10232c66d22","Type":"ContainerStarted","Data":"65489f9787aff99681540401410a83bb9a34be458a8628cdb9880a7b08e623a0"} Jul 10 17:44:59.107735 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:59.107617 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:44:59.107735 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:59.107701 2583 patch_prober.go:28] interesting pod/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 10 17:44:59.108097 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:44:59.107767 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" podUID="6292244c-8dd3-4cbd-abfa-f10232c66d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 10 17:45:00.107986 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:00.107949 2583 patch_prober.go:28] interesting pod/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 10 17:45:00.108649 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:00.108612 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" podUID="6292244c-8dd3-4cbd-abfa-f10232c66d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 10 17:45:00.132249 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:00.132215 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728425-tdrsv"] Jul 10 17:45:00.134966 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:00.134943 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728425-tdrsv" Jul 10 17:45:00.136149 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:00.136128 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3eb74-29728425-jlh4f"] Jul 10 17:45:00.137227 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:00.137207 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m7h25\"" Jul 10 17:45:00.138917 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:00.138899 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-050e5c-29728425-9m2nw"] Jul 10 17:45:00.139023 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:00.139004 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3eb74-29728425-jlh4f" Jul 10 17:45:00.141839 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:00.141820 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-050e5c-29728425-9m2nw" Jul 10 17:45:00.144604 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:00.144550 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728425-tdrsv"] Jul 10 17:45:00.147233 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:00.147215 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3eb74-29728425-jlh4f"] Jul 10 17:45:00.148998 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:00.148976 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728425-tdrsv" Jul 10 17:45:00.150693 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:00.150670 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-050e5c-29728425-9m2nw"] Jul 10 17:45:00.159462 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:00.159444 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3eb74-29728425-jlh4f" Jul 10 17:45:00.192237 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:00.192209 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-050e5c-29728425-9m2nw" Jul 10 17:45:00.390966 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:00.390937 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728425-tdrsv"] Jul 10 17:45:00.461585 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:00.461511 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3eb74-29728425-jlh4f"] Jul 10 17:45:00.464684 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:45:00.464652 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6075cd79_21af_404d_afc6_3cff45b5fcdd.slice/crio-e92435e5dc7b563af3aa16be05c73f0e3f5710df186296532d74752155dec2c6 WatchSource:0}: Error finding container e92435e5dc7b563af3aa16be05c73f0e3f5710df186296532d74752155dec2c6: Status 404 returned error can't find the container with id e92435e5dc7b563af3aa16be05c73f0e3f5710df186296532d74752155dec2c6 Jul 10 17:45:00.487546 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:00.487517 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-050e5c-29728425-9m2nw"] Jul 10 17:45:00.487771 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:45:00.487743 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod030a5157_3d18_49b7_a0c0_4108085b1d83.slice/crio-fcc539d61832067483306ebb21c9c189c7f29a463cb852daa64a1b52e2a6287e WatchSource:0}: Error finding container fcc539d61832067483306ebb21c9c189c7f29a463cb852daa64a1b52e2a6287e: Status 404 returned error can't find the container with id fcc539d61832067483306ebb21c9c189c7f29a463cb852daa64a1b52e2a6287e Jul 10 17:45:00.780197 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:00.780155 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728425-tdrsv" event={"ID":"9d44ebe9-7b44-44ee-ac3e-a1d56043d74b","Type":"ContainerStarted","Data":"7381f0c6a1bf19d4c0e2895bfee94fa7a8bfa11545da0bd28334bd23d9c83b8a"} Jul 10 17:45:00.781473 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:00.781438 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-050e5c-29728425-9m2nw" event={"ID":"030a5157-3d18-49b7-a0c0-4108085b1d83","Type":"ContainerStarted","Data":"fcc539d61832067483306ebb21c9c189c7f29a463cb852daa64a1b52e2a6287e"} Jul 10 17:45:00.782652 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:00.782626 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3eb74-29728425-jlh4f" event={"ID":"6075cd79-21af-404d-afc6-3cff45b5fcdd","Type":"ContainerStarted","Data":"e92435e5dc7b563af3aa16be05c73f0e3f5710df186296532d74752155dec2c6"} Jul 10 17:45:01.108185 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:01.108141 2583 patch_prober.go:28] interesting pod/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 10 17:45:01.108698 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:01.108235 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" podUID="6292244c-8dd3-4cbd-abfa-f10232c66d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 10 17:45:01.789818 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:01.789778 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728425-tdrsv" event={"ID":"9d44ebe9-7b44-44ee-ac3e-a1d56043d74b","Type":"ContainerStarted","Data":"9b26a96d4450ad235eb3ae00986f762cc4a5c5099c9d305fcf61500f18171123"} Jul 10 17:45:01.791354 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:01.791326 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-050e5c-29728425-9m2nw" event={"ID":"030a5157-3d18-49b7-a0c0-4108085b1d83","Type":"ContainerStarted","Data":"30e98d11d09f937763b8c7484abbc03cd8a45eec6ca0544313897d9b8e5fb61a"} Jul 10 17:45:01.793286 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:01.793262 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3eb74-29728425-jlh4f" event={"ID":"6075cd79-21af-404d-afc6-3cff45b5fcdd","Type":"ContainerStarted","Data":"f6064619793d25c3cc3f27027d8472eb3851cb6faa66b0864e15771bae063fc4"} Jul 10 17:45:01.809501 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:01.809422 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728425-tdrsv" podStartSLOduration=1.809379826 podStartE2EDuration="1.809379826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:45:01.804524224 +0000 UTC m=+2034.132221764" watchObservedRunningTime="2026-07-10 17:45:01.809379826 +0000 UTC m=+2034.137077367" Jul 10 17:45:01.839697 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:01.839641 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-050e5c-29728425-9m2nw" podStartSLOduration=1.839626257 podStartE2EDuration="1.839626257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:45:01.839426079 +0000 UTC m=+2034.167123617" watchObservedRunningTime="2026-07-10 17:45:01.839626257 +0000 UTC m=+2034.167323795" Jul 10 17:45:01.842072 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:01.842035 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3eb74-29728425-jlh4f" podStartSLOduration=1.8420221909999999 podStartE2EDuration="1.842022191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:45:01.825558329 +0000 UTC m=+2034.153255888" watchObservedRunningTime="2026-07-10 17:45:01.842022191 +0000 UTC m=+2034.169719822" Jul 10 17:45:02.108279 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:02.107798 2583 patch_prober.go:28] interesting pod/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 10 17:45:02.108279 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:02.107863 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" podUID="6292244c-8dd3-4cbd-abfa-f10232c66d22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 10 17:45:02.803100 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:02.803061 2583 generic.go:356] "Generic (PLEG): container finished" podID="9d44ebe9-7b44-44ee-ac3e-a1d56043d74b" containerID="9b26a96d4450ad235eb3ae00986f762cc4a5c5099c9d305fcf61500f18171123" exitCode=0 Jul 10 17:45:02.803313 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:02.803194 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728425-tdrsv" event={"ID":"9d44ebe9-7b44-44ee-ac3e-a1d56043d74b","Type":"ContainerDied","Data":"9b26a96d4450ad235eb3ae00986f762cc4a5c5099c9d305fcf61500f18171123"} Jul 10 17:45:02.805013 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:02.804987 2583 generic.go:356] "Generic (PLEG): container finished" podID="030a5157-3d18-49b7-a0c0-4108085b1d83" containerID="30e98d11d09f937763b8c7484abbc03cd8a45eec6ca0544313897d9b8e5fb61a" exitCode=0 Jul 10 17:45:02.805158 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:02.805018 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-050e5c-29728425-9m2nw" event={"ID":"030a5157-3d18-49b7-a0c0-4108085b1d83","Type":"ContainerDied","Data":"30e98d11d09f937763b8c7484abbc03cd8a45eec6ca0544313897d9b8e5fb61a"} Jul 10 17:45:02.806717 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:02.806693 2583 generic.go:356] "Generic (PLEG): container finished" podID="6075cd79-21af-404d-afc6-3cff45b5fcdd" containerID="f6064619793d25c3cc3f27027d8472eb3851cb6faa66b0864e15771bae063fc4" exitCode=0 Jul 10 17:45:02.806821 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:02.806744 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3eb74-29728425-jlh4f" event={"ID":"6075cd79-21af-404d-afc6-3cff45b5fcdd","Type":"ContainerDied","Data":"f6064619793d25c3cc3f27027d8472eb3851cb6faa66b0864e15771bae063fc4"} Jul 10 17:45:03.110916 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:03.110808 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:45:03.111355 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:03.111197 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:45:03.112070 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:03.112044 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:45:03.975708 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:03.975684 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728425-tdrsv" Jul 10 17:45:04.012379 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:04.012354 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3eb74-29728425-jlh4f" Jul 10 17:45:04.016101 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:04.016074 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-050e5c-29728425-9m2nw" Jul 10 17:45:04.819690 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:04.819661 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3eb74-29728425-jlh4f" Jul 10 17:45:04.820120 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:04.819705 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3eb74-29728425-jlh4f" event={"ID":"6075cd79-21af-404d-afc6-3cff45b5fcdd","Type":"ContainerDied","Data":"e92435e5dc7b563af3aa16be05c73f0e3f5710df186296532d74752155dec2c6"} Jul 10 17:45:04.820120 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:04.819767 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e92435e5dc7b563af3aa16be05c73f0e3f5710df186296532d74752155dec2c6" Jul 10 17:45:04.821166 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:04.821149 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728425-tdrsv" Jul 10 17:45:04.821278 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:04.821152 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728425-tdrsv" event={"ID":"9d44ebe9-7b44-44ee-ac3e-a1d56043d74b","Type":"ContainerDied","Data":"7381f0c6a1bf19d4c0e2895bfee94fa7a8bfa11545da0bd28334bd23d9c83b8a"} Jul 10 17:45:04.821278 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:04.821270 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7381f0c6a1bf19d4c0e2895bfee94fa7a8bfa11545da0bd28334bd23d9c83b8a" Jul 10 17:45:04.822650 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:04.822635 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-050e5c-29728425-9m2nw" Jul 10 17:45:04.822743 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:04.822693 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-050e5c-29728425-9m2nw" event={"ID":"030a5157-3d18-49b7-a0c0-4108085b1d83","Type":"ContainerDied","Data":"fcc539d61832067483306ebb21c9c189c7f29a463cb852daa64a1b52e2a6287e"} Jul 10 17:45:04.822800 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:04.822741 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcc539d61832067483306ebb21c9c189c7f29a463cb852daa64a1b52e2a6287e" Jul 10 17:45:08.340355 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:08.340319 2583 scope.go:117] "RemoveContainer" containerID="b336aeac3c13908bdf4b35f6e1d50a83a865c7c4df64562775ab61823387e485" Jul 10 17:45:50.589898 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:50.589856 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3eb74-9bxm2"] Jul 10 17:45:50.590537 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:50.590274 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6075cd79-21af-404d-afc6-3cff45b5fcdd" containerName="cleanup" Jul 10 17:45:50.590537 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:50.590286 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6075cd79-21af-404d-afc6-3cff45b5fcdd" containerName="cleanup" Jul 10 17:45:50.590537 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:50.590304 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="030a5157-3d18-49b7-a0c0-4108085b1d83" containerName="cleanup" Jul 10 17:45:50.590537 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:50.590309 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="030a5157-3d18-49b7-a0c0-4108085b1d83" containerName="cleanup" Jul 10 17:45:50.590537 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:50.590322 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d44ebe9-7b44-44ee-ac3e-a1d56043d74b" containerName="cleanup" Jul 10 17:45:50.590537 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:50.590333 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d44ebe9-7b44-44ee-ac3e-a1d56043d74b" containerName="cleanup" Jul 10 17:45:50.590537 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:50.590411 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="030a5157-3d18-49b7-a0c0-4108085b1d83" containerName="cleanup" Jul 10 17:45:50.590537 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:50.590425 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d44ebe9-7b44-44ee-ac3e-a1d56043d74b" containerName="cleanup" Jul 10 17:45:50.590537 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:50.590433 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6075cd79-21af-404d-afc6-3cff45b5fcdd" containerName="cleanup" Jul 10 17:45:50.593631 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:50.593609 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3eb74-9bxm2" Jul 10 17:45:50.595926 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:50.595898 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m7h25\"" Jul 10 17:45:50.601062 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:50.601038 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3eb74-9bxm2"] Jul 10 17:45:50.691632 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:50.691597 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/046bbc7c-21a0-41d5-9fe3-ace5b5589eed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a3eb74-9bxm2\" (UID: \"046bbc7c-21a0-41d5-9fe3-ace5b5589eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3eb74-9bxm2" Jul 10 17:45:50.792056 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:50.792024 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/046bbc7c-21a0-41d5-9fe3-ace5b5589eed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a3eb74-9bxm2\" (UID: \"046bbc7c-21a0-41d5-9fe3-ace5b5589eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3eb74-9bxm2" Jul 10 17:45:50.792743 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:50.792715 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/046bbc7c-21a0-41d5-9fe3-ace5b5589eed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a3eb74-9bxm2\" (UID: \"046bbc7c-21a0-41d5-9fe3-ace5b5589eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3eb74-9bxm2" Jul 10 17:45:50.908660 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:50.908579 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3eb74-9bxm2" Jul 10 17:45:51.047774 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:51.047739 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3eb74-9bxm2"] Jul 10 17:45:51.050037 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:45:51.050012 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod046bbc7c_21a0_41d5_9fe3_ace5b5589eed.slice/crio-f26f393419fde768ac24156508d1a65ebce63057770838920476a4a333368aef WatchSource:0}: Error finding container f26f393419fde768ac24156508d1a65ebce63057770838920476a4a333368aef: Status 404 returned error can't find the container with id f26f393419fde768ac24156508d1a65ebce63057770838920476a4a333368aef Jul 10 17:45:51.088153 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:51.088119 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3eb74-9bxm2" event={"ID":"046bbc7c-21a0-41d5-9fe3-ace5b5589eed","Type":"ContainerStarted","Data":"f26f393419fde768ac24156508d1a65ebce63057770838920476a4a333368aef"} Jul 10 17:45:52.095549 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:52.095451 2583 generic.go:356] "Generic (PLEG): container finished" podID="046bbc7c-21a0-41d5-9fe3-ace5b5589eed" containerID="9cbeafd469093746502378c9620cca3805b875039d0f91dc85e2fbbc12bbc9a8" exitCode=0 Jul 10 17:45:52.095549 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:52.095511 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3eb74-9bxm2" event={"ID":"046bbc7c-21a0-41d5-9fe3-ace5b5589eed","Type":"ContainerDied","Data":"9cbeafd469093746502378c9620cca3805b875039d0f91dc85e2fbbc12bbc9a8"} Jul 10 17:45:53.242947 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:53.242924 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3eb74-9bxm2" Jul 10 17:45:53.316141 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:53.316100 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/046bbc7c-21a0-41d5-9fe3-ace5b5589eed-maas-api-service-ca\") pod \"046bbc7c-21a0-41d5-9fe3-ace5b5589eed\" (UID: \"046bbc7c-21a0-41d5-9fe3-ace5b5589eed\") " Jul 10 17:45:53.316452 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:53.316433 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/046bbc7c-21a0-41d5-9fe3-ace5b5589eed-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "046bbc7c-21a0-41d5-9fe3-ace5b5589eed" (UID: "046bbc7c-21a0-41d5-9fe3-ace5b5589eed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:45:53.416806 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:53.416778 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/046bbc7c-21a0-41d5-9fe3-ace5b5589eed-maas-api-service-ca\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:45:54.111743 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:54.111711 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3eb74-9bxm2" Jul 10 17:45:54.111912 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:54.111710 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a3eb74-9bxm2" event={"ID":"046bbc7c-21a0-41d5-9fe3-ace5b5589eed","Type":"ContainerDied","Data":"f26f393419fde768ac24156508d1a65ebce63057770838920476a4a333368aef"} Jul 10 17:45:54.111912 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:45:54.111838 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f26f393419fde768ac24156508d1a65ebce63057770838920476a4a333368aef" Jul 10 17:46:01.021036 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:01.020993 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3eb74-29728425-jlh4f"] Jul 10 17:46:01.024217 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:01.024192 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a3eb74-29728425-jlh4f"] Jul 10 17:46:02.206815 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:02.206776 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6075cd79-21af-404d-afc6-3cff45b5fcdd" path="/var/lib/kubelet/pods/6075cd79-21af-404d-afc6-3cff45b5fcdd/volumes" Jul 10 17:46:08.273704 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:08.273661 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js69w_e41baaf7-4171-4df1-8bdd-d0697a61fdce/ovn-acl-logging/0.log" Jul 10 17:46:08.279911 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:08.279889 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js69w_e41baaf7-4171-4df1-8bdd-d0697a61fdce/ovn-acl-logging/0.log" Jul 10 17:46:27.151330 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:27.151241 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-050e5c-29728425-9m2nw"] Jul 10 17:46:27.154113 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:27.154084 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-050e5c-29728425-9m2nw"] Jul 10 17:46:28.206185 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:28.206150 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="030a5157-3d18-49b7-a0c0-4108085b1d83" path="/var/lib/kubelet/pods/030a5157-3d18-49b7-a0c0-4108085b1d83/volumes" Jul 10 17:46:42.500305 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:42.500271 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq"] Jul 10 17:46:42.500748 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:42.500573 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" podUID="6292244c-8dd3-4cbd-abfa-f10232c66d22" containerName="istio-proxy" containerID="cri-o://65489f9787aff99681540401410a83bb9a34be458a8628cdb9880a7b08e623a0" gracePeriod=30 Jul 10 17:46:47.772289 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:47.772225 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:46:47.790821 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:47.790791 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6292244c-8dd3-4cbd-abfa-f10232c66d22-workload-certs\") pod \"6292244c-8dd3-4cbd-abfa-f10232c66d22\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " Jul 10 17:46:47.790821 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:47.790827 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6292244c-8dd3-4cbd-abfa-f10232c66d22-istiod-ca-cert\") pod \"6292244c-8dd3-4cbd-abfa-f10232c66d22\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " Jul 10 17:46:47.791036 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:47.790877 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6292244c-8dd3-4cbd-abfa-f10232c66d22-istio-podinfo\") pod \"6292244c-8dd3-4cbd-abfa-f10232c66d22\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " Jul 10 17:46:47.791036 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:47.790896 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6292244c-8dd3-4cbd-abfa-f10232c66d22-workload-socket\") pod \"6292244c-8dd3-4cbd-abfa-f10232c66d22\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " Jul 10 17:46:47.791036 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:47.790930 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6292244c-8dd3-4cbd-abfa-f10232c66d22-istio-envoy\") pod \"6292244c-8dd3-4cbd-abfa-f10232c66d22\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " Jul 10 17:46:47.791036 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:47.790956 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6292244c-8dd3-4cbd-abfa-f10232c66d22-istio-data\") pod \"6292244c-8dd3-4cbd-abfa-f10232c66d22\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " Jul 10 17:46:47.791036 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:47.790990 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kq46\" (UniqueName: \"kubernetes.io/projected/6292244c-8dd3-4cbd-abfa-f10232c66d22-kube-api-access-4kq46\") pod \"6292244c-8dd3-4cbd-abfa-f10232c66d22\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " Jul 10 17:46:47.791036 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:47.791022 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6292244c-8dd3-4cbd-abfa-f10232c66d22-istio-token\") pod \"6292244c-8dd3-4cbd-abfa-f10232c66d22\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " Jul 10 17:46:47.791335 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:47.791062 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6292244c-8dd3-4cbd-abfa-f10232c66d22-credential-socket\") pod \"6292244c-8dd3-4cbd-abfa-f10232c66d22\" (UID: \"6292244c-8dd3-4cbd-abfa-f10232c66d22\") " Jul 10 17:46:47.791335 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:47.791120 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6292244c-8dd3-4cbd-abfa-f10232c66d22-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6292244c-8dd3-4cbd-abfa-f10232c66d22" (UID: "6292244c-8dd3-4cbd-abfa-f10232c66d22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:46:47.791335 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:47.791245 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6292244c-8dd3-4cbd-abfa-f10232c66d22-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6292244c-8dd3-4cbd-abfa-f10232c66d22" (UID: "6292244c-8dd3-4cbd-abfa-f10232c66d22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:46:47.791335 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:47.791315 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6292244c-8dd3-4cbd-abfa-f10232c66d22-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6292244c-8dd3-4cbd-abfa-f10232c66d22" (UID: "6292244c-8dd3-4cbd-abfa-f10232c66d22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:46:47.791573 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:47.791348 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6292244c-8dd3-4cbd-abfa-f10232c66d22-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6292244c-8dd3-4cbd-abfa-f10232c66d22" (UID: "6292244c-8dd3-4cbd-abfa-f10232c66d22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:46:47.791573 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:47.791415 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6292244c-8dd3-4cbd-abfa-f10232c66d22-credential-socket\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:46:47.791573 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:47.791431 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6292244c-8dd3-4cbd-abfa-f10232c66d22-workload-certs\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:46:47.791573 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:47.791442 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6292244c-8dd3-4cbd-abfa-f10232c66d22-workload-socket\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:46:47.792553 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:47.792513 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6292244c-8dd3-4cbd-abfa-f10232c66d22-istio-data" (OuterVolumeSpecName: "istio-data") pod "6292244c-8dd3-4cbd-abfa-f10232c66d22" (UID: "6292244c-8dd3-4cbd-abfa-f10232c66d22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:46:47.795138 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:47.794711 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6292244c-8dd3-4cbd-abfa-f10232c66d22-istio-token" (OuterVolumeSpecName: "istio-token") pod "6292244c-8dd3-4cbd-abfa-f10232c66d22" (UID: "6292244c-8dd3-4cbd-abfa-f10232c66d22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:46:47.795138 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:47.794826 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6292244c-8dd3-4cbd-abfa-f10232c66d22-kube-api-access-4kq46" (OuterVolumeSpecName: "kube-api-access-4kq46") pod "6292244c-8dd3-4cbd-abfa-f10232c66d22" (UID: "6292244c-8dd3-4cbd-abfa-f10232c66d22"). InnerVolumeSpecName "kube-api-access-4kq46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:46:47.795138 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:47.795087 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6292244c-8dd3-4cbd-abfa-f10232c66d22-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6292244c-8dd3-4cbd-abfa-f10232c66d22" (UID: "6292244c-8dd3-4cbd-abfa-f10232c66d22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:46:47.795481 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:47.795454 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6292244c-8dd3-4cbd-abfa-f10232c66d22-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6292244c-8dd3-4cbd-abfa-f10232c66d22" (UID: "6292244c-8dd3-4cbd-abfa-f10232c66d22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 17:46:47.892093 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:47.892054 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kq46\" (UniqueName: \"kubernetes.io/projected/6292244c-8dd3-4cbd-abfa-f10232c66d22-kube-api-access-4kq46\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:46:47.892093 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:47.892088 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6292244c-8dd3-4cbd-abfa-f10232c66d22-istio-token\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:46:47.892093 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:47.892097 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6292244c-8dd3-4cbd-abfa-f10232c66d22-istiod-ca-cert\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:46:47.892093 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:47.892105 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6292244c-8dd3-4cbd-abfa-f10232c66d22-istio-podinfo\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:46:47.892364 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:47.892112 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6292244c-8dd3-4cbd-abfa-f10232c66d22-istio-envoy\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:46:47.892364 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:47.892119 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6292244c-8dd3-4cbd-abfa-f10232c66d22-istio-data\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:46:48.419445 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:48.419379 2583 generic.go:356] "Generic (PLEG): container finished" podID="6292244c-8dd3-4cbd-abfa-f10232c66d22" containerID="65489f9787aff99681540401410a83bb9a34be458a8628cdb9880a7b08e623a0" exitCode=0 Jul 10 17:46:48.419645 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:48.419470 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" Jul 10 17:46:48.419645 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:48.419469 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" event={"ID":"6292244c-8dd3-4cbd-abfa-f10232c66d22","Type":"ContainerDied","Data":"65489f9787aff99681540401410a83bb9a34be458a8628cdb9880a7b08e623a0"} Jul 10 17:46:48.419645 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:48.419605 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq" event={"ID":"6292244c-8dd3-4cbd-abfa-f10232c66d22","Type":"ContainerDied","Data":"2aa05e6df447a334d61c5b7459c60328caacf4284ca6690276edd8976b25c446"} Jul 10 17:46:48.419645 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:48.419637 2583 scope.go:117] "RemoveContainer" containerID="65489f9787aff99681540401410a83bb9a34be458a8628cdb9880a7b08e623a0" Jul 10 17:46:48.429229 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:48.429210 2583 scope.go:117] "RemoveContainer" containerID="ed984372de4aa1ee556eb3fd82f37cce3cb34e83a509b9bbae3b5c06f208051a" Jul 10 17:46:48.437762 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:48.437736 2583 scope.go:117] "RemoveContainer" containerID="65489f9787aff99681540401410a83bb9a34be458a8628cdb9880a7b08e623a0" Jul 10 17:46:48.438038 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:46:48.438016 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65489f9787aff99681540401410a83bb9a34be458a8628cdb9880a7b08e623a0\": container with ID starting with 65489f9787aff99681540401410a83bb9a34be458a8628cdb9880a7b08e623a0 not found: ID does not exist" containerID="65489f9787aff99681540401410a83bb9a34be458a8628cdb9880a7b08e623a0" Jul 10 17:46:48.438127 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:48.438043 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65489f9787aff99681540401410a83bb9a34be458a8628cdb9880a7b08e623a0"} err="failed to get container status \"65489f9787aff99681540401410a83bb9a34be458a8628cdb9880a7b08e623a0\": rpc error: code = NotFound desc = could not find container \"65489f9787aff99681540401410a83bb9a34be458a8628cdb9880a7b08e623a0\": container with ID starting with 65489f9787aff99681540401410a83bb9a34be458a8628cdb9880a7b08e623a0 not found: ID does not exist" Jul 10 17:46:48.438127 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:48.438064 2583 scope.go:117] "RemoveContainer" containerID="ed984372de4aa1ee556eb3fd82f37cce3cb34e83a509b9bbae3b5c06f208051a" Jul 10 17:46:48.438313 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:46:48.438299 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed984372de4aa1ee556eb3fd82f37cce3cb34e83a509b9bbae3b5c06f208051a\": container with ID starting with ed984372de4aa1ee556eb3fd82f37cce3cb34e83a509b9bbae3b5c06f208051a not found: ID does not exist" containerID="ed984372de4aa1ee556eb3fd82f37cce3cb34e83a509b9bbae3b5c06f208051a" Jul 10 17:46:48.438350 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:48.438316 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed984372de4aa1ee556eb3fd82f37cce3cb34e83a509b9bbae3b5c06f208051a"} err="failed to get container status \"ed984372de4aa1ee556eb3fd82f37cce3cb34e83a509b9bbae3b5c06f208051a\": rpc error: code = NotFound desc = could not find container \"ed984372de4aa1ee556eb3fd82f37cce3cb34e83a509b9bbae3b5c06f208051a\": container with ID starting with ed984372de4aa1ee556eb3fd82f37cce3cb34e83a509b9bbae3b5c06f208051a not found: ID does not exist" Jul 10 17:46:48.442484 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:48.442461 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq"] Jul 10 17:46:48.447303 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:48.447280 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-050e5c-openshift-default-5cb8c5f7cf-2mnmq"] Jul 10 17:46:50.206377 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:50.206347 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6292244c-8dd3-4cbd-abfa-f10232c66d22" path="/var/lib/kubelet/pods/6292244c-8dd3-4cbd-abfa-f10232c66d22/volumes" Jul 10 17:46:51.739430 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.739377 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc"] Jul 10 17:46:51.740516 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.740489 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6292244c-8dd3-4cbd-abfa-f10232c66d22" containerName="istio-proxy" Jul 10 17:46:51.740516 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.740516 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6292244c-8dd3-4cbd-abfa-f10232c66d22" containerName="istio-proxy" Jul 10 17:46:51.740675 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.740536 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="046bbc7c-21a0-41d5-9fe3-ace5b5589eed" containerName="revoke-keys" Jul 10 17:46:51.740675 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.740545 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="046bbc7c-21a0-41d5-9fe3-ace5b5589eed" containerName="revoke-keys" Jul 10 17:46:51.740675 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.740565 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6292244c-8dd3-4cbd-abfa-f10232c66d22" containerName="istio-proxy" Jul 10 17:46:51.740675 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.740573 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6292244c-8dd3-4cbd-abfa-f10232c66d22" containerName="istio-proxy" Jul 10 17:46:51.740675 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.740654 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6292244c-8dd3-4cbd-abfa-f10232c66d22" containerName="istio-proxy" Jul 10 17:46:51.740675 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.740666 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6292244c-8dd3-4cbd-abfa-f10232c66d22" containerName="istio-proxy" Jul 10 17:46:51.740860 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.740691 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="046bbc7c-21a0-41d5-9fe3-ace5b5589eed" containerName="revoke-keys" Jul 10 17:46:51.745417 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.745380 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:46:51.747778 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.747758 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-ad332c-openshift-default-dockercfg-8mzj9\"" Jul 10 17:46:51.755495 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.755457 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc"] Jul 10 17:46:51.823718 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.823680 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7479bc51-6333-4836-b528-6338bcb3b62c-istio-token\") pod \"e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:46:51.823891 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.823732 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7479bc51-6333-4836-b528-6338bcb3b62c-credential-socket\") pod \"e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:46:51.823891 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.823753 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzwpx\" (UniqueName: \"kubernetes.io/projected/7479bc51-6333-4836-b528-6338bcb3b62c-kube-api-access-wzwpx\") pod \"e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:46:51.823891 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.823826 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7479bc51-6333-4836-b528-6338bcb3b62c-istio-data\") pod \"e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:46:51.823891 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.823862 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7479bc51-6333-4836-b528-6338bcb3b62c-istiod-ca-cert\") pod \"e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:46:51.824057 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.823904 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7479bc51-6333-4836-b528-6338bcb3b62c-istio-podinfo\") pod \"e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:46:51.824057 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.823924 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7479bc51-6333-4836-b528-6338bcb3b62c-istio-envoy\") pod \"e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:46:51.824057 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.823950 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7479bc51-6333-4836-b528-6338bcb3b62c-workload-certs\") pod \"e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:46:51.824057 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.823978 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7479bc51-6333-4836-b528-6338bcb3b62c-workload-socket\") pod \"e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:46:51.925330 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.925292 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7479bc51-6333-4836-b528-6338bcb3b62c-workload-certs\") pod \"e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:46:51.925330 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.925332 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7479bc51-6333-4836-b528-6338bcb3b62c-workload-socket\") pod \"e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:46:51.925616 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.925384 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7479bc51-6333-4836-b528-6338bcb3b62c-istio-token\") pod \"e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:46:51.925616 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.925433 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7479bc51-6333-4836-b528-6338bcb3b62c-credential-socket\") pod \"e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:46:51.925616 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.925469 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzwpx\" (UniqueName: \"kubernetes.io/projected/7479bc51-6333-4836-b528-6338bcb3b62c-kube-api-access-wzwpx\") pod \"e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:46:51.925616 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.925500 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7479bc51-6333-4836-b528-6338bcb3b62c-istio-data\") pod \"e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:46:51.925616 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.925525 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7479bc51-6333-4836-b528-6338bcb3b62c-istiod-ca-cert\") pod \"e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:46:51.925872 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.925712 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7479bc51-6333-4836-b528-6338bcb3b62c-istio-podinfo\") pod \"e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:46:51.925872 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.925764 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7479bc51-6333-4836-b528-6338bcb3b62c-istio-envoy\") pod \"e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:46:51.925872 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.925779 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7479bc51-6333-4836-b528-6338bcb3b62c-workload-certs\") pod \"e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:46:51.925872 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.925819 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7479bc51-6333-4836-b528-6338bcb3b62c-workload-socket\") pod \"e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:46:51.926082 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.925901 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7479bc51-6333-4836-b528-6338bcb3b62c-credential-socket\") pod \"e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:46:51.926082 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.925967 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7479bc51-6333-4836-b528-6338bcb3b62c-istio-data\") pod \"e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:46:51.926186 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.926113 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7479bc51-6333-4836-b528-6338bcb3b62c-istiod-ca-cert\") pod \"e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:46:51.928036 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.928013 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7479bc51-6333-4836-b528-6338bcb3b62c-istio-envoy\") pod \"e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:46:51.928401 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.928363 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7479bc51-6333-4836-b528-6338bcb3b62c-istio-podinfo\") pod \"e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:46:51.934449 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.934429 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7479bc51-6333-4836-b528-6338bcb3b62c-istio-token\") pod \"e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:46:51.934554 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:51.934458 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzwpx\" (UniqueName: \"kubernetes.io/projected/7479bc51-6333-4836-b528-6338bcb3b62c-kube-api-access-wzwpx\") pod \"e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:46:52.057545 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:52.057457 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:46:52.203272 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:46:52.202678 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7479bc51_6333_4836_b528_6338bcb3b62c.slice/crio-3401081033d8a8cc4e078c431d40698892e9fe9c3118fdc5812a9975ca279d7c WatchSource:0}: Error finding container 3401081033d8a8cc4e078c431d40698892e9fe9c3118fdc5812a9975ca279d7c: Status 404 returned error can't find the container with id 3401081033d8a8cc4e078c431d40698892e9fe9c3118fdc5812a9975ca279d7c Jul 10 17:46:52.204879 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:52.204842 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:46:52.205057 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:52.204913 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:46:52.205057 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:52.204941 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:46:52.207497 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:52.207477 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc"] Jul 10 17:46:52.445359 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:52.445313 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" event={"ID":"7479bc51-6333-4836-b528-6338bcb3b62c","Type":"ContainerStarted","Data":"9bf5fa36d34bfca02f378ac58c4bce8c4a42dfe1e26e7be0558d66a12f29550e"} Jul 10 17:46:52.445583 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:52.445383 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" event={"ID":"7479bc51-6333-4836-b528-6338bcb3b62c","Type":"ContainerStarted","Data":"3401081033d8a8cc4e078c431d40698892e9fe9c3118fdc5812a9975ca279d7c"} Jul 10 17:46:52.466579 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:52.466531 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" podStartSLOduration=1.466514948 podStartE2EDuration="1.466514948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:46:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:46:52.463655054 +0000 UTC m=+2144.791352593" watchObservedRunningTime="2026-07-10 17:46:52.466514948 +0000 UTC m=+2144.794212486" Jul 10 17:46:53.058534 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:53.058499 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:46:53.063434 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:53.063410 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:46:53.451647 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:53.451613 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:46:53.452723 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:46:53.452700 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:47:07.054442 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:07.054382 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bb8e-jj6sf"] Jul 10 17:47:07.058661 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:07.058644 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bb8e-jj6sf" Jul 10 17:47:07.061371 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:07.061339 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m7h25\"" Jul 10 17:47:07.063740 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:07.063676 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bb8e-jj6sf"] Jul 10 17:47:07.168162 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:07.168120 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86d04545-4836-436b-9834-f163427b2a72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-43bb8e-jj6sf\" (UID: \"86d04545-4836-436b-9834-f163427b2a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bb8e-jj6sf" Jul 10 17:47:07.268984 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:07.268947 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86d04545-4836-436b-9834-f163427b2a72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-43bb8e-jj6sf\" (UID: \"86d04545-4836-436b-9834-f163427b2a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bb8e-jj6sf" Jul 10 17:47:07.269635 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:07.269613 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86d04545-4836-436b-9834-f163427b2a72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-43bb8e-jj6sf\" (UID: \"86d04545-4836-436b-9834-f163427b2a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bb8e-jj6sf" Jul 10 17:47:07.371202 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:07.371113 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bb8e-jj6sf" Jul 10 17:47:07.539559 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:07.539526 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bb8e-jj6sf"] Jul 10 17:47:07.540494 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:47:07.540450 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d04545_4836_436b_9834_f163427b2a72.slice/crio-f9cf73cb168a1cdab0b3e5977182060b3424194ed9bf59915f3313bb244e207d WatchSource:0}: Error finding container f9cf73cb168a1cdab0b3e5977182060b3424194ed9bf59915f3313bb244e207d: Status 404 returned error can't find the container with id f9cf73cb168a1cdab0b3e5977182060b3424194ed9bf59915f3313bb244e207d Jul 10 17:47:08.546525 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:08.546482 2583 generic.go:356] "Generic (PLEG): container finished" podID="86d04545-4836-436b-9834-f163427b2a72" containerID="f6f362bcb328f6ef51c0408b060022db0f1c0f9407fae00ecd93c75a5488a136" exitCode=0 Jul 10 17:47:08.546960 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:08.546545 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bb8e-jj6sf" event={"ID":"86d04545-4836-436b-9834-f163427b2a72","Type":"ContainerDied","Data":"f6f362bcb328f6ef51c0408b060022db0f1c0f9407fae00ecd93c75a5488a136"} Jul 10 17:47:08.546960 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:08.546590 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bb8e-jj6sf" event={"ID":"86d04545-4836-436b-9834-f163427b2a72","Type":"ContainerStarted","Data":"f9cf73cb168a1cdab0b3e5977182060b3424194ed9bf59915f3313bb244e207d"} Jul 10 17:47:09.492673 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:09.492628 2583 patch_prober.go:28] interesting pod/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 17:47:09.492943 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:09.492690 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" podUID="7479bc51-6333-4836-b528-6338bcb3b62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 17:47:09.692848 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:09.692823 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bb8e-jj6sf" Jul 10 17:47:09.792024 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:09.791945 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86d04545-4836-436b-9834-f163427b2a72-maas-api-service-ca\") pod \"86d04545-4836-436b-9834-f163427b2a72\" (UID: \"86d04545-4836-436b-9834-f163427b2a72\") " Jul 10 17:47:09.792428 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:09.792368 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86d04545-4836-436b-9834-f163427b2a72-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "86d04545-4836-436b-9834-f163427b2a72" (UID: "86d04545-4836-436b-9834-f163427b2a72"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:47:09.893595 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:09.893565 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86d04545-4836-436b-9834-f163427b2a72-maas-api-service-ca\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:47:10.558128 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:10.558101 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bb8e-jj6sf" Jul 10 17:47:10.558342 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:10.558139 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bb8e-jj6sf" event={"ID":"86d04545-4836-436b-9834-f163427b2a72","Type":"ContainerDied","Data":"f9cf73cb168a1cdab0b3e5977182060b3424194ed9bf59915f3313bb244e207d"} Jul 10 17:47:10.558342 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:10.558188 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9cf73cb168a1cdab0b3e5977182060b3424194ed9bf59915f3313bb244e207d" Jul 10 17:47:10.559854 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:10.559827 2583 generic.go:356] "Generic (PLEG): container finished" podID="7479bc51-6333-4836-b528-6338bcb3b62c" containerID="9bf5fa36d34bfca02f378ac58c4bce8c4a42dfe1e26e7be0558d66a12f29550e" exitCode=0 Jul 10 17:47:10.559983 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:10.559907 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" event={"ID":"7479bc51-6333-4836-b528-6338bcb3b62c","Type":"ContainerDied","Data":"9bf5fa36d34bfca02f378ac58c4bce8c4a42dfe1e26e7be0558d66a12f29550e"} Jul 10 17:47:10.560328 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:10.560310 2583 scope.go:117] "RemoveContainer" containerID="9bf5fa36d34bfca02f378ac58c4bce8c4a42dfe1e26e7be0558d66a12f29550e" Jul 10 17:47:10.561290 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:10.561262 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:47:10.561378 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:10.561344 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:47:10.561484 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:10.561413 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:47:11.567548 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:11.567507 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" event={"ID":"7479bc51-6333-4836-b528-6338bcb3b62c","Type":"ContainerStarted","Data":"0c76f513d454e4e5daccd3d22a23933a7ffd742455dd18c8fe9b328af58b46c8"} Jul 10 17:47:12.058468 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:12.058427 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:47:12.058766 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:12.058738 2583 patch_prober.go:28] interesting pod/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 10 17:47:12.058866 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:12.058803 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" podUID="7479bc51-6333-4836-b528-6338bcb3b62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 10 17:47:13.057956 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:13.057915 2583 patch_prober.go:28] interesting pod/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 10 17:47:13.058348 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:13.057969 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" podUID="7479bc51-6333-4836-b528-6338bcb3b62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 10 17:47:14.058191 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:14.058158 2583 patch_prober.go:28] interesting pod/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 10 17:47:14.058602 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:14.058219 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" podUID="7479bc51-6333-4836-b528-6338bcb3b62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 10 17:47:15.057789 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:15.057757 2583 patch_prober.go:28] interesting pod/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 10 17:47:15.057989 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:15.057815 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" podUID="7479bc51-6333-4836-b528-6338bcb3b62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 10 17:47:16.061485 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:16.061454 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:47:16.062012 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:16.061988 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:47:16.062747 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:16.062726 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:47:59.096844 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:59.096809 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc"] Jul 10 17:47:59.097480 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:47:59.097101 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" podUID="7479bc51-6333-4836-b528-6338bcb3b62c" containerName="istio-proxy" containerID="cri-o://0c76f513d454e4e5daccd3d22a23933a7ffd742455dd18c8fe9b328af58b46c8" gracePeriod=30 Jul 10 17:48:01.064762 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:01.064719 2583 patch_prober.go:28] interesting pod/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 17:48:01.065223 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:01.064784 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" podUID="7479bc51-6333-4836-b528-6338bcb3b62c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 17:48:04.361892 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.361815 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:48:04.489245 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.489208 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7479bc51-6333-4836-b528-6338bcb3b62c-workload-socket\") pod \"7479bc51-6333-4836-b528-6338bcb3b62c\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " Jul 10 17:48:04.489474 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.489259 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7479bc51-6333-4836-b528-6338bcb3b62c-istio-envoy\") pod \"7479bc51-6333-4836-b528-6338bcb3b62c\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " Jul 10 17:48:04.489474 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.489295 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzwpx\" (UniqueName: \"kubernetes.io/projected/7479bc51-6333-4836-b528-6338bcb3b62c-kube-api-access-wzwpx\") pod \"7479bc51-6333-4836-b528-6338bcb3b62c\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " Jul 10 17:48:04.489474 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.489316 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7479bc51-6333-4836-b528-6338bcb3b62c-istio-data\") pod \"7479bc51-6333-4836-b528-6338bcb3b62c\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " Jul 10 17:48:04.489474 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.489358 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7479bc51-6333-4836-b528-6338bcb3b62c-istiod-ca-cert\") pod \"7479bc51-6333-4836-b528-6338bcb3b62c\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " Jul 10 17:48:04.489474 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.489411 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7479bc51-6333-4836-b528-6338bcb3b62c-credential-socket\") pod \"7479bc51-6333-4836-b528-6338bcb3b62c\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " Jul 10 17:48:04.489474 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.489457 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7479bc51-6333-4836-b528-6338bcb3b62c-istio-token\") pod \"7479bc51-6333-4836-b528-6338bcb3b62c\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " Jul 10 17:48:04.489818 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.489488 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7479bc51-6333-4836-b528-6338bcb3b62c-workload-certs\") pod \"7479bc51-6333-4836-b528-6338bcb3b62c\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " Jul 10 17:48:04.489818 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.489528 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7479bc51-6333-4836-b528-6338bcb3b62c-istio-podinfo\") pod \"7479bc51-6333-4836-b528-6338bcb3b62c\" (UID: \"7479bc51-6333-4836-b528-6338bcb3b62c\") " Jul 10 17:48:04.489818 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.489550 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7479bc51-6333-4836-b528-6338bcb3b62c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7479bc51-6333-4836-b528-6338bcb3b62c" (UID: "7479bc51-6333-4836-b528-6338bcb3b62c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:48:04.489818 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.489729 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7479bc51-6333-4836-b528-6338bcb3b62c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7479bc51-6333-4836-b528-6338bcb3b62c" (UID: "7479bc51-6333-4836-b528-6338bcb3b62c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:48:04.490099 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.489818 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7479bc51-6333-4836-b528-6338bcb3b62c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7479bc51-6333-4836-b528-6338bcb3b62c" (UID: "7479bc51-6333-4836-b528-6338bcb3b62c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:48:04.490099 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.489851 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7479bc51-6333-4836-b528-6338bcb3b62c-workload-socket\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:48:04.490099 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.489872 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7479bc51-6333-4836-b528-6338bcb3b62c-credential-socket\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:48:04.490099 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.489889 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7479bc51-6333-4836-b528-6338bcb3b62c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7479bc51-6333-4836-b528-6338bcb3b62c" (UID: "7479bc51-6333-4836-b528-6338bcb3b62c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:48:04.490631 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.490578 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7479bc51-6333-4836-b528-6338bcb3b62c-istio-data" (OuterVolumeSpecName: "istio-data") pod "7479bc51-6333-4836-b528-6338bcb3b62c" (UID: "7479bc51-6333-4836-b528-6338bcb3b62c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:48:04.491814 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.491785 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7479bc51-6333-4836-b528-6338bcb3b62c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7479bc51-6333-4836-b528-6338bcb3b62c" (UID: "7479bc51-6333-4836-b528-6338bcb3b62c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:48:04.491907 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.491833 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7479bc51-6333-4836-b528-6338bcb3b62c-kube-api-access-wzwpx" (OuterVolumeSpecName: "kube-api-access-wzwpx") pod "7479bc51-6333-4836-b528-6338bcb3b62c" (UID: "7479bc51-6333-4836-b528-6338bcb3b62c"). InnerVolumeSpecName "kube-api-access-wzwpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:48:04.492261 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.492245 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7479bc51-6333-4836-b528-6338bcb3b62c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7479bc51-6333-4836-b528-6338bcb3b62c" (UID: "7479bc51-6333-4836-b528-6338bcb3b62c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 17:48:04.492305 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.492256 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7479bc51-6333-4836-b528-6338bcb3b62c-istio-token" (OuterVolumeSpecName: "istio-token") pod "7479bc51-6333-4836-b528-6338bcb3b62c" (UID: "7479bc51-6333-4836-b528-6338bcb3b62c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:48:04.591043 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.591008 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7479bc51-6333-4836-b528-6338bcb3b62c-istio-token\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:48:04.591043 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.591040 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7479bc51-6333-4836-b528-6338bcb3b62c-workload-certs\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:48:04.591043 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.591053 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7479bc51-6333-4836-b528-6338bcb3b62c-istio-podinfo\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:48:04.591294 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.591064 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7479bc51-6333-4836-b528-6338bcb3b62c-istio-envoy\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:48:04.591294 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.591076 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzwpx\" (UniqueName: \"kubernetes.io/projected/7479bc51-6333-4836-b528-6338bcb3b62c-kube-api-access-wzwpx\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:48:04.591294 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.591087 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7479bc51-6333-4836-b528-6338bcb3b62c-istio-data\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:48:04.591294 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.591098 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7479bc51-6333-4836-b528-6338bcb3b62c-istiod-ca-cert\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:48:04.873969 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.873936 2583 generic.go:356] "Generic (PLEG): container finished" podID="7479bc51-6333-4836-b528-6338bcb3b62c" containerID="0c76f513d454e4e5daccd3d22a23933a7ffd742455dd18c8fe9b328af58b46c8" exitCode=0 Jul 10 17:48:04.874146 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.873983 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" Jul 10 17:48:04.874146 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.874019 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" event={"ID":"7479bc51-6333-4836-b528-6338bcb3b62c","Type":"ContainerDied","Data":"0c76f513d454e4e5daccd3d22a23933a7ffd742455dd18c8fe9b328af58b46c8"} Jul 10 17:48:04.874146 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.874077 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc" event={"ID":"7479bc51-6333-4836-b528-6338bcb3b62c","Type":"ContainerDied","Data":"3401081033d8a8cc4e078c431d40698892e9fe9c3118fdc5812a9975ca279d7c"} Jul 10 17:48:04.874146 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.874107 2583 scope.go:117] "RemoveContainer" containerID="0c76f513d454e4e5daccd3d22a23933a7ffd742455dd18c8fe9b328af58b46c8" Jul 10 17:48:04.884632 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.884611 2583 scope.go:117] "RemoveContainer" containerID="9bf5fa36d34bfca02f378ac58c4bce8c4a42dfe1e26e7be0558d66a12f29550e" Jul 10 17:48:04.893423 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.893381 2583 scope.go:117] "RemoveContainer" containerID="0c76f513d454e4e5daccd3d22a23933a7ffd742455dd18c8fe9b328af58b46c8" Jul 10 17:48:04.893821 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:48:04.893789 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c76f513d454e4e5daccd3d22a23933a7ffd742455dd18c8fe9b328af58b46c8\": container with ID starting with 0c76f513d454e4e5daccd3d22a23933a7ffd742455dd18c8fe9b328af58b46c8 not found: ID does not exist" containerID="0c76f513d454e4e5daccd3d22a23933a7ffd742455dd18c8fe9b328af58b46c8" Jul 10 17:48:04.893933 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.893832 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c76f513d454e4e5daccd3d22a23933a7ffd742455dd18c8fe9b328af58b46c8"} err="failed to get container status \"0c76f513d454e4e5daccd3d22a23933a7ffd742455dd18c8fe9b328af58b46c8\": rpc error: code = NotFound desc = could not find container \"0c76f513d454e4e5daccd3d22a23933a7ffd742455dd18c8fe9b328af58b46c8\": container with ID starting with 0c76f513d454e4e5daccd3d22a23933a7ffd742455dd18c8fe9b328af58b46c8 not found: ID does not exist" Jul 10 17:48:04.893933 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.893857 2583 scope.go:117] "RemoveContainer" containerID="9bf5fa36d34bfca02f378ac58c4bce8c4a42dfe1e26e7be0558d66a12f29550e" Jul 10 17:48:04.894173 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:48:04.894133 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bf5fa36d34bfca02f378ac58c4bce8c4a42dfe1e26e7be0558d66a12f29550e\": container with ID starting with 9bf5fa36d34bfca02f378ac58c4bce8c4a42dfe1e26e7be0558d66a12f29550e not found: ID does not exist" containerID="9bf5fa36d34bfca02f378ac58c4bce8c4a42dfe1e26e7be0558d66a12f29550e" Jul 10 17:48:04.894244 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.894168 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bf5fa36d34bfca02f378ac58c4bce8c4a42dfe1e26e7be0558d66a12f29550e"} err="failed to get container status \"9bf5fa36d34bfca02f378ac58c4bce8c4a42dfe1e26e7be0558d66a12f29550e\": rpc error: code = NotFound desc = could not find container \"9bf5fa36d34bfca02f378ac58c4bce8c4a42dfe1e26e7be0558d66a12f29550e\": container with ID starting with 9bf5fa36d34bfca02f378ac58c4bce8c4a42dfe1e26e7be0558d66a12f29550e not found: ID does not exist" Jul 10 17:48:04.899978 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.899956 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc"] Jul 10 17:48:04.904739 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:04.904716 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ad332c-openshift-default-869f49ff8f-mpptc"] Jul 10 17:48:06.206026 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:06.205992 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7479bc51-6333-4836-b528-6338bcb3b62c" path="/var/lib/kubelet/pods/7479bc51-6333-4836-b528-6338bcb3b62c/volumes" Jul 10 17:48:08.172495 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.172454 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq"] Jul 10 17:48:08.173016 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.172995 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86d04545-4836-436b-9834-f163427b2a72" containerName="revoke-keys" Jul 10 17:48:08.173016 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.173015 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="86d04545-4836-436b-9834-f163427b2a72" containerName="revoke-keys" Jul 10 17:48:08.173209 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.173036 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7479bc51-6333-4836-b528-6338bcb3b62c" containerName="istio-proxy" Jul 10 17:48:08.173209 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.173042 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7479bc51-6333-4836-b528-6338bcb3b62c" containerName="istio-proxy" Jul 10 17:48:08.173209 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.173053 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7479bc51-6333-4836-b528-6338bcb3b62c" containerName="istio-proxy" Jul 10 17:48:08.173209 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.173059 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7479bc51-6333-4836-b528-6338bcb3b62c" containerName="istio-proxy" Jul 10 17:48:08.173209 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.173134 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="86d04545-4836-436b-9834-f163427b2a72" containerName="revoke-keys" Jul 10 17:48:08.173209 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.173144 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7479bc51-6333-4836-b528-6338bcb3b62c" containerName="istio-proxy" Jul 10 17:48:08.177910 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.177889 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:08.180305 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.180283 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-9f66f0-openshift-default-dockercfg-ltr2n\"" Jul 10 17:48:08.193053 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.190604 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq"] Jul 10 17:48:08.323052 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.323013 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3987cf24-27e4-4644-8b76-1afa294f218b-credential-socket\") pod \"e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:08.323222 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.323072 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3987cf24-27e4-4644-8b76-1afa294f218b-istio-envoy\") pod \"e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:08.323222 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.323126 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3987cf24-27e4-4644-8b76-1afa294f218b-istiod-ca-cert\") pod \"e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:08.323222 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.323157 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrd8k\" (UniqueName: \"kubernetes.io/projected/3987cf24-27e4-4644-8b76-1afa294f218b-kube-api-access-nrd8k\") pod \"e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:08.323374 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.323237 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3987cf24-27e4-4644-8b76-1afa294f218b-istio-podinfo\") pod \"e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:08.323374 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.323274 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3987cf24-27e4-4644-8b76-1afa294f218b-istio-data\") pod \"e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:08.323374 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.323333 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3987cf24-27e4-4644-8b76-1afa294f218b-workload-certs\") pod \"e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:08.323374 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.323349 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3987cf24-27e4-4644-8b76-1afa294f218b-istio-token\") pod \"e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:08.323374 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.323370 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3987cf24-27e4-4644-8b76-1afa294f218b-workload-socket\") pod \"e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:08.424075 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.423973 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3987cf24-27e4-4644-8b76-1afa294f218b-credential-socket\") pod \"e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:08.424075 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.424029 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3987cf24-27e4-4644-8b76-1afa294f218b-istio-envoy\") pod \"e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:08.424075 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.424059 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3987cf24-27e4-4644-8b76-1afa294f218b-istiod-ca-cert\") pod \"e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:08.424366 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.424089 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrd8k\" (UniqueName: \"kubernetes.io/projected/3987cf24-27e4-4644-8b76-1afa294f218b-kube-api-access-nrd8k\") pod \"e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:08.424366 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.424111 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3987cf24-27e4-4644-8b76-1afa294f218b-istio-podinfo\") pod \"e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:08.424366 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.424133 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3987cf24-27e4-4644-8b76-1afa294f218b-istio-data\") pod \"e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:08.424366 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.424177 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3987cf24-27e4-4644-8b76-1afa294f218b-workload-certs\") pod \"e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:08.424366 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.424202 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3987cf24-27e4-4644-8b76-1afa294f218b-istio-token\") pod \"e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:08.424366 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.424246 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3987cf24-27e4-4644-8b76-1afa294f218b-workload-socket\") pod \"e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:08.424708 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.424497 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3987cf24-27e4-4644-8b76-1afa294f218b-credential-socket\") pod \"e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:08.424708 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.424511 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3987cf24-27e4-4644-8b76-1afa294f218b-istio-data\") pod \"e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:08.424708 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.424589 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3987cf24-27e4-4644-8b76-1afa294f218b-workload-certs\") pod \"e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:08.424708 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.424627 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3987cf24-27e4-4644-8b76-1afa294f218b-workload-socket\") pod \"e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:08.425089 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.425057 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3987cf24-27e4-4644-8b76-1afa294f218b-istiod-ca-cert\") pod \"e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:08.427479 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.427451 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3987cf24-27e4-4644-8b76-1afa294f218b-istio-envoy\") pod \"e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:08.427834 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.427805 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3987cf24-27e4-4644-8b76-1afa294f218b-istio-podinfo\") pod \"e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:08.433550 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.433527 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3987cf24-27e4-4644-8b76-1afa294f218b-istio-token\") pod \"e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:08.433797 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.433774 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrd8k\" (UniqueName: \"kubernetes.io/projected/3987cf24-27e4-4644-8b76-1afa294f218b-kube-api-access-nrd8k\") pod \"e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:08.490278 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.490251 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:08.655310 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.655281 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq"] Jul 10 17:48:08.656743 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:48:08.656714 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3987cf24_27e4_4644_8b76_1afa294f218b.slice/crio-87434403954abb20d95f727da82b71ad23ea12e428f4e3ebbf159cd3e7f75488 WatchSource:0}: Error finding container 87434403954abb20d95f727da82b71ad23ea12e428f4e3ebbf159cd3e7f75488: Status 404 returned error can't find the container with id 87434403954abb20d95f727da82b71ad23ea12e428f4e3ebbf159cd3e7f75488 Jul 10 17:48:08.659139 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.659109 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:48:08.659220 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.659198 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:48:08.659283 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.659229 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:48:08.902260 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.902209 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" event={"ID":"3987cf24-27e4-4644-8b76-1afa294f218b","Type":"ContainerStarted","Data":"a3a77dc163b4de66bfc0f10620033435c74c1b8c313399e9400833c488fa9a5c"} Jul 10 17:48:08.902471 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.902287 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" event={"ID":"3987cf24-27e4-4644-8b76-1afa294f218b","Type":"ContainerStarted","Data":"87434403954abb20d95f727da82b71ad23ea12e428f4e3ebbf159cd3e7f75488"} Jul 10 17:48:08.922038 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:08.921968 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" podStartSLOduration=0.921945949 podStartE2EDuration="921.945949ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:48:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:48:08.921740073 +0000 UTC m=+2221.249437613" watchObservedRunningTime="2026-07-10 17:48:08.921945949 +0000 UTC m=+2221.249643487" Jul 10 17:48:09.491202 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:09.491166 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:09.497256 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:09.497229 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:09.908409 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:09.908288 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:09.909234 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:09.909215 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:23.996759 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:23.996724 2583 generic.go:356] "Generic (PLEG): container finished" podID="3987cf24-27e4-4644-8b76-1afa294f218b" containerID="a3a77dc163b4de66bfc0f10620033435c74c1b8c313399e9400833c488fa9a5c" exitCode=0 Jul 10 17:48:23.997472 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:23.996803 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" event={"ID":"3987cf24-27e4-4644-8b76-1afa294f218b","Type":"ContainerDied","Data":"a3a77dc163b4de66bfc0f10620033435c74c1b8c313399e9400833c488fa9a5c"} Jul 10 17:48:23.997472 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:23.997311 2583 scope.go:117] "RemoveContainer" containerID="a3a77dc163b4de66bfc0f10620033435c74c1b8c313399e9400833c488fa9a5c" Jul 10 17:48:23.998512 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:23.998483 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:48:23.998599 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:23.998571 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:48:23.998663 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:23.998619 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 10 17:48:24.909330 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:24.909259 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:25.007692 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:25.007642 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" event={"ID":"3987cf24-27e4-4644-8b76-1afa294f218b","Type":"ContainerStarted","Data":"54fc40af10fb4814b0b6a2a4c336bfba67e630e6c2defd84d0e56ef4e9149168"} Jul 10 17:48:25.490880 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:25.490838 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:25.492020 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:25.492000 2583 patch_prober.go:28] interesting pod/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 10 17:48:25.492110 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:25.492049 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" podUID="3987cf24-27e4-4644-8b76-1afa294f218b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 10 17:48:26.491129 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:26.491091 2583 patch_prober.go:28] interesting pod/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 10 17:48:26.491646 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:26.491155 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" podUID="3987cf24-27e4-4644-8b76-1afa294f218b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 10 17:48:27.490954 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:27.490915 2583 patch_prober.go:28] interesting pod/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 10 17:48:27.491138 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:27.490975 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" podUID="3987cf24-27e4-4644-8b76-1afa294f218b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 10 17:48:28.491628 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:28.491591 2583 patch_prober.go:28] interesting pod/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 10 17:48:28.492028 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:28.491661 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" podUID="3987cf24-27e4-4644-8b76-1afa294f218b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 10 17:48:29.495050 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:29.495020 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:29.495533 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:29.495514 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:48:29.496319 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:48:29.496301 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:51:08.317048 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:51:08.317017 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js69w_e41baaf7-4171-4df1-8bdd-d0697a61fdce/ovn-acl-logging/0.log" Jul 10 17:51:08.322963 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:51:08.322942 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js69w_e41baaf7-4171-4df1-8bdd-d0697a61fdce/ovn-acl-logging/0.log" Jul 10 17:51:08.417860 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:51:08.417832 2583 scope.go:117] "RemoveContainer" containerID="30e98d11d09f937763b8c7484abbc03cd8a45eec6ca0544313897d9b8e5fb61a" Jul 10 17:51:08.426607 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:51:08.426586 2583 scope.go:117] "RemoveContainer" containerID="f6064619793d25c3cc3f27027d8472eb3851cb6faa66b0864e15771bae063fc4" Jul 10 17:53:51.242481 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:51.242450 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7417-140566968716592"] Jul 10 17:53:51.244787 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:51.242882 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7479bc51-6333-4836-b528-6338bcb3b62c" containerName="istio-proxy" Jul 10 17:53:51.245613 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:51.245598 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7417-140566968716592" Jul 10 17:53:51.247994 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:51.247974 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-79p6b\"" Jul 10 17:53:51.251860 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:51.251839 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7417-140566968716592"] Jul 10 17:53:51.360112 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:51.360079 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k446t\" (UniqueName: \"kubernetes.io/projected/022ed4d9-fbfc-4516-8be4-a6aa857ec9c2-kube-api-access-k446t\") pod \"test-curl-7417-140566968716592\" (UID: \"022ed4d9-fbfc-4516-8be4-a6aa857ec9c2\") " pod="openshift-ingress/test-curl-7417-140566968716592" Jul 10 17:53:51.460942 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:51.460909 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k446t\" (UniqueName: \"kubernetes.io/projected/022ed4d9-fbfc-4516-8be4-a6aa857ec9c2-kube-api-access-k446t\") pod \"test-curl-7417-140566968716592\" (UID: \"022ed4d9-fbfc-4516-8be4-a6aa857ec9c2\") " pod="openshift-ingress/test-curl-7417-140566968716592" Jul 10 17:53:51.469159 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:51.469135 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k446t\" (UniqueName: \"kubernetes.io/projected/022ed4d9-fbfc-4516-8be4-a6aa857ec9c2-kube-api-access-k446t\") pod \"test-curl-7417-140566968716592\" (UID: \"022ed4d9-fbfc-4516-8be4-a6aa857ec9c2\") " pod="openshift-ingress/test-curl-7417-140566968716592" Jul 10 17:53:51.555895 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:51.555815 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7417-140566968716592" Jul 10 17:53:51.694212 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:51.694186 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7417-140566968716592"] Jul 10 17:53:51.696006 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:53:51.695975 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod022ed4d9_fbfc_4516_8be4_a6aa857ec9c2.slice/crio-7d54fe6b72424c488d70807a26aa3e1bfb695b5fd06a691091330a9967b019cf WatchSource:0}: Error finding container 7d54fe6b72424c488d70807a26aa3e1bfb695b5fd06a691091330a9967b019cf: Status 404 returned error can't find the container with id 7d54fe6b72424c488d70807a26aa3e1bfb695b5fd06a691091330a9967b019cf Jul 10 17:53:51.697285 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:51.697268 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 17:53:51.878462 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:51.878362 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7417-140566968716592" event={"ID":"022ed4d9-fbfc-4516-8be4-a6aa857ec9c2","Type":"ContainerStarted","Data":"7d54fe6b72424c488d70807a26aa3e1bfb695b5fd06a691091330a9967b019cf"} Jul 10 17:53:52.885593 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:52.885495 2583 generic.go:356] "Generic (PLEG): container finished" podID="022ed4d9-fbfc-4516-8be4-a6aa857ec9c2" containerID="1566a7beb2e37e48d7f412847fccd48fc4f49c87cf3240a5a1a7f94b16b6aa0e" exitCode=0 Jul 10 17:53:52.885593 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:52.885570 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7417-140566968716592" event={"ID":"022ed4d9-fbfc-4516-8be4-a6aa857ec9c2","Type":"ContainerDied","Data":"1566a7beb2e37e48d7f412847fccd48fc4f49c87cf3240a5a1a7f94b16b6aa0e"} Jul 10 17:53:54.037653 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:54.037627 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7417-140566968716592" Jul 10 17:53:54.073061 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:54.073024 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7417-140566968716592_022ed4d9-fbfc-4516-8be4-a6aa857ec9c2/test-curl-7417-140566968716592/0.log" Jul 10 17:53:54.095936 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:54.095900 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7417-140566968716592"] Jul 10 17:53:54.099652 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:54.099628 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7417-140566968716592"] Jul 10 17:53:54.185384 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:54.185351 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k446t\" (UniqueName: \"kubernetes.io/projected/022ed4d9-fbfc-4516-8be4-a6aa857ec9c2-kube-api-access-k446t\") pod \"022ed4d9-fbfc-4516-8be4-a6aa857ec9c2\" (UID: \"022ed4d9-fbfc-4516-8be4-a6aa857ec9c2\") " Jul 10 17:53:54.187680 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:54.187655 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/022ed4d9-fbfc-4516-8be4-a6aa857ec9c2-kube-api-access-k446t" (OuterVolumeSpecName: "kube-api-access-k446t") pod "022ed4d9-fbfc-4516-8be4-a6aa857ec9c2" (UID: "022ed4d9-fbfc-4516-8be4-a6aa857ec9c2"). InnerVolumeSpecName "kube-api-access-k446t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:53:54.206995 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:54.206967 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="022ed4d9-fbfc-4516-8be4-a6aa857ec9c2" path="/var/lib/kubelet/pods/022ed4d9-fbfc-4516-8be4-a6aa857ec9c2/volumes" Jul 10 17:53:54.218982 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:54.218955 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7417-140566968862896"] Jul 10 17:53:54.219355 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:54.219343 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="022ed4d9-fbfc-4516-8be4-a6aa857ec9c2" containerName="test-curl-7417-140566968716592" Jul 10 17:53:54.219416 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:54.219357 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="022ed4d9-fbfc-4516-8be4-a6aa857ec9c2" containerName="test-curl-7417-140566968716592" Jul 10 17:53:54.219459 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:54.219441 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="022ed4d9-fbfc-4516-8be4-a6aa857ec9c2" containerName="test-curl-7417-140566968716592" Jul 10 17:53:54.222475 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:54.222458 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7417-140566968862896" Jul 10 17:53:54.227857 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:54.227833 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7417-140566968862896"] Jul 10 17:53:54.286443 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:54.286375 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k446t\" (UniqueName: \"kubernetes.io/projected/022ed4d9-fbfc-4516-8be4-a6aa857ec9c2-kube-api-access-k446t\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:53:54.387714 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:54.387674 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h87r7\" (UniqueName: \"kubernetes.io/projected/cf083aa7-723a-4f1e-9584-c91e24e9e81b-kube-api-access-h87r7\") pod \"test-curl-7417-140566968862896\" (UID: \"cf083aa7-723a-4f1e-9584-c91e24e9e81b\") " pod="openshift-ingress/test-curl-7417-140566968862896" Jul 10 17:53:54.488668 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:54.488573 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h87r7\" (UniqueName: \"kubernetes.io/projected/cf083aa7-723a-4f1e-9584-c91e24e9e81b-kube-api-access-h87r7\") pod \"test-curl-7417-140566968862896\" (UID: \"cf083aa7-723a-4f1e-9584-c91e24e9e81b\") " pod="openshift-ingress/test-curl-7417-140566968862896" Jul 10 17:53:54.498379 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:54.498345 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h87r7\" (UniqueName: \"kubernetes.io/projected/cf083aa7-723a-4f1e-9584-c91e24e9e81b-kube-api-access-h87r7\") pod \"test-curl-7417-140566968862896\" (UID: \"cf083aa7-723a-4f1e-9584-c91e24e9e81b\") " pod="openshift-ingress/test-curl-7417-140566968862896" Jul 10 17:53:54.532223 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:54.532193 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7417-140566968862896" Jul 10 17:53:54.676414 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:54.676354 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7417-140566968862896"] Jul 10 17:53:54.679824 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:53:54.679795 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf083aa7_723a_4f1e_9584_c91e24e9e81b.slice/crio-edba15c4113d3baea6c64dd564a974f80ed6e6b671bdb5267759109e75095c54 WatchSource:0}: Error finding container edba15c4113d3baea6c64dd564a974f80ed6e6b671bdb5267759109e75095c54: Status 404 returned error can't find the container with id edba15c4113d3baea6c64dd564a974f80ed6e6b671bdb5267759109e75095c54 Jul 10 17:53:54.899712 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:54.899601 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7417-140566968862896" event={"ID":"cf083aa7-723a-4f1e-9584-c91e24e9e81b","Type":"ContainerStarted","Data":"edba15c4113d3baea6c64dd564a974f80ed6e6b671bdb5267759109e75095c54"} Jul 10 17:53:54.901092 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:54.901069 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7417-140566968716592" Jul 10 17:53:54.901218 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:54.901096 2583 scope.go:117] "RemoveContainer" containerID="1566a7beb2e37e48d7f412847fccd48fc4f49c87cf3240a5a1a7f94b16b6aa0e" Jul 10 17:53:55.909093 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:55.909058 2583 generic.go:356] "Generic (PLEG): container finished" podID="cf083aa7-723a-4f1e-9584-c91e24e9e81b" containerID="9de929d5dfeb128a34f7e001d32c02196604c295a3ef4ff066b3cbd74348a5fc" exitCode=0 Jul 10 17:53:55.909564 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:55.909116 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7417-140566968862896" event={"ID":"cf083aa7-723a-4f1e-9584-c91e24e9e81b","Type":"ContainerDied","Data":"9de929d5dfeb128a34f7e001d32c02196604c295a3ef4ff066b3cbd74348a5fc"} Jul 10 17:53:57.059074 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:57.059052 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7417-140566968862896" Jul 10 17:53:57.078828 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:57.078791 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7417-140566968862896_cf083aa7-723a-4f1e-9584-c91e24e9e81b/test-curl-7417-140566968862896/0.log" Jul 10 17:53:57.102057 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:57.102024 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7417-140566968862896"] Jul 10 17:53:57.105481 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:57.105452 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7417-140566968862896"] Jul 10 17:53:57.212493 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:57.212381 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h87r7\" (UniqueName: \"kubernetes.io/projected/cf083aa7-723a-4f1e-9584-c91e24e9e81b-kube-api-access-h87r7\") pod \"cf083aa7-723a-4f1e-9584-c91e24e9e81b\" (UID: \"cf083aa7-723a-4f1e-9584-c91e24e9e81b\") " Jul 10 17:53:57.214615 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:57.214585 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf083aa7-723a-4f1e-9584-c91e24e9e81b-kube-api-access-h87r7" (OuterVolumeSpecName: "kube-api-access-h87r7") pod "cf083aa7-723a-4f1e-9584-c91e24e9e81b" (UID: "cf083aa7-723a-4f1e-9584-c91e24e9e81b"). InnerVolumeSpecName "kube-api-access-h87r7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:53:57.268892 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:57.268844 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7417-140566968841824"] Jul 10 17:53:57.269259 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:57.269244 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf083aa7-723a-4f1e-9584-c91e24e9e81b" containerName="test-curl-7417-140566968862896" Jul 10 17:53:57.269319 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:57.269260 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf083aa7-723a-4f1e-9584-c91e24e9e81b" containerName="test-curl-7417-140566968862896" Jul 10 17:53:57.269357 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:57.269328 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf083aa7-723a-4f1e-9584-c91e24e9e81b" containerName="test-curl-7417-140566968862896" Jul 10 17:53:57.272254 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:57.272237 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7417-140566968841824" Jul 10 17:53:57.278047 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:57.278016 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7417-140566968841824"] Jul 10 17:53:57.313962 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:57.313892 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h87r7\" (UniqueName: \"kubernetes.io/projected/cf083aa7-723a-4f1e-9584-c91e24e9e81b-kube-api-access-h87r7\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:53:57.414863 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:57.414828 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5lsw\" (UniqueName: \"kubernetes.io/projected/a9fa1d80-0a93-4ef4-ac64-d05a27568367-kube-api-access-f5lsw\") pod \"test-curl-7417-140566968841824\" (UID: \"a9fa1d80-0a93-4ef4-ac64-d05a27568367\") " pod="openshift-ingress/test-curl-7417-140566968841824" Jul 10 17:53:57.515945 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:57.515852 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5lsw\" (UniqueName: \"kubernetes.io/projected/a9fa1d80-0a93-4ef4-ac64-d05a27568367-kube-api-access-f5lsw\") pod \"test-curl-7417-140566968841824\" (UID: \"a9fa1d80-0a93-4ef4-ac64-d05a27568367\") " pod="openshift-ingress/test-curl-7417-140566968841824" Jul 10 17:53:57.523486 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:57.523433 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5lsw\" (UniqueName: \"kubernetes.io/projected/a9fa1d80-0a93-4ef4-ac64-d05a27568367-kube-api-access-f5lsw\") pod \"test-curl-7417-140566968841824\" (UID: \"a9fa1d80-0a93-4ef4-ac64-d05a27568367\") " pod="openshift-ingress/test-curl-7417-140566968841824" Jul 10 17:53:57.582346 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:57.582291 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7417-140566968841824" Jul 10 17:53:57.718865 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:57.718837 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7417-140566968841824"] Jul 10 17:53:57.720231 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:53:57.720199 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9fa1d80_0a93_4ef4_ac64_d05a27568367.slice/crio-8e8cbba191c072fec685efd78fa1eef71d9ad1ab4b3ca7d799c23a19efae7640 WatchSource:0}: Error finding container 8e8cbba191c072fec685efd78fa1eef71d9ad1ab4b3ca7d799c23a19efae7640: Status 404 returned error can't find the container with id 8e8cbba191c072fec685efd78fa1eef71d9ad1ab4b3ca7d799c23a19efae7640 Jul 10 17:53:57.922597 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:57.922565 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7417-140566968862896" Jul 10 17:53:57.922802 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:57.922608 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edba15c4113d3baea6c64dd564a974f80ed6e6b671bdb5267759109e75095c54" Jul 10 17:53:57.924084 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:57.924060 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7417-140566968841824" event={"ID":"a9fa1d80-0a93-4ef4-ac64-d05a27568367","Type":"ContainerStarted","Data":"8e8cbba191c072fec685efd78fa1eef71d9ad1ab4b3ca7d799c23a19efae7640"} Jul 10 17:53:57.932790 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:53:57.932761 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7417-140566968862896\" is forbidden: User \"system:node:ip-10-0-134-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-134-65.ec2.internal' and this object" podUID="cf083aa7-723a-4f1e-9584-c91e24e9e81b" pod="openshift-ingress/test-curl-7417-140566968862896" Jul 10 17:53:58.205813 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:53:58.205722 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7417-140566968862896\" is forbidden: User \"system:node:ip-10-0-134-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-134-65.ec2.internal' and this object" podUID="cf083aa7-723a-4f1e-9584-c91e24e9e81b" pod="openshift-ingress/test-curl-7417-140566968862896" Jul 10 17:53:58.207342 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:58.207320 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf083aa7-723a-4f1e-9584-c91e24e9e81b" path="/var/lib/kubelet/pods/cf083aa7-723a-4f1e-9584-c91e24e9e81b/volumes" Jul 10 17:53:58.939720 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:58.939680 2583 generic.go:356] "Generic (PLEG): container finished" podID="a9fa1d80-0a93-4ef4-ac64-d05a27568367" containerID="bf8533b3bd4cc7dff9c1bb86e9a47e49795720ae4eb9ea7f2e0820f7148d5166" exitCode=0 Jul 10 17:53:58.939949 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:53:58.939758 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7417-140566968841824" event={"ID":"a9fa1d80-0a93-4ef4-ac64-d05a27568367","Type":"ContainerDied","Data":"bf8533b3bd4cc7dff9c1bb86e9a47e49795720ae4eb9ea7f2e0820f7148d5166"} Jul 10 17:54:00.093339 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:00.093314 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7417-140566968841824" Jul 10 17:54:00.112530 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:00.112502 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7417-140566968841824_a9fa1d80-0a93-4ef4-ac64-d05a27568367/test-curl-7417-140566968841824/0.log" Jul 10 17:54:00.134147 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:00.134114 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7417-140566968841824"] Jul 10 17:54:00.138083 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:00.138060 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7417-140566968841824"] Jul 10 17:54:00.243255 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:00.243167 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5lsw\" (UniqueName: \"kubernetes.io/projected/a9fa1d80-0a93-4ef4-ac64-d05a27568367-kube-api-access-f5lsw\") pod \"a9fa1d80-0a93-4ef4-ac64-d05a27568367\" (UID: \"a9fa1d80-0a93-4ef4-ac64-d05a27568367\") " Jul 10 17:54:00.245575 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:00.245548 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9fa1d80-0a93-4ef4-ac64-d05a27568367-kube-api-access-f5lsw" (OuterVolumeSpecName: "kube-api-access-f5lsw") pod "a9fa1d80-0a93-4ef4-ac64-d05a27568367" (UID: "a9fa1d80-0a93-4ef4-ac64-d05a27568367"). InnerVolumeSpecName "kube-api-access-f5lsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:54:00.253492 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:00.253470 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7417-140566968841824"] Jul 10 17:54:00.253846 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:00.253834 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9fa1d80-0a93-4ef4-ac64-d05a27568367" containerName="test-curl-7417-140566968841824" Jul 10 17:54:00.253888 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:00.253848 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9fa1d80-0a93-4ef4-ac64-d05a27568367" containerName="test-curl-7417-140566968841824" Jul 10 17:54:00.253930 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:00.253921 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9fa1d80-0a93-4ef4-ac64-d05a27568367" containerName="test-curl-7417-140566968841824" Jul 10 17:54:00.256946 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:00.256931 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7417-140566968841824" Jul 10 17:54:00.261576 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:00.261548 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7417-140566968841824"] Jul 10 17:54:00.347907 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:00.347859 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffg7d\" (UniqueName: \"kubernetes.io/projected/5c361a74-61e8-4c0b-8e56-ea80f65e22a0-kube-api-access-ffg7d\") pod \"test-curl-7417-140566968841824\" (UID: \"5c361a74-61e8-4c0b-8e56-ea80f65e22a0\") " pod="openshift-ingress/test-curl-7417-140566968841824" Jul 10 17:54:00.348321 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:00.348243 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5lsw\" (UniqueName: \"kubernetes.io/projected/a9fa1d80-0a93-4ef4-ac64-d05a27568367-kube-api-access-f5lsw\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:54:00.449194 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:00.449156 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffg7d\" (UniqueName: \"kubernetes.io/projected/5c361a74-61e8-4c0b-8e56-ea80f65e22a0-kube-api-access-ffg7d\") pod \"test-curl-7417-140566968841824\" (UID: \"5c361a74-61e8-4c0b-8e56-ea80f65e22a0\") " pod="openshift-ingress/test-curl-7417-140566968841824" Jul 10 17:54:00.456980 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:00.456952 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffg7d\" (UniqueName: \"kubernetes.io/projected/5c361a74-61e8-4c0b-8e56-ea80f65e22a0-kube-api-access-ffg7d\") pod \"test-curl-7417-140566968841824\" (UID: \"5c361a74-61e8-4c0b-8e56-ea80f65e22a0\") " pod="openshift-ingress/test-curl-7417-140566968841824" Jul 10 17:54:00.568185 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:00.568092 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7417-140566968841824" Jul 10 17:54:00.704432 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:00.704404 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7417-140566968841824"] Jul 10 17:54:00.706061 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:54:00.706035 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c361a74_61e8_4c0b_8e56_ea80f65e22a0.slice/crio-5ef1fb34e9c15b282e4b600629ecb240ff5abac4cde3dc3bd7c6f03d26fba022 WatchSource:0}: Error finding container 5ef1fb34e9c15b282e4b600629ecb240ff5abac4cde3dc3bd7c6f03d26fba022: Status 404 returned error can't find the container with id 5ef1fb34e9c15b282e4b600629ecb240ff5abac4cde3dc3bd7c6f03d26fba022 Jul 10 17:54:00.953574 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:00.953546 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7417-140566968841824" Jul 10 17:54:00.953756 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:00.953552 2583 scope.go:117] "RemoveContainer" containerID="bf8533b3bd4cc7dff9c1bb86e9a47e49795720ae4eb9ea7f2e0820f7148d5166" Jul 10 17:54:00.954846 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:00.954755 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7417-140566968841824" event={"ID":"5c361a74-61e8-4c0b-8e56-ea80f65e22a0","Type":"ContainerStarted","Data":"5ef1fb34e9c15b282e4b600629ecb240ff5abac4cde3dc3bd7c6f03d26fba022"} Jul 10 17:54:01.964142 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:01.964107 2583 generic.go:356] "Generic (PLEG): container finished" podID="5c361a74-61e8-4c0b-8e56-ea80f65e22a0" containerID="cf4677bac5e7b2adaddce35391d45d592d451f69b2b135ce5b3cde58f166a697" exitCode=0 Jul 10 17:54:01.964576 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:01.964189 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7417-140566968841824" event={"ID":"5c361a74-61e8-4c0b-8e56-ea80f65e22a0","Type":"ContainerDied","Data":"cf4677bac5e7b2adaddce35391d45d592d451f69b2b135ce5b3cde58f166a697"} Jul 10 17:54:02.206992 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:02.206959 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9fa1d80-0a93-4ef4-ac64-d05a27568367" path="/var/lib/kubelet/pods/a9fa1d80-0a93-4ef4-ac64-d05a27568367/volumes" Jul 10 17:54:03.119108 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:03.119087 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7417-140566968841824" Jul 10 17:54:03.137142 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:03.137121 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7417-140566968841824_5c361a74-61e8-4c0b-8e56-ea80f65e22a0/test-curl-7417-140566968841824/0.log" Jul 10 17:54:03.159677 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:03.159655 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7417-140566968841824"] Jul 10 17:54:03.161986 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:03.161968 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7417-140566968841824"] Jul 10 17:54:03.272433 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:03.272339 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7417-140566968842832"] Jul 10 17:54:03.272740 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:03.272727 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c361a74-61e8-4c0b-8e56-ea80f65e22a0" containerName="test-curl-7417-140566968841824" Jul 10 17:54:03.272790 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:03.272741 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c361a74-61e8-4c0b-8e56-ea80f65e22a0" containerName="test-curl-7417-140566968841824" Jul 10 17:54:03.272827 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:03.272815 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c361a74-61e8-4c0b-8e56-ea80f65e22a0" containerName="test-curl-7417-140566968841824" Jul 10 17:54:03.275739 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:03.275721 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7417-140566968842832" Jul 10 17:54:03.276122 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:03.276104 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffg7d\" (UniqueName: \"kubernetes.io/projected/5c361a74-61e8-4c0b-8e56-ea80f65e22a0-kube-api-access-ffg7d\") pod \"5c361a74-61e8-4c0b-8e56-ea80f65e22a0\" (UID: \"5c361a74-61e8-4c0b-8e56-ea80f65e22a0\") " Jul 10 17:54:03.278682 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:03.278643 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c361a74-61e8-4c0b-8e56-ea80f65e22a0-kube-api-access-ffg7d" (OuterVolumeSpecName: "kube-api-access-ffg7d") pod "5c361a74-61e8-4c0b-8e56-ea80f65e22a0" (UID: "5c361a74-61e8-4c0b-8e56-ea80f65e22a0"). InnerVolumeSpecName "kube-api-access-ffg7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:54:03.281846 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:03.281824 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7417-140566968842832"] Jul 10 17:54:03.376768 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:03.376744 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwmsk\" (UniqueName: \"kubernetes.io/projected/c87d9ae5-e4a5-49a8-8141-4836e174ef2c-kube-api-access-qwmsk\") pod \"test-curl-7417-140566968842832\" (UID: \"c87d9ae5-e4a5-49a8-8141-4836e174ef2c\") " pod="openshift-ingress/test-curl-7417-140566968842832" Jul 10 17:54:03.376903 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:03.376820 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffg7d\" (UniqueName: \"kubernetes.io/projected/5c361a74-61e8-4c0b-8e56-ea80f65e22a0-kube-api-access-ffg7d\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:54:03.477755 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:03.477720 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwmsk\" (UniqueName: \"kubernetes.io/projected/c87d9ae5-e4a5-49a8-8141-4836e174ef2c-kube-api-access-qwmsk\") pod \"test-curl-7417-140566968842832\" (UID: \"c87d9ae5-e4a5-49a8-8141-4836e174ef2c\") " pod="openshift-ingress/test-curl-7417-140566968842832" Jul 10 17:54:03.485862 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:03.485842 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwmsk\" (UniqueName: \"kubernetes.io/projected/c87d9ae5-e4a5-49a8-8141-4836e174ef2c-kube-api-access-qwmsk\") pod \"test-curl-7417-140566968842832\" (UID: \"c87d9ae5-e4a5-49a8-8141-4836e174ef2c\") " pod="openshift-ingress/test-curl-7417-140566968842832" Jul 10 17:54:03.594030 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:03.593921 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7417-140566968842832" Jul 10 17:54:03.729342 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:03.729314 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7417-140566968842832"] Jul 10 17:54:03.730805 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:54:03.730776 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc87d9ae5_e4a5_49a8_8141_4836e174ef2c.slice/crio-6d620bbb7c52eb5c5c06f65228541d03d525f5f0aff55762ecfac1ebd842a151 WatchSource:0}: Error finding container 6d620bbb7c52eb5c5c06f65228541d03d525f5f0aff55762ecfac1ebd842a151: Status 404 returned error can't find the container with id 6d620bbb7c52eb5c5c06f65228541d03d525f5f0aff55762ecfac1ebd842a151 Jul 10 17:54:03.975449 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:03.975420 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7417-140566968841824" Jul 10 17:54:03.975449 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:03.975447 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ef1fb34e9c15b282e4b600629ecb240ff5abac4cde3dc3bd7c6f03d26fba022" Jul 10 17:54:03.980533 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:03.980506 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7417-140566968842832" event={"ID":"c87d9ae5-e4a5-49a8-8141-4836e174ef2c","Type":"ContainerStarted","Data":"6d620bbb7c52eb5c5c06f65228541d03d525f5f0aff55762ecfac1ebd842a151"} Jul 10 17:54:03.987139 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:54:03.987113 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7417-140566968841824\" is forbidden: User \"system:node:ip-10-0-134-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-134-65.ec2.internal' and this object" podUID="5c361a74-61e8-4c0b-8e56-ea80f65e22a0" pod="openshift-ingress/test-curl-7417-140566968841824" Jul 10 17:54:04.209718 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:04.209688 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c361a74-61e8-4c0b-8e56-ea80f65e22a0" path="/var/lib/kubelet/pods/5c361a74-61e8-4c0b-8e56-ea80f65e22a0/volumes" Jul 10 17:54:04.988295 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:04.988260 2583 generic.go:356] "Generic (PLEG): container finished" podID="c87d9ae5-e4a5-49a8-8141-4836e174ef2c" containerID="6e07b7e8e19511596514fe3304e5bff731812b8e3ad98eb55536b1fb56788668" exitCode=0 Jul 10 17:54:04.988470 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:04.988346 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7417-140566968842832" event={"ID":"c87d9ae5-e4a5-49a8-8141-4836e174ef2c","Type":"ContainerDied","Data":"6e07b7e8e19511596514fe3304e5bff731812b8e3ad98eb55536b1fb56788668"} Jul 10 17:54:06.138362 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:06.138335 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7417-140566968842832" Jul 10 17:54:06.158811 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:06.158782 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7417-140566968842832_c87d9ae5-e4a5-49a8-8141-4836e174ef2c/test-curl-7417-140566968842832/0.log" Jul 10 17:54:06.180737 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:06.180699 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7417-140566968842832"] Jul 10 17:54:06.186605 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:06.186576 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7417-140566968842832"] Jul 10 17:54:06.204145 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:06.204118 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwmsk\" (UniqueName: \"kubernetes.io/projected/c87d9ae5-e4a5-49a8-8141-4836e174ef2c-kube-api-access-qwmsk\") pod \"c87d9ae5-e4a5-49a8-8141-4836e174ef2c\" (UID: \"c87d9ae5-e4a5-49a8-8141-4836e174ef2c\") " Jul 10 17:54:06.206727 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:06.206699 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c87d9ae5-e4a5-49a8-8141-4836e174ef2c-kube-api-access-qwmsk" (OuterVolumeSpecName: "kube-api-access-qwmsk") pod "c87d9ae5-e4a5-49a8-8141-4836e174ef2c" (UID: "c87d9ae5-e4a5-49a8-8141-4836e174ef2c"). InnerVolumeSpecName "kube-api-access-qwmsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:54:06.305276 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:06.305177 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwmsk\" (UniqueName: \"kubernetes.io/projected/c87d9ae5-e4a5-49a8-8141-4836e174ef2c-kube-api-access-qwmsk\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:54:06.315314 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:06.315283 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7417-140566968842832"] Jul 10 17:54:06.315697 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:06.315684 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c87d9ae5-e4a5-49a8-8141-4836e174ef2c" containerName="test-curl-7417-140566968842832" Jul 10 17:54:06.315753 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:06.315699 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c87d9ae5-e4a5-49a8-8141-4836e174ef2c" containerName="test-curl-7417-140566968842832" Jul 10 17:54:06.315787 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:06.315764 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c87d9ae5-e4a5-49a8-8141-4836e174ef2c" containerName="test-curl-7417-140566968842832" Jul 10 17:54:06.318752 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:06.318734 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7417-140566968842832" Jul 10 17:54:06.322237 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:06.322215 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7417-140566968842832"] Jul 10 17:54:06.406082 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:06.406046 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkp4c\" (UniqueName: \"kubernetes.io/projected/331eaaee-851f-4514-b830-5f97ba9fdd6f-kube-api-access-rkp4c\") pod \"test-curl-7417-140566968842832\" (UID: \"331eaaee-851f-4514-b830-5f97ba9fdd6f\") " pod="openshift-ingress/test-curl-7417-140566968842832" Jul 10 17:54:06.507233 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:06.507198 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkp4c\" (UniqueName: \"kubernetes.io/projected/331eaaee-851f-4514-b830-5f97ba9fdd6f-kube-api-access-rkp4c\") pod \"test-curl-7417-140566968842832\" (UID: \"331eaaee-851f-4514-b830-5f97ba9fdd6f\") " pod="openshift-ingress/test-curl-7417-140566968842832" Jul 10 17:54:06.514945 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:06.514912 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkp4c\" (UniqueName: \"kubernetes.io/projected/331eaaee-851f-4514-b830-5f97ba9fdd6f-kube-api-access-rkp4c\") pod \"test-curl-7417-140566968842832\" (UID: \"331eaaee-851f-4514-b830-5f97ba9fdd6f\") " pod="openshift-ingress/test-curl-7417-140566968842832" Jul 10 17:54:06.628742 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:06.628655 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7417-140566968842832" Jul 10 17:54:06.770312 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:06.770288 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7417-140566968842832"] Jul 10 17:54:06.771976 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:54:06.771945 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod331eaaee_851f_4514_b830_5f97ba9fdd6f.slice/crio-e756465268f69f3419554d96bb2de9e1b5c20b96bcbd6af4211c1af5f44fcff7 WatchSource:0}: Error finding container e756465268f69f3419554d96bb2de9e1b5c20b96bcbd6af4211c1af5f44fcff7: Status 404 returned error can't find the container with id e756465268f69f3419554d96bb2de9e1b5c20b96bcbd6af4211c1af5f44fcff7 Jul 10 17:54:07.000809 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:07.000774 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7417-140566968842832" event={"ID":"331eaaee-851f-4514-b830-5f97ba9fdd6f","Type":"ContainerStarted","Data":"e756465268f69f3419554d96bb2de9e1b5c20b96bcbd6af4211c1af5f44fcff7"} Jul 10 17:54:07.002093 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:07.002070 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7417-140566968842832" Jul 10 17:54:07.002093 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:07.002083 2583 scope.go:117] "RemoveContainer" containerID="6e07b7e8e19511596514fe3304e5bff731812b8e3ad98eb55536b1fb56788668" Jul 10 17:54:08.009789 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:08.009753 2583 generic.go:356] "Generic (PLEG): container finished" podID="331eaaee-851f-4514-b830-5f97ba9fdd6f" containerID="b27b78212d232ae25502ba9639a053d87df499f7167c55aee6bf42fbd5eef4a6" exitCode=0 Jul 10 17:54:08.010187 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:08.009838 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7417-140566968842832" event={"ID":"331eaaee-851f-4514-b830-5f97ba9fdd6f","Type":"ContainerDied","Data":"b27b78212d232ae25502ba9639a053d87df499f7167c55aee6bf42fbd5eef4a6"} Jul 10 17:54:08.214238 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:08.214205 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c87d9ae5-e4a5-49a8-8141-4836e174ef2c" path="/var/lib/kubelet/pods/c87d9ae5-e4a5-49a8-8141-4836e174ef2c/volumes" Jul 10 17:54:09.160437 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:09.160411 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7417-140566968842832" Jul 10 17:54:09.179035 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:09.179009 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7417-140566968842832_331eaaee-851f-4514-b830-5f97ba9fdd6f/test-curl-7417-140566968842832/0.log" Jul 10 17:54:09.200211 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:09.200186 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7417-140566968842832"] Jul 10 17:54:09.207511 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:09.207480 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7417-140566968842832"] Jul 10 17:54:09.229427 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:09.229380 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkp4c\" (UniqueName: \"kubernetes.io/projected/331eaaee-851f-4514-b830-5f97ba9fdd6f-kube-api-access-rkp4c\") pod \"331eaaee-851f-4514-b830-5f97ba9fdd6f\" (UID: \"331eaaee-851f-4514-b830-5f97ba9fdd6f\") " Jul 10 17:54:09.231412 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:09.231380 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/331eaaee-851f-4514-b830-5f97ba9fdd6f-kube-api-access-rkp4c" (OuterVolumeSpecName: "kube-api-access-rkp4c") pod "331eaaee-851f-4514-b830-5f97ba9fdd6f" (UID: "331eaaee-851f-4514-b830-5f97ba9fdd6f"). InnerVolumeSpecName "kube-api-access-rkp4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:54:09.330710 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:09.330644 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkp4c\" (UniqueName: \"kubernetes.io/projected/331eaaee-851f-4514-b830-5f97ba9fdd6f-kube-api-access-rkp4c\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:54:09.385674 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:09.385641 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3beb3b-8ntx9"] Jul 10 17:54:09.386049 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:09.386035 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="331eaaee-851f-4514-b830-5f97ba9fdd6f" containerName="test-curl-7417-140566968842832" Jul 10 17:54:09.386100 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:09.386050 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="331eaaee-851f-4514-b830-5f97ba9fdd6f" containerName="test-curl-7417-140566968842832" Jul 10 17:54:09.386133 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:09.386109 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="331eaaee-851f-4514-b830-5f97ba9fdd6f" containerName="test-curl-7417-140566968842832" Jul 10 17:54:09.389261 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:09.389245 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3beb3b-8ntx9" Jul 10 17:54:09.391735 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:09.391711 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m7h25\"" Jul 10 17:54:09.396098 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:09.396075 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3beb3b-8ntx9"] Jul 10 17:54:09.431456 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:09.431426 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2535265d-7b75-4bd3-b23b-7efbfe62cf28-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-3beb3b-8ntx9\" (UID: \"2535265d-7b75-4bd3-b23b-7efbfe62cf28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3beb3b-8ntx9" Jul 10 17:54:09.532424 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:09.532375 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2535265d-7b75-4bd3-b23b-7efbfe62cf28-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-3beb3b-8ntx9\" (UID: \"2535265d-7b75-4bd3-b23b-7efbfe62cf28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3beb3b-8ntx9" Jul 10 17:54:09.533027 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:09.533008 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2535265d-7b75-4bd3-b23b-7efbfe62cf28-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-3beb3b-8ntx9\" (UID: \"2535265d-7b75-4bd3-b23b-7efbfe62cf28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3beb3b-8ntx9" Jul 10 17:54:09.702980 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:09.702943 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3beb3b-8ntx9" Jul 10 17:54:09.842787 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:09.842759 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3beb3b-8ntx9"] Jul 10 17:54:09.844574 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:54:09.844543 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2535265d_7b75_4bd3_b23b_7efbfe62cf28.slice/crio-a9926949e65a81f16f0bb07444933ff810ce609b3d0cad3b9e02ab039650a2b8 WatchSource:0}: Error finding container a9926949e65a81f16f0bb07444933ff810ce609b3d0cad3b9e02ab039650a2b8: Status 404 returned error can't find the container with id a9926949e65a81f16f0bb07444933ff810ce609b3d0cad3b9e02ab039650a2b8 Jul 10 17:54:10.024602 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:10.024578 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e756465268f69f3419554d96bb2de9e1b5c20b96bcbd6af4211c1af5f44fcff7" Jul 10 17:54:10.024602 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:10.024578 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7417-140566968842832" Jul 10 17:54:10.026128 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:10.026104 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3beb3b-8ntx9" event={"ID":"2535265d-7b75-4bd3-b23b-7efbfe62cf28","Type":"ContainerStarted","Data":"a9926949e65a81f16f0bb07444933ff810ce609b3d0cad3b9e02ab039650a2b8"} Jul 10 17:54:10.033984 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:54:10.033957 2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7417-140566968842832\" is forbidden: User \"system:node:ip-10-0-134-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-134-65.ec2.internal' and this object" podUID="331eaaee-851f-4514-b830-5f97ba9fdd6f" pod="openshift-ingress/test-curl-7417-140566968842832" Jul 10 17:54:10.208280 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:10.208246 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="331eaaee-851f-4514-b830-5f97ba9fdd6f" path="/var/lib/kubelet/pods/331eaaee-851f-4514-b830-5f97ba9fdd6f/volumes" Jul 10 17:54:11.033067 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:11.033034 2583 generic.go:356] "Generic (PLEG): container finished" podID="2535265d-7b75-4bd3-b23b-7efbfe62cf28" containerID="2bcc036316f7b442328eb8cc861ffd627acbeff126eda3aed4d6c0428d3e26a3" exitCode=0 Jul 10 17:54:11.033236 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:11.033115 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3beb3b-8ntx9" event={"ID":"2535265d-7b75-4bd3-b23b-7efbfe62cf28","Type":"ContainerDied","Data":"2bcc036316f7b442328eb8cc861ffd627acbeff126eda3aed4d6c0428d3e26a3"} Jul 10 17:54:12.175765 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:12.175741 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3beb3b-8ntx9" Jul 10 17:54:12.255501 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:12.255469 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2535265d-7b75-4bd3-b23b-7efbfe62cf28-maas-api-service-ca\") pod \"2535265d-7b75-4bd3-b23b-7efbfe62cf28\" (UID: \"2535265d-7b75-4bd3-b23b-7efbfe62cf28\") " Jul 10 17:54:12.255841 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:12.255820 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2535265d-7b75-4bd3-b23b-7efbfe62cf28-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2535265d-7b75-4bd3-b23b-7efbfe62cf28" (UID: "2535265d-7b75-4bd3-b23b-7efbfe62cf28"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:54:12.356955 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:12.356853 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2535265d-7b75-4bd3-b23b-7efbfe62cf28-maas-api-service-ca\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:54:13.044846 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:13.044808 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3beb3b-8ntx9" event={"ID":"2535265d-7b75-4bd3-b23b-7efbfe62cf28","Type":"ContainerDied","Data":"a9926949e65a81f16f0bb07444933ff810ce609b3d0cad3b9e02ab039650a2b8"} Jul 10 17:54:13.044846 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:13.044838 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3beb3b-8ntx9" Jul 10 17:54:13.045060 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:13.044859 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9926949e65a81f16f0bb07444933ff810ce609b3d0cad3b9e02ab039650a2b8" Jul 10 17:54:35.472881 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:35.472845 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9f66f0-kks8x"] Jul 10 17:54:35.473407 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:35.473277 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2535265d-7b75-4bd3-b23b-7efbfe62cf28" containerName="revoke-keys" Jul 10 17:54:35.473407 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:35.473290 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2535265d-7b75-4bd3-b23b-7efbfe62cf28" containerName="revoke-keys" Jul 10 17:54:35.473407 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:35.473369 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2535265d-7b75-4bd3-b23b-7efbfe62cf28" containerName="revoke-keys" Jul 10 17:54:35.478408 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:35.478379 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9f66f0-kks8x" Jul 10 17:54:35.481228 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:35.480962 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m7h25\"" Jul 10 17:54:35.483253 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:35.483219 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9f66f0-kks8x"] Jul 10 17:54:35.567000 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:35.566971 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71bc6d15-1251-4b63-b885-eb19f56e3bc5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-9f66f0-kks8x\" (UID: \"71bc6d15-1251-4b63-b885-eb19f56e3bc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9f66f0-kks8x" Jul 10 17:54:35.667803 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:35.667762 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71bc6d15-1251-4b63-b885-eb19f56e3bc5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-9f66f0-kks8x\" (UID: \"71bc6d15-1251-4b63-b885-eb19f56e3bc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9f66f0-kks8x" Jul 10 17:54:35.668486 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:35.668468 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71bc6d15-1251-4b63-b885-eb19f56e3bc5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-9f66f0-kks8x\" (UID: \"71bc6d15-1251-4b63-b885-eb19f56e3bc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9f66f0-kks8x" Jul 10 17:54:35.790335 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:35.790220 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9f66f0-kks8x" Jul 10 17:54:35.927929 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:35.927906 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9f66f0-kks8x"] Jul 10 17:54:35.929985 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:54:35.929952 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71bc6d15_1251_4b63_b885_eb19f56e3bc5.slice/crio-2ba95e621e9179a2a130d62c095803d16fbee5068c02af8fc42dc4fcfce4f90e WatchSource:0}: Error finding container 2ba95e621e9179a2a130d62c095803d16fbee5068c02af8fc42dc4fcfce4f90e: Status 404 returned error can't find the container with id 2ba95e621e9179a2a130d62c095803d16fbee5068c02af8fc42dc4fcfce4f90e Jul 10 17:54:36.180505 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:36.180470 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9f66f0-kks8x" event={"ID":"71bc6d15-1251-4b63-b885-eb19f56e3bc5","Type":"ContainerStarted","Data":"2ba95e621e9179a2a130d62c095803d16fbee5068c02af8fc42dc4fcfce4f90e"} Jul 10 17:54:37.187547 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:37.187510 2583 generic.go:356] "Generic (PLEG): container finished" podID="71bc6d15-1251-4b63-b885-eb19f56e3bc5" containerID="737b9af5cef9cd2c3a2e67a605092035ba7fcdf6d5ab4b1df3396552d3853b97" exitCode=0 Jul 10 17:54:37.187919 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:37.187593 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9f66f0-kks8x" event={"ID":"71bc6d15-1251-4b63-b885-eb19f56e3bc5","Type":"ContainerDied","Data":"737b9af5cef9cd2c3a2e67a605092035ba7fcdf6d5ab4b1df3396552d3853b97"} Jul 10 17:54:38.343836 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:38.343801 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9f66f0-kks8x" Jul 10 17:54:38.494756 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:38.494664 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71bc6d15-1251-4b63-b885-eb19f56e3bc5-maas-api-service-ca\") pod \"71bc6d15-1251-4b63-b885-eb19f56e3bc5\" (UID: \"71bc6d15-1251-4b63-b885-eb19f56e3bc5\") " Jul 10 17:54:38.495103 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:38.495076 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71bc6d15-1251-4b63-b885-eb19f56e3bc5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "71bc6d15-1251-4b63-b885-eb19f56e3bc5" (UID: "71bc6d15-1251-4b63-b885-eb19f56e3bc5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:54:38.596070 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:38.596025 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71bc6d15-1251-4b63-b885-eb19f56e3bc5-maas-api-service-ca\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:54:39.201822 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:39.201784 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9f66f0-kks8x" Jul 10 17:54:39.201822 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:39.201814 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9f66f0-kks8x" event={"ID":"71bc6d15-1251-4b63-b885-eb19f56e3bc5","Type":"ContainerDied","Data":"2ba95e621e9179a2a130d62c095803d16fbee5068c02af8fc42dc4fcfce4f90e"} Jul 10 17:54:39.202051 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:54:39.201861 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ba95e621e9179a2a130d62c095803d16fbee5068c02af8fc42dc4fcfce4f90e" Jul 10 17:55:01.239838 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:01.239801 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq"] Jul 10 17:55:01.240266 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:01.240064 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" podUID="3987cf24-27e4-4644-8b76-1afa294f218b" containerName="istio-proxy" containerID="cri-o://54fc40af10fb4814b0b6a2a4c336bfba67e630e6c2defd84d0e56ef4e9149168" gracePeriod=30 Jul 10 17:55:06.491525 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:06.491502 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:55:06.541971 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:06.541896 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3987cf24-27e4-4644-8b76-1afa294f218b-istiod-ca-cert\") pod \"3987cf24-27e4-4644-8b76-1afa294f218b\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " Jul 10 17:55:06.541971 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:06.541943 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3987cf24-27e4-4644-8b76-1afa294f218b-istio-data\") pod \"3987cf24-27e4-4644-8b76-1afa294f218b\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " Jul 10 17:55:06.542176 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:06.542006 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3987cf24-27e4-4644-8b76-1afa294f218b-credential-socket\") pod \"3987cf24-27e4-4644-8b76-1afa294f218b\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " Jul 10 17:55:06.542176 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:06.542026 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3987cf24-27e4-4644-8b76-1afa294f218b-workload-certs\") pod \"3987cf24-27e4-4644-8b76-1afa294f218b\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " Jul 10 17:55:06.542176 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:06.542051 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3987cf24-27e4-4644-8b76-1afa294f218b-workload-socket\") pod \"3987cf24-27e4-4644-8b76-1afa294f218b\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " Jul 10 17:55:06.542176 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:06.542067 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3987cf24-27e4-4644-8b76-1afa294f218b-istio-podinfo\") pod \"3987cf24-27e4-4644-8b76-1afa294f218b\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " Jul 10 17:55:06.542176 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:06.542098 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3987cf24-27e4-4644-8b76-1afa294f218b-istio-envoy\") pod \"3987cf24-27e4-4644-8b76-1afa294f218b\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " Jul 10 17:55:06.542176 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:06.542129 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrd8k\" (UniqueName: \"kubernetes.io/projected/3987cf24-27e4-4644-8b76-1afa294f218b-kube-api-access-nrd8k\") pod \"3987cf24-27e4-4644-8b76-1afa294f218b\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " Jul 10 17:55:06.542176 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:06.542155 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3987cf24-27e4-4644-8b76-1afa294f218b-istio-token\") pod \"3987cf24-27e4-4644-8b76-1afa294f218b\" (UID: \"3987cf24-27e4-4644-8b76-1afa294f218b\") " Jul 10 17:55:06.542561 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:06.542309 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3987cf24-27e4-4644-8b76-1afa294f218b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3987cf24-27e4-4644-8b76-1afa294f218b" (UID: "3987cf24-27e4-4644-8b76-1afa294f218b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:55:06.542561 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:06.542340 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3987cf24-27e4-4644-8b76-1afa294f218b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3987cf24-27e4-4644-8b76-1afa294f218b" (UID: "3987cf24-27e4-4644-8b76-1afa294f218b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:55:06.542561 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:06.542352 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3987cf24-27e4-4644-8b76-1afa294f218b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3987cf24-27e4-4644-8b76-1afa294f218b" (UID: "3987cf24-27e4-4644-8b76-1afa294f218b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:55:06.542561 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:06.542363 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3987cf24-27e4-4644-8b76-1afa294f218b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3987cf24-27e4-4644-8b76-1afa294f218b" (UID: "3987cf24-27e4-4644-8b76-1afa294f218b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:55:06.542561 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:06.542523 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3987cf24-27e4-4644-8b76-1afa294f218b-workload-socket\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:55:06.542561 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:06.542544 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3987cf24-27e4-4644-8b76-1afa294f218b-istiod-ca-cert\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:55:06.542561 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:06.542558 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3987cf24-27e4-4644-8b76-1afa294f218b-credential-socket\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:55:06.542891 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:06.542571 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3987cf24-27e4-4644-8b76-1afa294f218b-workload-certs\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:55:06.543252 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:06.543227 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3987cf24-27e4-4644-8b76-1afa294f218b-istio-data" (OuterVolumeSpecName: "istio-data") pod "3987cf24-27e4-4644-8b76-1afa294f218b" (UID: "3987cf24-27e4-4644-8b76-1afa294f218b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:55:06.544541 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:06.544520 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3987cf24-27e4-4644-8b76-1afa294f218b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3987cf24-27e4-4644-8b76-1afa294f218b" (UID: "3987cf24-27e4-4644-8b76-1afa294f218b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:55:06.544541 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:06.544528 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3987cf24-27e4-4644-8b76-1afa294f218b-kube-api-access-nrd8k" (OuterVolumeSpecName: "kube-api-access-nrd8k") pod "3987cf24-27e4-4644-8b76-1afa294f218b" (UID: "3987cf24-27e4-4644-8b76-1afa294f218b"). InnerVolumeSpecName "kube-api-access-nrd8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:55:06.544976 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:06.544957 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3987cf24-27e4-4644-8b76-1afa294f218b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3987cf24-27e4-4644-8b76-1afa294f218b" (UID: "3987cf24-27e4-4644-8b76-1afa294f218b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 17:55:06.545025 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:06.544984 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3987cf24-27e4-4644-8b76-1afa294f218b-istio-token" (OuterVolumeSpecName: "istio-token") pod "3987cf24-27e4-4644-8b76-1afa294f218b" (UID: "3987cf24-27e4-4644-8b76-1afa294f218b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:55:06.642942 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:06.642914 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3987cf24-27e4-4644-8b76-1afa294f218b-istio-data\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:55:06.642942 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:06.642938 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3987cf24-27e4-4644-8b76-1afa294f218b-istio-podinfo\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:55:06.642942 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:06.642946 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3987cf24-27e4-4644-8b76-1afa294f218b-istio-envoy\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:55:06.643162 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:06.642954 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrd8k\" (UniqueName: \"kubernetes.io/projected/3987cf24-27e4-4644-8b76-1afa294f218b-kube-api-access-nrd8k\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:55:06.643162 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:06.642963 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3987cf24-27e4-4644-8b76-1afa294f218b-istio-token\") on node \"ip-10-0-134-65.ec2.internal\" DevicePath \"\"" Jul 10 17:55:06.857904 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:06.857823 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-8htph_9b800003-9484-432d-b004-452a75d4765b/manager/0.log" Jul 10 17:55:06.972094 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:06.972064 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-p7lgl_86283d78-f4af-4cd5-aaab-a075694c5a37/manager/0.log" Jul 10 17:55:07.090491 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:07.090456 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b4655bb55-h9s24_49640ea0-1609-4683-bf14-f14f0bdb5882/manager/0.log" Jul 10 17:55:07.199456 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:07.199428 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-7cxn7_dae95bcd-c4af-4c58-bf11-2aef952610c5/server/0.log" Jul 10 17:55:07.313375 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:07.313344 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-h8qcx_313596b5-3368-4ca3-8f71-0be58e50e7f4/manager/0.log" Jul 10 17:55:07.359445 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:07.359385 2583 generic.go:356] "Generic (PLEG): container finished" podID="3987cf24-27e4-4644-8b76-1afa294f218b" containerID="54fc40af10fb4814b0b6a2a4c336bfba67e630e6c2defd84d0e56ef4e9149168" exitCode=0 Jul 10 17:55:07.359637 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:07.359458 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" Jul 10 17:55:07.359637 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:07.359491 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" event={"ID":"3987cf24-27e4-4644-8b76-1afa294f218b","Type":"ContainerDied","Data":"54fc40af10fb4814b0b6a2a4c336bfba67e630e6c2defd84d0e56ef4e9149168"} Jul 10 17:55:07.359637 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:07.359547 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq" event={"ID":"3987cf24-27e4-4644-8b76-1afa294f218b","Type":"ContainerDied","Data":"87434403954abb20d95f727da82b71ad23ea12e428f4e3ebbf159cd3e7f75488"} Jul 10 17:55:07.359637 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:07.359570 2583 scope.go:117] "RemoveContainer" containerID="54fc40af10fb4814b0b6a2a4c336bfba67e630e6c2defd84d0e56ef4e9149168" Jul 10 17:55:07.369942 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:07.369922 2583 scope.go:117] "RemoveContainer" containerID="a3a77dc163b4de66bfc0f10620033435c74c1b8c313399e9400833c488fa9a5c" Jul 10 17:55:07.378696 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:07.378680 2583 scope.go:117] "RemoveContainer" containerID="54fc40af10fb4814b0b6a2a4c336bfba67e630e6c2defd84d0e56ef4e9149168" Jul 10 17:55:07.378985 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:55:07.378966 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54fc40af10fb4814b0b6a2a4c336bfba67e630e6c2defd84d0e56ef4e9149168\": container with ID starting with 54fc40af10fb4814b0b6a2a4c336bfba67e630e6c2defd84d0e56ef4e9149168 not found: ID does not exist" containerID="54fc40af10fb4814b0b6a2a4c336bfba67e630e6c2defd84d0e56ef4e9149168" Jul 10 17:55:07.379029 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:07.378996 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54fc40af10fb4814b0b6a2a4c336bfba67e630e6c2defd84d0e56ef4e9149168"} err="failed to get container status \"54fc40af10fb4814b0b6a2a4c336bfba67e630e6c2defd84d0e56ef4e9149168\": rpc error: code = NotFound desc = could not find container \"54fc40af10fb4814b0b6a2a4c336bfba67e630e6c2defd84d0e56ef4e9149168\": container with ID starting with 54fc40af10fb4814b0b6a2a4c336bfba67e630e6c2defd84d0e56ef4e9149168 not found: ID does not exist" Jul 10 17:55:07.379029 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:07.379015 2583 scope.go:117] "RemoveContainer" containerID="a3a77dc163b4de66bfc0f10620033435c74c1b8c313399e9400833c488fa9a5c" Jul 10 17:55:07.379261 ip-10-0-134-65 kubenswrapper[2583]: E0710 17:55:07.379239 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3a77dc163b4de66bfc0f10620033435c74c1b8c313399e9400833c488fa9a5c\": container with ID starting with a3a77dc163b4de66bfc0f10620033435c74c1b8c313399e9400833c488fa9a5c not found: ID does not exist" containerID="a3a77dc163b4de66bfc0f10620033435c74c1b8c313399e9400833c488fa9a5c" Jul 10 17:55:07.379322 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:07.379270 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3a77dc163b4de66bfc0f10620033435c74c1b8c313399e9400833c488fa9a5c"} err="failed to get container status \"a3a77dc163b4de66bfc0f10620033435c74c1b8c313399e9400833c488fa9a5c\": rpc error: code = NotFound desc = could not find container \"a3a77dc163b4de66bfc0f10620033435c74c1b8c313399e9400833c488fa9a5c\": container with ID starting with a3a77dc163b4de66bfc0f10620033435c74c1b8c313399e9400833c488fa9a5c not found: ID does not exist" Jul 10 17:55:07.383823 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:07.383801 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq"] Jul 10 17:55:07.387438 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:07.387417 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-9f66f0-openshift-default-68648b8d6c-ts9mq"] Jul 10 17:55:07.534323 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:07.534248 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b884cfc77-g4rrm_c545fd6c-d7a9-4ad4-845c-2504f3cf59fc/copy-manifests/0.log" Jul 10 17:55:07.546711 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:07.546690 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b884cfc77-g4rrm_c545fd6c-d7a9-4ad4-845c-2504f3cf59fc/manager/0.log" Jul 10 17:55:08.207323 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:08.207288 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3987cf24-27e4-4644-8b76-1afa294f218b" path="/var/lib/kubelet/pods/3987cf24-27e4-4644-8b76-1afa294f218b/volumes" Jul 10 17:55:08.258997 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:08.258965 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29728425-tdrsv_9d44ebe9-7b44-44ee-ac3e-a1d56043d74b/cleanup/0.log" Jul 10 17:55:08.470598 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:08.470517 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-dddaaba5-pbxjc_342379b1-29b3-470a-aade-18e6a468989c/revoke-keys/0.log" Jul 10 17:55:08.581431 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:08.581376 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-43bb8e-jj6sf_86d04545-4836-436b-9834-f163427b2a72/revoke-keys/0.log" Jul 10 17:55:08.795053 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:08.794968 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-5d9215e4-5xjrw_d77a1607-4417-46b1-b6e0-0267da87cb51/revoke-keys/0.log" Jul 10 17:55:08.919786 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:08.919763 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-a3eb74-9bxm2_046bbc7c-21a0-41d5-9fe3-ace5b5589eed/revoke-keys/0.log" Jul 10 17:55:09.133981 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:09.133901 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-039c4f53-brp62_c092fe77-2742-4f35-b8dc-41e5a5fb9819/revoke-keys/0.log" Jul 10 17:55:09.237818 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:09.237774 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-c1c07f63-7nbs4_da2a94db-de75-4bab-a128-eb6b5ce0505e/revoke-keys/0.log" Jul 10 17:55:09.351354 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:09.351324 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-3beb3b-8ntx9_2535265d-7b75-4bd3-b23b-7efbfe62cf28/revoke-keys/0.log" Jul 10 17:55:09.455727 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:09.455703 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-9f66f0-kks8x_71bc6d15-1251-4b63-b885-eb19f56e3bc5/revoke-keys/0.log" Jul 10 17:55:09.564759 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:09.564732 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-hvqc8_89e2948a-bd5e-4f18-a615-5ba8aa318e63/postgres/0.log" Jul 10 17:55:10.326044 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:10.326011 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw_d81b6b2e-0d3c-4f29-ae16-361c0b8682a6/util/0.log" Jul 10 17:55:10.332167 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:10.332139 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw_d81b6b2e-0d3c-4f29-ae16-361c0b8682a6/pull/0.log" Jul 10 17:55:10.337714 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:10.337692 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw_d81b6b2e-0d3c-4f29-ae16-361c0b8682a6/extract/0.log" Jul 10 17:55:10.444274 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:10.444241 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s_59ca63e6-8478-4612-814f-44f062905e67/util/0.log" Jul 10 17:55:10.450749 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:10.450730 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s_59ca63e6-8478-4612-814f-44f062905e67/pull/0.log" Jul 10 17:55:10.457111 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:10.457094 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s_59ca63e6-8478-4612-814f-44f062905e67/extract/0.log" Jul 10 17:55:10.566187 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:10.566158 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79_e55f2ebf-0019-408b-bf88-2739a8763f78/util/0.log" Jul 10 17:55:10.572432 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:10.572407 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79_e55f2ebf-0019-408b-bf88-2739a8763f78/pull/0.log" Jul 10 17:55:10.578358 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:10.578312 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79_e55f2ebf-0019-408b-bf88-2739a8763f78/extract/0.log" Jul 10 17:55:10.685715 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:10.685692 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb_8db3a343-675d-40b5-a0a9-5a8a02374227/extract/0.log" Jul 10 17:55:10.691248 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:10.691232 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb_8db3a343-675d-40b5-a0a9-5a8a02374227/util/0.log" Jul 10 17:55:10.696857 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:10.696839 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb_8db3a343-675d-40b5-a0a9-5a8a02374227/pull/0.log" Jul 10 17:55:10.936543 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:10.936514 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-vzrr7_c7b4a154-f747-4cea-889b-cdb3adeb7b13/manager/0.log" Jul 10 17:55:11.049032 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:11.049003 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-b2kmw_fd6d0d14-eced-40cb-bd67-12b7fbd9e539/manager/0.log" Jul 10 17:55:11.271565 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:11.271489 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-rf9pk_88d0b0b6-91bf-4f00-be4c-6c8191e5176e/registry-server/0.log" Jul 10 17:55:11.389104 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:11.389075 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-db25c_e0279bca-c281-4a5e-ab40-f74c854cde9c/manager/0.log" Jul 10 17:55:11.616958 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:11.616880 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-gskkj_79ebdd43-2bce-43f5-a14f-1c3090448a38/manager/0.log" Jul 10 17:55:12.306056 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:12.306029 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-545f68d885-qzl6x_7875067d-0704-4751-9605-5e851f2421a8/kube-auth-proxy/0.log" Jul 10 17:55:12.425690 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:12.425660 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-mlrv6_553d3bf3-9e01-4b25-9ac2-01b0da4faf79/istio-proxy/1.log" Jul 10 17:55:12.538332 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:12.538306 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-qrscc_ee5cf1f6-d727-424a-82b4-4581245dba8c/payload-pre-processing/0.log" Jul 10 17:55:12.645192 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:12.645122 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-6bjhw_c69c3ec1-fa81-4c47-ac65-c478feee0a14/payload-processing/0.log" Jul 10 17:55:13.088714 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:13.088690 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-mw8pl_eb1e1e1c-a404-4c9a-a017-3c9ec4d0fdbb/main/0.log" Jul 10 17:55:13.422985 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:13.422955 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj45bv_5a6ae435-53b4-49d0-9baa-72d31ed73529/main/0.log" Jul 10 17:55:17.231988 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:17.231957 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vnztt/must-gather-rjsl8"] Jul 10 17:55:17.232368 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:17.232359 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71bc6d15-1251-4b63-b885-eb19f56e3bc5" containerName="revoke-keys" Jul 10 17:55:17.232432 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:17.232370 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="71bc6d15-1251-4b63-b885-eb19f56e3bc5" containerName="revoke-keys" Jul 10 17:55:17.232432 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:17.232381 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3987cf24-27e4-4644-8b76-1afa294f218b" containerName="istio-proxy" Jul 10 17:55:17.232432 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:17.232407 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3987cf24-27e4-4644-8b76-1afa294f218b" containerName="istio-proxy" Jul 10 17:55:17.232432 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:17.232421 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3987cf24-27e4-4644-8b76-1afa294f218b" containerName="istio-proxy" Jul 10 17:55:17.232432 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:17.232426 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3987cf24-27e4-4644-8b76-1afa294f218b" containerName="istio-proxy" Jul 10 17:55:17.232590 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:17.232481 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3987cf24-27e4-4644-8b76-1afa294f218b" containerName="istio-proxy" Jul 10 17:55:17.232590 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:17.232491 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="71bc6d15-1251-4b63-b885-eb19f56e3bc5" containerName="revoke-keys" Jul 10 17:55:17.232652 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:17.232604 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3987cf24-27e4-4644-8b76-1afa294f218b" containerName="istio-proxy" Jul 10 17:55:17.235559 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:17.235543 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vnztt/must-gather-rjsl8" Jul 10 17:55:17.238236 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:17.238214 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vnztt\"/\"kube-root-ca.crt\"" Jul 10 17:55:17.239055 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:17.239039 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vnztt\"/\"default-dockercfg-6c996\"" Jul 10 17:55:17.239118 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:17.239054 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vnztt\"/\"openshift-service-ca.crt\"" Jul 10 17:55:17.244427 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:17.244384 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vnztt/must-gather-rjsl8"] Jul 10 17:55:17.336607 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:17.336563 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnnxz\" (UniqueName: \"kubernetes.io/projected/bbd80816-ec47-40ac-8a06-855402c37923-kube-api-access-xnnxz\") pod \"must-gather-rjsl8\" (UID: \"bbd80816-ec47-40ac-8a06-855402c37923\") " pod="openshift-must-gather-vnztt/must-gather-rjsl8" Jul 10 17:55:17.336771 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:17.336635 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bbd80816-ec47-40ac-8a06-855402c37923-must-gather-output\") pod \"must-gather-rjsl8\" (UID: \"bbd80816-ec47-40ac-8a06-855402c37923\") " pod="openshift-must-gather-vnztt/must-gather-rjsl8" Jul 10 17:55:17.437326 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:17.437287 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bbd80816-ec47-40ac-8a06-855402c37923-must-gather-output\") pod \"must-gather-rjsl8\" (UID: \"bbd80816-ec47-40ac-8a06-855402c37923\") " pod="openshift-must-gather-vnztt/must-gather-rjsl8" Jul 10 17:55:17.437563 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:17.437473 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnnxz\" (UniqueName: \"kubernetes.io/projected/bbd80816-ec47-40ac-8a06-855402c37923-kube-api-access-xnnxz\") pod \"must-gather-rjsl8\" (UID: \"bbd80816-ec47-40ac-8a06-855402c37923\") " pod="openshift-must-gather-vnztt/must-gather-rjsl8" Jul 10 17:55:17.437670 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:17.437648 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bbd80816-ec47-40ac-8a06-855402c37923-must-gather-output\") pod \"must-gather-rjsl8\" (UID: \"bbd80816-ec47-40ac-8a06-855402c37923\") " pod="openshift-must-gather-vnztt/must-gather-rjsl8" Jul 10 17:55:17.445957 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:17.445924 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnnxz\" (UniqueName: \"kubernetes.io/projected/bbd80816-ec47-40ac-8a06-855402c37923-kube-api-access-xnnxz\") pod \"must-gather-rjsl8\" (UID: \"bbd80816-ec47-40ac-8a06-855402c37923\") " pod="openshift-must-gather-vnztt/must-gather-rjsl8" Jul 10 17:55:17.545781 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:17.545669 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vnztt/must-gather-rjsl8" Jul 10 17:55:17.685613 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:17.685578 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vnztt/must-gather-rjsl8"] Jul 10 17:55:17.689074 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:55:17.689044 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbd80816_ec47_40ac_8a06_855402c37923.slice/crio-68dc950cadd753d3a583eb9fe69c1b382ae14432f2aa5448200259fd9edfb368 WatchSource:0}: Error finding container 68dc950cadd753d3a583eb9fe69c1b382ae14432f2aa5448200259fd9edfb368: Status 404 returned error can't find the container with id 68dc950cadd753d3a583eb9fe69c1b382ae14432f2aa5448200259fd9edfb368 Jul 10 17:55:18.424040 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:18.424002 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vnztt/must-gather-rjsl8" event={"ID":"bbd80816-ec47-40ac-8a06-855402c37923","Type":"ContainerStarted","Data":"3c4f731523aa252cbcd89e86c8abe8cd99ec4c165aba6603d0d8ac0079870889"} Jul 10 17:55:18.424040 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:18.424042 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vnztt/must-gather-rjsl8" event={"ID":"bbd80816-ec47-40ac-8a06-855402c37923","Type":"ContainerStarted","Data":"68dc950cadd753d3a583eb9fe69c1b382ae14432f2aa5448200259fd9edfb368"} Jul 10 17:55:19.435125 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:19.435081 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vnztt/must-gather-rjsl8" event={"ID":"bbd80816-ec47-40ac-8a06-855402c37923","Type":"ContainerStarted","Data":"0470b2d68a6b758a7a71f7086a6252ce85b15ee02b48d3cdbd513543a3eb6b03"} Jul 10 17:55:19.454935 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:19.454886 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vnztt/must-gather-rjsl8" podStartSLOduration=1.817847064 podStartE2EDuration="2.454868804s" totalImagesPullingTime="637.02174ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:55:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:55:19.451712862 +0000 UTC m=+2651.779410403" watchObservedRunningTime="2026-07-10 17:55:19.454868804 +0000 UTC m=+2651.782566343" Jul 10 17:55:19.985656 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:19.985623 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-smz4h_ea2e6b67-bf39-4196-a657-7859b8b73199/global-pull-secret-syncer/0.log" Jul 10 17:55:20.071219 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:20.071178 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lql7p_3756f101-49b1-4754-ad4a-13c572eec69c/konnectivity-agent/0.log" Jul 10 17:55:20.143518 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:20.143484 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-65.ec2.internal_84e6cfddc1f961a111c9509968c60461/haproxy/0.log" Jul 10 17:55:24.216252 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:24.216199 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw_d81b6b2e-0d3c-4f29-ae16-361c0b8682a6/extract/0.log" Jul 10 17:55:24.243605 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:24.243574 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw_d81b6b2e-0d3c-4f29-ae16-361c0b8682a6/util/0.log" Jul 10 17:55:24.270994 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:24.270963 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759przvw_d81b6b2e-0d3c-4f29-ae16-361c0b8682a6/pull/0.log" Jul 10 17:55:24.295477 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:24.295446 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s_59ca63e6-8478-4612-814f-44f062905e67/extract/0.log" Jul 10 17:55:24.328000 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:24.327902 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s_59ca63e6-8478-4612-814f-44f062905e67/util/0.log" Jul 10 17:55:24.353116 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:24.353086 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gg96s_59ca63e6-8478-4612-814f-44f062905e67/pull/0.log" Jul 10 17:55:24.383267 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:24.383240 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79_e55f2ebf-0019-408b-bf88-2739a8763f78/extract/0.log" Jul 10 17:55:24.407517 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:24.407494 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79_e55f2ebf-0019-408b-bf88-2739a8763f78/util/0.log" Jul 10 17:55:24.437032 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:24.437006 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73s6m79_e55f2ebf-0019-408b-bf88-2739a8763f78/pull/0.log" Jul 10 17:55:24.467293 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:24.467194 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb_8db3a343-675d-40b5-a0a9-5a8a02374227/extract/0.log" Jul 10 17:55:24.503455 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:24.503430 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb_8db3a343-675d-40b5-a0a9-5a8a02374227/util/0.log" Jul 10 17:55:24.535192 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:24.535163 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1s4gqb_8db3a343-675d-40b5-a0a9-5a8a02374227/pull/0.log" Jul 10 17:55:24.860625 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:24.860498 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-vzrr7_c7b4a154-f747-4cea-889b-cdb3adeb7b13/manager/0.log" Jul 10 17:55:24.900820 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:24.900741 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-b2kmw_fd6d0d14-eced-40cb-bd67-12b7fbd9e539/manager/0.log" Jul 10 17:55:24.971438 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:24.971361 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-rf9pk_88d0b0b6-91bf-4f00-be4c-6c8191e5176e/registry-server/0.log" Jul 10 17:55:25.176420 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:25.176317 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-db25c_e0279bca-c281-4a5e-ab40-f74c854cde9c/manager/0.log" Jul 10 17:55:25.308145 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:25.306485 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-gskkj_79ebdd43-2bce-43f5-a14f-1c3090448a38/manager/0.log" Jul 10 17:55:27.433846 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:27.433813 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vgp2l_5f45c7ca-5e81-4c97-8b49-6efa2419024e/node-exporter/0.log" Jul 10 17:55:27.460603 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:27.460579 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vgp2l_5f45c7ca-5e81-4c97-8b49-6efa2419024e/kube-rbac-proxy/0.log" Jul 10 17:55:27.507617 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:27.507591 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vgp2l_5f45c7ca-5e81-4c97-8b49-6efa2419024e/init-textfile/0.log" Jul 10 17:55:30.442104 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:30.442070 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vnztt/perf-node-gather-daemonset-wpfcl"] Jul 10 17:55:30.448939 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:30.448904 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-wpfcl" Jul 10 17:55:30.457151 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:30.457123 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vnztt/perf-node-gather-daemonset-wpfcl"] Jul 10 17:55:30.582655 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:30.582622 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aee2898f-604e-4651-8fda-eceb2badf8bd-proc\") pod \"perf-node-gather-daemonset-wpfcl\" (UID: \"aee2898f-604e-4651-8fda-eceb2badf8bd\") " pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-wpfcl" Jul 10 17:55:30.582889 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:30.582873 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aee2898f-604e-4651-8fda-eceb2badf8bd-podres\") pod \"perf-node-gather-daemonset-wpfcl\" (UID: \"aee2898f-604e-4651-8fda-eceb2badf8bd\") " pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-wpfcl" Jul 10 17:55:30.583078 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:30.583065 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aee2898f-604e-4651-8fda-eceb2badf8bd-sys\") pod \"perf-node-gather-daemonset-wpfcl\" (UID: \"aee2898f-604e-4651-8fda-eceb2badf8bd\") " pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-wpfcl" Jul 10 17:55:30.583267 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:30.583229 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aee2898f-604e-4651-8fda-eceb2badf8bd-lib-modules\") pod \"perf-node-gather-daemonset-wpfcl\" (UID: \"aee2898f-604e-4651-8fda-eceb2badf8bd\") " pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-wpfcl" Jul 10 17:55:30.583434 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:30.583275 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnj9k\" (UniqueName: \"kubernetes.io/projected/aee2898f-604e-4651-8fda-eceb2badf8bd-kube-api-access-cnj9k\") pod \"perf-node-gather-daemonset-wpfcl\" (UID: \"aee2898f-604e-4651-8fda-eceb2badf8bd\") " pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-wpfcl" Jul 10 17:55:30.684019 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:30.683966 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aee2898f-604e-4651-8fda-eceb2badf8bd-sys\") pod \"perf-node-gather-daemonset-wpfcl\" (UID: \"aee2898f-604e-4651-8fda-eceb2badf8bd\") " pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-wpfcl" Jul 10 17:55:30.684224 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:30.684129 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aee2898f-604e-4651-8fda-eceb2badf8bd-lib-modules\") pod \"perf-node-gather-daemonset-wpfcl\" (UID: \"aee2898f-604e-4651-8fda-eceb2badf8bd\") " pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-wpfcl" Jul 10 17:55:30.684224 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:30.684159 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnj9k\" (UniqueName: \"kubernetes.io/projected/aee2898f-604e-4651-8fda-eceb2badf8bd-kube-api-access-cnj9k\") pod \"perf-node-gather-daemonset-wpfcl\" (UID: \"aee2898f-604e-4651-8fda-eceb2badf8bd\") " pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-wpfcl" Jul 10 17:55:30.684224 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:30.684077 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aee2898f-604e-4651-8fda-eceb2badf8bd-sys\") pod \"perf-node-gather-daemonset-wpfcl\" (UID: \"aee2898f-604e-4651-8fda-eceb2badf8bd\") " pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-wpfcl" Jul 10 17:55:30.684438 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:30.684221 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aee2898f-604e-4651-8fda-eceb2badf8bd-proc\") pod \"perf-node-gather-daemonset-wpfcl\" (UID: \"aee2898f-604e-4651-8fda-eceb2badf8bd\") " pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-wpfcl" Jul 10 17:55:30.684438 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:30.684276 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aee2898f-604e-4651-8fda-eceb2badf8bd-podres\") pod \"perf-node-gather-daemonset-wpfcl\" (UID: \"aee2898f-604e-4651-8fda-eceb2badf8bd\") " pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-wpfcl" Jul 10 17:55:30.684438 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:30.684348 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aee2898f-604e-4651-8fda-eceb2badf8bd-lib-modules\") pod \"perf-node-gather-daemonset-wpfcl\" (UID: \"aee2898f-604e-4651-8fda-eceb2badf8bd\") " pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-wpfcl" Jul 10 17:55:30.684438 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:30.684422 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aee2898f-604e-4651-8fda-eceb2badf8bd-podres\") pod \"perf-node-gather-daemonset-wpfcl\" (UID: \"aee2898f-604e-4651-8fda-eceb2badf8bd\") " pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-wpfcl" Jul 10 17:55:30.684668 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:30.684483 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aee2898f-604e-4651-8fda-eceb2badf8bd-proc\") pod \"perf-node-gather-daemonset-wpfcl\" (UID: \"aee2898f-604e-4651-8fda-eceb2badf8bd\") " pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-wpfcl" Jul 10 17:55:30.694336 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:30.694262 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnj9k\" (UniqueName: \"kubernetes.io/projected/aee2898f-604e-4651-8fda-eceb2badf8bd-kube-api-access-cnj9k\") pod \"perf-node-gather-daemonset-wpfcl\" (UID: \"aee2898f-604e-4651-8fda-eceb2badf8bd\") " pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-wpfcl" Jul 10 17:55:30.766287 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:30.766240 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-wpfcl" Jul 10 17:55:30.933938 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:30.933909 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vnztt/perf-node-gather-daemonset-wpfcl"] Jul 10 17:55:30.935791 ip-10-0-134-65 kubenswrapper[2583]: W0710 17:55:30.935737 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podaee2898f_604e_4651_8fda_eceb2badf8bd.slice/crio-f3c49a89a0b278037c9c51a50bcf6a8794c8681fa328280c7ff87a67bae261d9 WatchSource:0}: Error finding container f3c49a89a0b278037c9c51a50bcf6a8794c8681fa328280c7ff87a67bae261d9: Status 404 returned error can't find the container with id f3c49a89a0b278037c9c51a50bcf6a8794c8681fa328280c7ff87a67bae261d9 Jul 10 17:55:31.547325 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:31.547277 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-wpfcl" event={"ID":"aee2898f-604e-4651-8fda-eceb2badf8bd","Type":"ContainerStarted","Data":"7d239bde6dd0257d69b837f15c43c847558abab81a082e9a7ac03b6bb9587746"} Jul 10 17:55:31.547851 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:31.547410 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-wpfcl" event={"ID":"aee2898f-604e-4651-8fda-eceb2badf8bd","Type":"ContainerStarted","Data":"f3c49a89a0b278037c9c51a50bcf6a8794c8681fa328280c7ff87a67bae261d9"} Jul 10 17:55:31.548281 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:31.548252 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-wpfcl" Jul 10 17:55:31.573456 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:31.573357 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-wpfcl" podStartSLOduration=1.573335591 podStartE2EDuration="1.573335591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:55:31.57005509 +0000 UTC m=+2663.897752629" watchObservedRunningTime="2026-07-10 17:55:31.573335591 +0000 UTC m=+2663.901033133" Jul 10 17:55:32.467963 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:32.467926 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r9l2z_d4ebb446-d269-4e98-b1e4-258948073db1/dns/0.log" Jul 10 17:55:32.492576 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:32.492543 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r9l2z_d4ebb446-d269-4e98-b1e4-258948073db1/kube-rbac-proxy/0.log" Jul 10 17:55:32.520315 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:32.520258 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bq9tx_9369f652-8e06-4d0a-b00e-07ad43e2bec1/dns-node-resolver/0.log" Jul 10 17:55:33.148944 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:33.148900 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-cb8684c9b-nckr9_44b85e98-52d3-4c78-84b7-7715d239e1bd/registry/0.log" Jul 10 17:55:33.198080 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:33.198050 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5krz5_df9374c1-e474-4ceb-b6d6-a8879a52e376/node-ca/0.log" Jul 10 17:55:34.442840 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:34.442812 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-545f68d885-qzl6x_7875067d-0704-4751-9605-5e851f2421a8/kube-auth-proxy/0.log" Jul 10 17:55:34.477485 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:34.477455 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-mlrv6_553d3bf3-9e01-4b25-9ac2-01b0da4faf79/istio-proxy/0.log" Jul 10 17:55:34.516346 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:34.516321 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-mlrv6_553d3bf3-9e01-4b25-9ac2-01b0da4faf79/istio-proxy/1.log" Jul 10 17:55:34.549740 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:34.549674 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-qrscc_ee5cf1f6-d727-424a-82b4-4581245dba8c/payload-pre-processing/0.log" Jul 10 17:55:34.584470 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:34.584435 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-6bjhw_c69c3ec1-fa81-4c47-ac65-c478feee0a14/payload-processing/0.log" Jul 10 17:55:35.366008 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:35.365978 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wr97q_b8edbb95-6b40-46ed-a7ad-0de010dbf2dd/serve-healthcheck-canary/0.log" Jul 10 17:55:36.042890 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:36.042857 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fvjjb_29817af8-c839-459c-a69d-733e4926b16b/kube-rbac-proxy/0.log" Jul 10 17:55:36.069880 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:36.069847 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fvjjb_29817af8-c839-459c-a69d-733e4926b16b/exporter/0.log" Jul 10 17:55:36.102292 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:36.102263 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fvjjb_29817af8-c839-459c-a69d-733e4926b16b/extractor/0.log" Jul 10 17:55:38.268612 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:38.268587 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-8htph_9b800003-9484-432d-b004-452a75d4765b/manager/0.log" Jul 10 17:55:38.445141 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:38.445100 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-p7lgl_86283d78-f4af-4cd5-aaab-a075694c5a37/manager/0.log" Jul 10 17:55:38.578845 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:38.578757 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vnztt/perf-node-gather-daemonset-wpfcl" Jul 10 17:55:38.588666 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:38.588626 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b4655bb55-h9s24_49640ea0-1609-4683-bf14-f14f0bdb5882/manager/0.log" Jul 10 17:55:38.631864 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:38.631833 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-7cxn7_dae95bcd-c4af-4c58-bf11-2aef952610c5/server/0.log" Jul 10 17:55:38.741628 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:38.741592 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-h8qcx_313596b5-3368-4ca3-8f71-0be58e50e7f4/manager/0.log" Jul 10 17:55:38.914838 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:38.914775 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b884cfc77-g4rrm_c545fd6c-d7a9-4ad4-845c-2504f3cf59fc/manager/0.log" Jul 10 17:55:38.940492 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:38.940464 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b884cfc77-g4rrm_c545fd6c-d7a9-4ad4-845c-2504f3cf59fc/copy-manifests/0.log" Jul 10 17:55:41.123292 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:41.123202 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-545f68d885-qzl6x_7875067d-0704-4751-9605-5e851f2421a8/kube-auth-proxy/0.log" Jul 10 17:55:41.146626 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:41.146594 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-mlrv6_553d3bf3-9e01-4b25-9ac2-01b0da4faf79/istio-proxy/0.log" Jul 10 17:55:41.185070 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:41.184874 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-mlrv6_553d3bf3-9e01-4b25-9ac2-01b0da4faf79/istio-proxy/1.log" Jul 10 17:55:41.210308 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:41.210276 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-qrscc_ee5cf1f6-d727-424a-82b4-4581245dba8c/payload-pre-processing/0.log" Jul 10 17:55:41.235566 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:41.235534 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-6bjhw_c69c3ec1-fa81-4c47-ac65-c478feee0a14/payload-processing/0.log" Jul 10 17:55:42.729654 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:42.729619 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-545f68d885-qzl6x_7875067d-0704-4751-9605-5e851f2421a8/kube-auth-proxy/0.log" Jul 10 17:55:42.753178 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:42.753147 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-mlrv6_553d3bf3-9e01-4b25-9ac2-01b0da4faf79/istio-proxy/0.log" Jul 10 17:55:42.798467 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:42.798351 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-mlrv6_553d3bf3-9e01-4b25-9ac2-01b0da4faf79/istio-proxy/1.log" Jul 10 17:55:42.824343 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:42.824310 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-qrscc_ee5cf1f6-d727-424a-82b4-4581245dba8c/payload-pre-processing/0.log" Jul 10 17:55:42.851022 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:42.850994 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-6bjhw_c69c3ec1-fa81-4c47-ac65-c478feee0a14/payload-processing/0.log" Jul 10 17:55:45.638336 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:45.638307 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-55q9h_96d6166e-fa6f-42f6-81d9-18d199c2516a/migrator/0.log" Jul 10 17:55:45.660087 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:45.660056 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-55q9h_96d6166e-fa6f-42f6-81d9-18d199c2516a/graceful-termination/0.log" Jul 10 17:55:47.021791 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:47.021761 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-52frq_7a7bf0c2-8b0e-432f-aa79-a4b6d8f95eb7/kube-multus/0.log" Jul 10 17:55:47.365067 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:47.364988 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vk6rc_1e13ba88-7d4c-42b8-a992-95b3c447b222/kube-multus-additional-cni-plugins/0.log" Jul 10 17:55:47.385814 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:47.385785 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vk6rc_1e13ba88-7d4c-42b8-a992-95b3c447b222/egress-router-binary-copy/0.log" Jul 10 17:55:47.406859 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:47.406832 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vk6rc_1e13ba88-7d4c-42b8-a992-95b3c447b222/cni-plugins/0.log" Jul 10 17:55:47.427655 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:47.427627 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vk6rc_1e13ba88-7d4c-42b8-a992-95b3c447b222/bond-cni-plugin/0.log" Jul 10 17:55:47.448703 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:47.448677 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vk6rc_1e13ba88-7d4c-42b8-a992-95b3c447b222/routeoverride-cni/0.log" Jul 10 17:55:47.469024 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:47.468999 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vk6rc_1e13ba88-7d4c-42b8-a992-95b3c447b222/whereabouts-cni-bincopy/0.log" Jul 10 17:55:47.489565 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:47.489541 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vk6rc_1e13ba88-7d4c-42b8-a992-95b3c447b222/whereabouts-cni/0.log" Jul 10 17:55:47.683564 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:47.683541 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-h9jx7_d177790b-fe2d-469e-9dff-c653997fe096/network-metrics-daemon/0.log" Jul 10 17:55:47.702707 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:47.702683 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-h9jx7_d177790b-fe2d-469e-9dff-c653997fe096/kube-rbac-proxy/0.log" Jul 10 17:55:49.001431 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:49.001401 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js69w_e41baaf7-4171-4df1-8bdd-d0697a61fdce/ovn-controller/0.log" Jul 10 17:55:49.017738 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:49.017705 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js69w_e41baaf7-4171-4df1-8bdd-d0697a61fdce/ovn-acl-logging/0.log" Jul 10 17:55:49.018007 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:49.017990 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js69w_e41baaf7-4171-4df1-8bdd-d0697a61fdce/ovn-acl-logging/1.log" Jul 10 17:55:49.036500 ip-10-0-134-65 kubenswrapper[2583]: I0710 17:55:49.036463 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js69w_e41baaf7-4171-4df1-8bdd-d0697a61fdce/kube-rbac-proxy-node/0.log"