Jul 06 20:03:39.433344 ip-10-0-131-218 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 06 20:03:39.433357 ip-10-0-131-218 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 06 20:03:39.433365 ip-10-0-131-218 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 06 20:03:39.433659 ip-10-0-131-218 systemd[1]: Failed to start Kubernetes Kubelet. Jul 06 20:03:49.545162 ip-10-0-131-218 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 06 20:03:49.545179 ip-10-0-131-218 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 46a14ff4556b4f66844145c1e087cda0 -- Jul 06 20:05:48.651766 ip-10-0-131-218 systemd[1]: Starting Kubernetes Kubelet... Jul 06 20:05:49.081081 ip-10-0-131-218 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 20:05:49.081081 ip-10-0-131-218 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 20:05:49.081081 ip-10-0-131-218 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 20:05:49.081081 ip-10-0-131-218 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 20:05:49.081081 ip-10-0-131-218 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 20:05:49.083764 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.083626 2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 20:05:49.088285 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088264 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:05:49.088285 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088281 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:05:49.088285 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088287 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:05:49.088285 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088291 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:05:49.088443 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088294 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:05:49.088443 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088299 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:05:49.088443 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088302 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:05:49.088443 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088305 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:05:49.088443 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088308 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:05:49.088443 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088310 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:05:49.088443 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088313 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:05:49.088443 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088315 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:05:49.088443 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088318 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:05:49.088443 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088321 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:05:49.088443 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088324 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:05:49.088443 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088326 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:05:49.088443 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088329 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:05:49.088443 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088331 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:05:49.088443 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088334 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:05:49.088443 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088337 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:05:49.088443 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088339 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:05:49.088443 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088341 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:05:49.088443 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088344 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:05:49.088923 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088346 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:05:49.088923 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088349 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:05:49.088923 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088351 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:05:49.088923 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088354 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:05:49.088923 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088356 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:05:49.088923 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088359 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:05:49.088923 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088362 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:05:49.088923 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088368 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:05:49.088923 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088370 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:05:49.088923 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088373 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:05:49.088923 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088375 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:05:49.088923 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088378 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:05:49.088923 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088381 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:05:49.088923 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088383 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:05:49.088923 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088386 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:05:49.088923 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088389 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:05:49.088923 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088391 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:05:49.088923 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088394 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:05:49.088923 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088396 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:05:49.088923 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088399 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:05:49.089573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088402 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:05:49.089573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088404 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:05:49.089573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088407 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:05:49.089573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088409 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:05:49.089573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088412 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:05:49.089573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088415 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:05:49.089573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088417 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:05:49.089573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088420 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:05:49.089573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088423 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:05:49.089573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088426 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:05:49.089573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088428 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:05:49.089573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088431 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:05:49.089573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088433 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:05:49.089573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088435 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:05:49.089573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088438 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:05:49.089573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088441 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:05:49.089573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088444 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:05:49.089573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088446 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:05:49.089573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088463 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:05:49.089573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088469 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:05:49.090064 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088472 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:05:49.090064 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088474 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:05:49.090064 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088477 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:05:49.090064 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088481 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:05:49.090064 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088485 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:05:49.090064 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088489 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:05:49.090064 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088494 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:05:49.090064 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088497 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:05:49.090064 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088500 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:05:49.090064 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088503 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:05:49.090064 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088505 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:05:49.090064 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088510 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:05:49.090064 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088513 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:05:49.090064 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088516 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:05:49.090064 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088519 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:05:49.090064 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088522 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:05:49.090064 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088524 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:05:49.090064 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088527 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:05:49.090064 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088530 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:05:49.090064 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088533 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:05:49.090585 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088535 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:05:49.090585 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088538 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:05:49.090585 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088540 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:05:49.090585 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088543 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:05:49.090585 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088546 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:05:49.090585 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088548 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:05:49.090585 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088551 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:05:49.090585 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088554 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:05:49.090585 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088556 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:05:49.090585 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088559 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:05:49.090585 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088561 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:05:49.090585 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088564 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:05:49.090585 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088566 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:05:49.090585 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088569 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:05:49.090585 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088571 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:05:49.090585 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.088574 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:05:49.090585 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089263 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:05:49.090585 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089269 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:05:49.090585 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089273 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:05:49.090585 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089276 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:05:49.091062 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089278 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:05:49.091062 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089281 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:05:49.091062 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089283 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:05:49.091062 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089286 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:05:49.091062 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089289 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:05:49.091062 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089292 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:05:49.091062 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089294 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:05:49.091062 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089299 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:05:49.091062 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089303 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:05:49.091062 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089306 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:05:49.091062 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089309 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:05:49.091062 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089312 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:05:49.091062 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089315 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:05:49.091062 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089318 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:05:49.091062 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089320 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:05:49.091062 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089323 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:05:49.091062 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089325 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:05:49.091062 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089328 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:05:49.091062 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089330 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:05:49.091062 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089343 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:05:49.091573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089346 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:05:49.091573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089349 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:05:49.091573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089352 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:05:49.091573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089354 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:05:49.091573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089357 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:05:49.091573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089359 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:05:49.091573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089363 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:05:49.091573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089365 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:05:49.091573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089368 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:05:49.091573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089371 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:05:49.091573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089375 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:05:49.091573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089378 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:05:49.091573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089382 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:05:49.091573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089385 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:05:49.091573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089389 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:05:49.091573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089391 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:05:49.091573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089394 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:05:49.091573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089397 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:05:49.091573 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089402 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:05:49.092038 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089404 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:05:49.092038 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089407 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:05:49.092038 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089410 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:05:49.092038 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089413 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:05:49.092038 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089415 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:05:49.092038 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089419 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:05:49.092038 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089422 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:05:49.092038 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089425 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:05:49.092038 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089428 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:05:49.092038 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089430 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:05:49.092038 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089433 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:05:49.092038 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089436 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:05:49.092038 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089438 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:05:49.092038 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089441 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:05:49.092038 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089444 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:05:49.092038 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089447 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:05:49.092038 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089449 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:05:49.092038 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089470 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:05:49.092038 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089475 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:05:49.092500 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089479 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:05:49.092500 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089483 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:05:49.092500 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089487 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:05:49.092500 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089490 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:05:49.092500 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089493 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:05:49.092500 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089495 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:05:49.092500 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089498 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:05:49.092500 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089501 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:05:49.092500 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089504 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:05:49.092500 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089506 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:05:49.092500 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089509 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:05:49.092500 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089511 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:05:49.092500 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089514 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:05:49.092500 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089517 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:05:49.092500 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089520 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:05:49.092500 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089522 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:05:49.092500 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089525 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:05:49.092500 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089528 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:05:49.092500 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089531 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:05:49.092500 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089536 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:05:49.092977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089538 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:05:49.092977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089541 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:05:49.092977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089543 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:05:49.092977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089545 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:05:49.092977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089548 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:05:49.092977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089551 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:05:49.092977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089553 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:05:49.092977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089556 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:05:49.092977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089558 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:05:49.092977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089561 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:05:49.092977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089563 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:05:49.092977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089566 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:05:49.092977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089568 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:05:49.092977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089571 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:05:49.092977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089573 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:05:49.092977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089577 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:05:49.092977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089580 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:05:49.092977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089960 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:05:49.092977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089965 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:05:49.093481 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089967 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:05:49.093481 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089970 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:05:49.093481 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089973 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:05:49.093481 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089976 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:05:49.093481 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089978 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:05:49.093481 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089981 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:05:49.093481 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089984 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:05:49.093481 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089987 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:05:49.093481 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089990 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:05:49.093481 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089992 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:05:49.093481 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089995 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:05:49.093481 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.089998 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:05:49.093481 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090001 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:05:49.093481 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090004 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:05:49.093481 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090006 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:05:49.093481 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090009 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:05:49.093481 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090011 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:05:49.093481 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090014 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:05:49.093481 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090016 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:05:49.093481 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090019 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:05:49.093977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090022 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:05:49.093977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090024 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:05:49.093977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090026 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:05:49.093977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090029 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:05:49.093977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090031 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:05:49.093977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090034 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:05:49.093977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090036 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:05:49.093977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090039 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:05:49.093977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090042 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:05:49.093977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090048 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:05:49.093977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090050 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:05:49.093977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090053 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:05:49.093977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090056 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:05:49.093977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090058 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:05:49.093977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090061 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:05:49.093977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090064 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:05:49.093977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090066 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:05:49.093977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090069 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:05:49.093977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090072 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:05:49.093977 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090075 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:05:49.094478 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090078 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:05:49.094478 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090080 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:05:49.094478 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090083 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:05:49.094478 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090086 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:05:49.094478 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090088 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:05:49.094478 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090092 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:05:49.094478 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090095 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:05:49.094478 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090097 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:05:49.094478 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090099 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:05:49.094478 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090102 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:05:49.094478 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090105 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:05:49.094478 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090107 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:05:49.094478 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090109 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:05:49.094478 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090112 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:05:49.094478 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090114 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:05:49.094478 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090117 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:05:49.094478 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090120 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:05:49.094478 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090122 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:05:49.094478 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090125 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:05:49.094980 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090129 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:05:49.094980 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090133 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:05:49.094980 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090138 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:05:49.094980 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090141 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:05:49.094980 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090145 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:05:49.094980 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090173 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:05:49.094980 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090177 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:05:49.094980 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090180 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:05:49.094980 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090183 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:05:49.094980 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090186 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:05:49.094980 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090188 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:05:49.094980 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090191 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:05:49.094980 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090193 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:05:49.094980 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090196 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:05:49.094980 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090198 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:05:49.094980 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090201 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:05:49.094980 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090204 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:05:49.094980 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090206 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:05:49.094980 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090209 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:05:49.095429 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090212 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:05:49.095429 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090215 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:05:49.095429 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090217 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:05:49.095429 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090220 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:05:49.095429 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090222 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:05:49.095429 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090226 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:05:49.095429 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090228 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:05:49.095429 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090231 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:05:49.095429 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090233 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:05:49.095429 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090236 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:05:49.095429 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090238 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:05:49.095429 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090242 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:05:49.095429 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090245 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:05:49.095429 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090249 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:05:49.095429 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090252 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:05:49.095429 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090256 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:05:49.095429 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090258 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:05:49.095429 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090261 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:05:49.095429 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090263 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:05:49.095429 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090327 2578 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 20:05:49.095929 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090334 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 20:05:49.095929 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090340 2578 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 20:05:49.095929 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090344 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 20:05:49.095929 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090349 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 20:05:49.095929 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090354 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 20:05:49.095929 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090360 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 20:05:49.095929 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090376 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 20:05:49.095929 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090380 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 20:05:49.095929 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090383 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 20:05:49.095929 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090387 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 20:05:49.095929 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090391 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 20:05:49.095929 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090393 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 20:05:49.095929 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090396 2578 flags.go:64] FLAG: --cgroup-root="" Jul 06 20:05:49.095929 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090400 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 20:05:49.095929 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090403 2578 flags.go:64] FLAG: --client-ca-file="" Jul 06 20:05:49.095929 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090406 2578 flags.go:64] FLAG: --cloud-provider="external" Jul 06 20:05:49.095929 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090409 2578 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 20:05:49.095929 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090413 2578 flags.go:64] FLAG: --cluster-domain="" Jul 06 20:05:49.095929 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090416 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 20:05:49.095929 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090419 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 20:05:49.095929 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090422 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 20:05:49.095929 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090425 2578 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 20:05:49.095929 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090429 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 20:05:49.095929 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090431 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 20:05:49.096551 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090434 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 20:05:49.096551 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090441 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 20:05:49.096551 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090444 2578 flags.go:64] FLAG: --contention-profiling="false" Jul 06 20:05:49.096551 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090448 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 20:05:49.096551 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090471 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 20:05:49.096551 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090475 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 20:05:49.096551 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090478 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 20:05:49.096551 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090482 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 20:05:49.096551 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090485 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 20:05:49.096551 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090488 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 20:05:49.096551 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090491 2578 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 20:05:49.096551 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090494 2578 flags.go:64] FLAG: --enable-server="true" Jul 06 20:05:49.096551 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090497 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 20:05:49.096551 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090501 2578 flags.go:64] FLAG: --event-burst="100" Jul 06 20:05:49.096551 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090504 2578 flags.go:64] FLAG: --event-qps="50" Jul 06 20:05:49.096551 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090507 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 20:05:49.096551 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090510 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 20:05:49.096551 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090512 2578 flags.go:64] FLAG: --eviction-hard="" Jul 06 20:05:49.096551 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090516 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 20:05:49.096551 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090519 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 20:05:49.096551 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090521 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 20:05:49.096551 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090524 2578 flags.go:64] FLAG: --eviction-soft="" Jul 06 20:05:49.096551 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090528 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 20:05:49.096551 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090531 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 20:05:49.096551 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090534 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 20:05:49.097174 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090537 2578 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 20:05:49.097174 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090539 2578 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 20:05:49.097174 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090542 2578 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 20:05:49.097174 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090545 2578 flags.go:64] FLAG: --feature-gates="" Jul 06 20:05:49.097174 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090549 2578 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 20:05:49.097174 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090552 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 20:05:49.097174 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090554 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 20:05:49.097174 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090557 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 20:05:49.097174 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090561 2578 flags.go:64] FLAG: --healthz-port="10248" Jul 06 20:05:49.097174 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090564 2578 flags.go:64] FLAG: --help="false" Jul 06 20:05:49.097174 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090568 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-131-218.ec2.internal" Jul 06 20:05:49.097174 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090572 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 20:05:49.097174 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090574 2578 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 20:05:49.097174 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090577 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 20:05:49.097174 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090581 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 20:05:49.097174 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090584 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 20:05:49.097174 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090587 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 20:05:49.097174 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090590 2578 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 20:05:49.097174 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090592 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 20:05:49.097174 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090595 2578 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 20:05:49.097174 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090598 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 20:05:49.097174 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090602 2578 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 20:05:49.097174 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090604 2578 flags.go:64] FLAG: --kube-reserved="" Jul 06 20:05:49.097174 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090607 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 20:05:49.097781 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090610 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 20:05:49.097781 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090613 2578 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 20:05:49.097781 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090616 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 20:05:49.097781 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090618 2578 flags.go:64] FLAG: --lock-file="" Jul 06 20:05:49.097781 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090621 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 20:05:49.097781 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090624 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 20:05:49.097781 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090628 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 20:05:49.097781 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090633 2578 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 20:05:49.097781 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090635 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 20:05:49.097781 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090638 2578 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 20:05:49.097781 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090641 2578 flags.go:64] FLAG: --logging-format="text" Jul 06 20:05:49.097781 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090643 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 20:05:49.097781 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090646 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 20:05:49.097781 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090649 2578 flags.go:64] FLAG: --manifest-url="" Jul 06 20:05:49.097781 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090651 2578 flags.go:64] FLAG: --manifest-url-header="" Jul 06 20:05:49.097781 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090655 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 20:05:49.097781 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090660 2578 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 20:05:49.097781 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090664 2578 flags.go:64] FLAG: --max-pods="110" Jul 06 20:05:49.097781 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090668 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 20:05:49.097781 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090671 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 20:05:49.097781 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090674 2578 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 20:05:49.097781 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090676 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 20:05:49.097781 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090679 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 20:05:49.097781 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090682 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 20:05:49.097781 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090685 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 20:05:49.098384 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090692 2578 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 20:05:49.098384 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090695 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 20:05:49.098384 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090698 2578 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 20:05:49.098384 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090701 2578 flags.go:64] FLAG: --pod-cidr="" Jul 06 20:05:49.098384 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090704 2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 20:05:49.098384 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090710 2578 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 20:05:49.098384 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090712 2578 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 20:05:49.098384 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090715 2578 flags.go:64] FLAG: --pods-per-core="0" Jul 06 20:05:49.098384 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090718 2578 flags.go:64] FLAG: --port="10250" Jul 06 20:05:49.098384 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090721 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 20:05:49.098384 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090724 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03675c531ca2f865c" Jul 06 20:05:49.098384 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090727 2578 flags.go:64] FLAG: --qos-reserved="" Jul 06 20:05:49.098384 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090730 2578 flags.go:64] FLAG: --read-only-port="10255" Jul 06 20:05:49.098384 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090733 2578 flags.go:64] FLAG: --register-node="true" Jul 06 20:05:49.098384 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090737 2578 flags.go:64] FLAG: --register-with-taints="" Jul 06 20:05:49.098384 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090740 2578 flags.go:64] FLAG: --registry-burst="10" Jul 06 20:05:49.098384 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090743 2578 flags.go:64] FLAG: --registry-qps="5" Jul 06 20:05:49.098384 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090745 2578 flags.go:64] FLAG: --reserved-cpus="" Jul 06 20:05:49.098384 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090748 2578 flags.go:64] FLAG: --reserved-memory="" Jul 06 20:05:49.098384 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090751 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 20:05:49.098384 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090754 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 20:05:49.098384 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090757 2578 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 20:05:49.098384 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090760 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 20:05:49.098384 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090764 2578 flags.go:64] FLAG: --runonce="false" Jul 06 20:05:49.098384 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090767 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 20:05:49.099018 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090770 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 20:05:49.099018 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090774 2578 flags.go:64] FLAG: --seccomp-default="false" Jul 06 20:05:49.099018 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090776 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 20:05:49.099018 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090779 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 20:05:49.099018 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090782 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 20:05:49.099018 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090785 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 20:05:49.099018 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090788 2578 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 20:05:49.099018 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090791 2578 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 20:05:49.099018 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090794 2578 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 20:05:49.099018 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090796 2578 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 20:05:49.099018 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090799 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 20:05:49.099018 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090802 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 20:05:49.099018 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090805 2578 flags.go:64] FLAG: --system-cgroups="" Jul 06 20:05:49.099018 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090807 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 20:05:49.099018 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090813 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 20:05:49.099018 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090815 2578 flags.go:64] FLAG: --tls-cert-file="" Jul 06 20:05:49.099018 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090818 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 20:05:49.099018 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090822 2578 flags.go:64] FLAG: --tls-min-version="" Jul 06 20:05:49.099018 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090825 2578 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 20:05:49.099018 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090827 2578 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 20:05:49.099018 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090830 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 20:05:49.099018 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090833 2578 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 20:05:49.099018 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090840 2578 flags.go:64] FLAG: --v="2" Jul 06 20:05:49.099018 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090844 2578 flags.go:64] FLAG: --version="false" Jul 06 20:05:49.099018 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090848 2578 flags.go:64] FLAG: --vmodule="" Jul 06 20:05:49.099615 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090851 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 20:05:49.099615 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.090854 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 20:05:49.099615 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090973 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:05:49.099615 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090977 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:05:49.099615 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090980 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:05:49.099615 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090983 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:05:49.099615 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090986 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:05:49.099615 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090989 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:05:49.099615 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090992 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:05:49.099615 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090996 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:05:49.099615 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.090998 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:05:49.099615 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091001 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:05:49.099615 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091003 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:05:49.099615 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091005 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:05:49.099615 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091008 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:05:49.099615 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091012 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:05:49.099615 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091016 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:05:49.099615 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091019 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:05:49.099615 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091022 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:05:49.099615 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091024 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:05:49.100133 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091027 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:05:49.100133 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091029 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:05:49.100133 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091032 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:05:49.100133 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091034 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:05:49.100133 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091037 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:05:49.100133 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091039 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:05:49.100133 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091042 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:05:49.100133 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091045 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:05:49.100133 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091047 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:05:49.100133 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091050 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:05:49.100133 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091053 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:05:49.100133 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091055 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:05:49.100133 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091059 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:05:49.100133 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091062 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:05:49.100133 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091065 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:05:49.100133 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091068 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:05:49.100133 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091071 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:05:49.100133 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091076 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:05:49.100133 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091078 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:05:49.100614 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091081 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:05:49.100614 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091085 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:05:49.100614 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091087 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:05:49.100614 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091090 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:05:49.100614 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091093 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:05:49.100614 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091095 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:05:49.100614 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091098 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:05:49.100614 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091101 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:05:49.100614 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091103 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:05:49.100614 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091106 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:05:49.100614 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091108 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:05:49.100614 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091111 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:05:49.100614 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091114 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:05:49.100614 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091116 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:05:49.100614 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091119 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:05:49.100614 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091121 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:05:49.100614 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091124 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:05:49.100614 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091127 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:05:49.100614 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091129 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:05:49.100614 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091132 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:05:49.101103 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091134 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:05:49.101103 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091137 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:05:49.101103 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091139 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:05:49.101103 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091142 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:05:49.101103 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091145 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:05:49.101103 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091148 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:05:49.101103 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091150 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:05:49.101103 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091152 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:05:49.101103 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091155 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:05:49.101103 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091157 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:05:49.101103 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091161 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:05:49.101103 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091164 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:05:49.101103 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091167 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:05:49.101103 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091171 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:05:49.101103 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091173 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:05:49.101103 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091176 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:05:49.101103 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091179 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:05:49.101103 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091181 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:05:49.101103 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091184 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:05:49.101103 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091186 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:05:49.101627 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091190 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:05:49.101627 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091193 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:05:49.101627 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091196 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:05:49.101627 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091199 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:05:49.101627 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091201 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:05:49.101627 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091204 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:05:49.101627 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091206 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:05:49.101627 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091209 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:05:49.101627 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091212 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:05:49.101627 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091215 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:05:49.101627 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091217 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:05:49.101627 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091220 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:05:49.101627 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091223 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:05:49.101627 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091225 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:05:49.101627 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091227 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:05:49.101627 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091230 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:05:49.101627 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091232 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:05:49.101627 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091235 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:05:49.101627 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091238 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:05:49.102082 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091240 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:05:49.102082 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091242 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:05:49.102082 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.091245 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:05:49.102082 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.091252 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 20:05:49.102082 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.097783 2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 20:05:49.102082 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.097809 2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 20:05:49.102082 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.097828 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 20:05:49.102082 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.097834 2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 20:05:49.102082 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097888 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:05:49.102082 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097894 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:05:49.102082 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097897 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:05:49.102082 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097900 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:05:49.102082 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097903 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:05:49.102082 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097906 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:05:49.102082 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097909 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:05:49.102470 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097913 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:05:49.102470 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097916 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:05:49.102470 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097918 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:05:49.102470 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097922 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:05:49.102470 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097926 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:05:49.102470 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097929 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:05:49.102470 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097933 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:05:49.102470 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097935 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:05:49.102470 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097938 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:05:49.102470 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097940 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:05:49.102470 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097943 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:05:49.102470 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097946 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:05:49.102470 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097948 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:05:49.102470 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097951 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:05:49.102470 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097954 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:05:49.102470 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097956 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:05:49.102470 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097958 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:05:49.102470 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097961 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:05:49.102470 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097963 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:05:49.102929 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097966 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:05:49.102929 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097969 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:05:49.102929 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097972 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:05:49.102929 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097975 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:05:49.102929 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097978 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:05:49.102929 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097981 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:05:49.102929 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097984 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:05:49.102929 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097987 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:05:49.102929 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097989 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:05:49.102929 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097992 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:05:49.102929 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097995 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:05:49.102929 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.097997 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:05:49.102929 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098000 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:05:49.102929 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098002 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:05:49.102929 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098005 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:05:49.102929 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098007 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:05:49.102929 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098010 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:05:49.102929 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098013 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:05:49.102929 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098015 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:05:49.103384 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098018 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:05:49.103384 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098020 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:05:49.103384 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098023 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:05:49.103384 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098026 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:05:49.103384 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098028 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:05:49.103384 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098031 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:05:49.103384 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098033 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:05:49.103384 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098036 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:05:49.103384 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098038 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:05:49.103384 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098041 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:05:49.103384 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098044 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:05:49.103384 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098046 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:05:49.103384 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098048 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:05:49.103384 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098051 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:05:49.103384 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098054 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:05:49.103384 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098056 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:05:49.103384 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098059 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:05:49.103384 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098062 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:05:49.103384 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098065 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:05:49.103384 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098068 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:05:49.103881 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098070 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:05:49.103881 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098073 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:05:49.103881 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098076 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:05:49.103881 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098078 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:05:49.103881 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098081 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:05:49.103881 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098083 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:05:49.103881 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098086 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:05:49.103881 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098088 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:05:49.103881 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098090 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:05:49.103881 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098093 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:05:49.103881 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098095 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:05:49.103881 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098098 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:05:49.103881 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098100 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:05:49.103881 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098102 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:05:49.103881 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098105 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:05:49.103881 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098107 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:05:49.103881 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098110 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:05:49.103881 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098112 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:05:49.103881 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098115 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:05:49.103881 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098117 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:05:49.104364 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098120 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:05:49.104364 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098122 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:05:49.104364 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098124 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:05:49.104364 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098127 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:05:49.104364 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098129 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:05:49.104364 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098132 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:05:49.104364 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098134 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:05:49.104364 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098137 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:05:49.104364 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098142 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:05:49.104364 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098146 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:05:49.104364 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098151 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:05:49.104364 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098154 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:05:49.104364 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098157 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:05:49.104364 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098160 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:05:49.104719 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.098165 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 20:05:49.104719 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098277 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:05:49.104719 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098282 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:05:49.104719 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098285 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:05:49.104719 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098288 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:05:49.104719 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098291 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:05:49.104719 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098294 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:05:49.104719 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098296 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:05:49.104719 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098299 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:05:49.104719 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098301 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:05:49.104719 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098304 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:05:49.104719 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098307 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:05:49.104719 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098309 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:05:49.104719 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098311 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:05:49.104719 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098314 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:05:49.105234 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098316 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:05:49.105234 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098319 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:05:49.105234 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098321 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:05:49.105234 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098323 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:05:49.105234 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098326 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:05:49.105234 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098328 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:05:49.105234 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098331 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:05:49.105234 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098333 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:05:49.105234 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098336 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:05:49.105234 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098339 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:05:49.105234 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098342 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:05:49.105234 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098344 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:05:49.105234 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098347 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:05:49.105234 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098349 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:05:49.105234 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098353 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:05:49.105234 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098356 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:05:49.105234 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098358 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:05:49.105234 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098361 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:05:49.105234 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098363 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:05:49.105234 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098366 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:05:49.106070 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098368 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:05:49.106070 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098371 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:05:49.106070 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098373 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:05:49.106070 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098376 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:05:49.106070 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098378 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:05:49.106070 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098381 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:05:49.106070 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098383 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:05:49.106070 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098386 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:05:49.106070 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098389 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:05:49.106070 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098391 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:05:49.106070 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098394 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:05:49.106070 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098396 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:05:49.106070 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098399 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:05:49.106070 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098402 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:05:49.106070 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098405 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:05:49.106070 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098408 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:05:49.106070 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098411 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:05:49.106070 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098413 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:05:49.106070 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098415 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:05:49.106070 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098418 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:05:49.106674 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098420 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:05:49.106674 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098423 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:05:49.106674 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098426 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:05:49.106674 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098428 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:05:49.106674 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098431 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:05:49.106674 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098433 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:05:49.106674 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098436 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:05:49.106674 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098439 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:05:49.106674 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098441 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:05:49.106674 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098444 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:05:49.106674 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098446 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:05:49.106674 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098449 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:05:49.106674 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098466 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:05:49.106674 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098469 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:05:49.106674 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098472 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:05:49.106674 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098474 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:05:49.106674 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098477 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:05:49.106674 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098479 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:05:49.106674 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098482 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:05:49.107142 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098485 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:05:49.107142 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098487 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:05:49.107142 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098489 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:05:49.107142 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098493 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:05:49.107142 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098497 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:05:49.107142 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098500 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:05:49.107142 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098504 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:05:49.107142 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098507 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:05:49.107142 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098510 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:05:49.107142 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098512 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:05:49.107142 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098515 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:05:49.107142 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098518 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:05:49.107142 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098521 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:05:49.107142 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098524 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:05:49.107142 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098527 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:05:49.107142 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098530 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:05:49.107142 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098532 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:05:49.107142 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098535 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:05:49.107142 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098538 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:05:49.107634 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098540 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:05:49.107634 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098543 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:05:49.107634 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098546 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:05:49.107634 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098548 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:05:49.107634 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098551 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:05:49.107634 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098553 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:05:49.107634 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:49.098557 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:05:49.107634 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.098562 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 20:05:49.107634 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.098665 2578 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 20:05:49.107634 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.100516 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 20:05:49.107634 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.101942 2578 server.go:1019] "Starting client certificate rotation" Jul 06 20:05:49.107634 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.102424 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 20:05:49.107634 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.102479 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 20:05:49.107968 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.104241 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 20:05:49.107968 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.106613 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 20:05:49.116699 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.116681 2578 log.go:25] "Validated CRI v1 runtime API" Jul 06 20:05:49.118701 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.118685 2578 log.go:25] "Validated CRI v1 image API" Jul 06 20:05:49.119637 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.119608 2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 20:05:49.121995 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.121972 2578 fs.go:135] Filesystem UUIDs: map[2cdb521c-850c-4879-afaa-a5b236825530:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 920bc349-c324-4027-8be3-24e38806ce40:/dev/nvme0n1p3] Jul 06 20:05:49.122065 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.121997 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 20:05:49.126545 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.126423 2578 manager.go:216] Machine: {Timestamp:2026-07-06 20:05:49.125937162 +0000 UTC m=+0.337085372 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100186 MemoryCapacity:32812183552 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2695c58ca22d0834c0aec31c56184e SystemUUID:ec2695c5-8ca2-2d08-34c0-aec31c56184e BootID:46a14ff4-556b-4f66-8441-45c1e087cda0 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406093824 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:95:0c:38:a8:4d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:95:0c:38:a8:4d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:d9:82:9d:84:d4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812183552 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 20:05:49.126545 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.126543 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 20:05:49.126660 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.126612 2578 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 20:05:49.128357 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.128336 2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 20:05:49.128507 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.128362 2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-218.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 20:05:49.129625 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.129615 2578 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 20:05:49.129666 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.129628 2578 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 20:05:49.129666 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.129637 2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 20:05:49.129666 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.129655 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 20:05:49.130793 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.130780 2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 20:05:49.134126 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.134114 2578 state_mem.go:36] "Initialized new in-memory state store" Jul 06 20:05:49.134251 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.134242 2578 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 20:05:49.134917 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.134907 2578 kubelet.go:486] "Attempting to sync node with API server" Jul 06 20:05:49.134953 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.134925 2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 20:05:49.134953 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.134945 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 20:05:49.135006 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.134957 2578 kubelet.go:398] "Adding apiserver pod source" Jul 06 20:05:49.135006 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.134986 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 20:05:49.135378 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.135363 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 20:05:49.135429 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.135388 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 20:05:49.137396 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.137380 2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 20:05:49.138265 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.138248 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 20:05:49.139234 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.139213 2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 20:05:49.139314 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.139255 2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 20:05:49.140560 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.140545 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 20:05:49.140643 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.140568 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 20:05:49.140643 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.140578 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 20:05:49.140643 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.140589 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 20:05:49.140643 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.140602 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 20:05:49.140643 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.140614 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 20:05:49.140643 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.140629 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 20:05:49.140643 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.140639 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 20:05:49.140861 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.140652 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 20:05:49.140861 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.140661 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 20:05:49.140861 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.140683 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 20:05:49.140861 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.140693 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 20:05:49.141374 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.141364 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 20:05:49.141428 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.141376 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 20:05:49.144578 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.144563 2578 server.go:1268] "Started kubelet" Jul 06 20:05:49.145365 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.145159 2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 20:05:49.146089 ip-10-0-131-218 systemd[1]: Started Kubernetes Kubelet. Jul 06 20:05:49.146811 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.146656 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 20:05:49.147180 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.146964 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 20:05:49.148761 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.148705 2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 20:05:49.149412 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:49.149385 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-218.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 20:05:49.149520 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.149439 2578 server.go:310] "Adding debug handlers to kubelet server" Jul 06 20:05:49.149520 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.149481 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-218.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 20:05:49.149621 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:49.149525 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 20:05:49.150862 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:49.150844 2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 20:05:49.153971 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.153955 2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 20:05:49.154052 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.153972 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 20:05:49.154052 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:49.154021 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-218.ec2.internal\" not found" Jul 06 20:05:49.154757 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.154738 2578 factory.go:55] Registering systemd factory Jul 06 20:05:49.154855 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.154762 2578 factory.go:223] Registration of the systemd container factory successfully Jul 06 20:05:49.154913 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.154887 2578 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 20:05:49.154913 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.154904 2578 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 20:05:49.155012 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.154943 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 20:05:49.155066 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.155021 2578 reconstruct.go:97] "Volume reconstruction finished" Jul 06 20:05:49.155066 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.155029 2578 reconciler.go:29] "Reconciler: start to sync state" Jul 06 20:05:49.155164 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.155152 2578 factory.go:153] Registering CRI-O factory Jul 06 20:05:49.155196 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.155167 2578 factory.go:223] Registration of the crio container factory successfully Jul 06 20:05:49.155223 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.155210 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 20:05:49.155255 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.155229 2578 factory.go:103] Registering Raw factory Jul 06 20:05:49.155255 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.155239 2578 manager.go:1196] Started watching for new ooms in manager Jul 06 20:05:49.155562 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.155552 2578 manager.go:349] Starting recovery of all containers Jul 06 20:05:49.157215 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:49.157192 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 20:05:49.157297 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:49.157250 2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-218.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 20:05:49.157833 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:49.156775 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-218.ec2.internal.18bfcc109a6f2436 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-218.ec2.internal,UID:ip-10-0-131-218.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-218.ec2.internal,},FirstTimestamp:2026-07-06 20:05:49.14452383 +0000 UTC m=+0.355672039,LastTimestamp:2026-07-06 20:05:49.14452383 +0000 UTC m=+0.355672039,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-218.ec2.internal,}" Jul 06 20:05:49.166341 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.166222 2578 manager.go:354] Recovery completed Jul 06 20:05:49.170593 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.170570 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8dvj2" Jul 06 20:05:49.170795 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.170783 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:05:49.172679 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.172658 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-218.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:05:49.172745 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.172700 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-218.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:05:49.172745 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.172716 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-218.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:05:49.173281 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.173266 2578 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 20:05:49.173337 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.173282 2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 20:05:49.173337 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.173301 2578 state_mem.go:36] "Initialized new in-memory state store" Jul 06 20:05:49.173337 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.173307 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:05:49.174046 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.174031 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-218.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:05:49.174139 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.174061 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-218.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:05:49.174139 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.174076 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-218.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:05:49.174512 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.174492 2578 policy_none.go:49] "None policy: Start" Jul 06 20:05:49.174512 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.174514 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 20:05:49.174649 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.174527 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 20:05:49.175659 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.175647 2578 policy_none.go:47] "Start" Jul 06 20:05:49.176559 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:49.176448 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-218.ec2.internal.18bfcc109c1cc78a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-218.ec2.internal,UID:ip-10-0-131-218.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-218.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-218.ec2.internal,},FirstTimestamp:2026-07-06 20:05:49.172680586 +0000 UTC m=+0.383828799,LastTimestamp:2026-07-06 20:05:49.172680586 +0000 UTC m=+0.383828799,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-218.ec2.internal,}" Jul 06 20:05:49.184327 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.184310 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8dvj2" Jul 06 20:05:49.186848 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:49.186791 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-218.ec2.internal.18bfcc109c1d31fd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-218.ec2.internal,UID:ip-10-0-131-218.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-218.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-218.ec2.internal,},FirstTimestamp:2026-07-06 20:05:49.172707837 +0000 UTC m=+0.383856049,LastTimestamp:2026-07-06 20:05:49.172707837 +0000 UTC m=+0.383856049,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-218.ec2.internal,}" Jul 06 20:05:49.190005 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:49.189939 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-218.ec2.internal.18bfcc109c1d6cc0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-218.ec2.internal,UID:ip-10-0-131-218.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-218.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-218.ec2.internal,},FirstTimestamp:2026-07-06 20:05:49.17272288 +0000 UTC m=+0.383871091,LastTimestamp:2026-07-06 20:05:49.17272288 +0000 UTC m=+0.383871091,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-218.ec2.internal,}" Jul 06 20:05:49.212019 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.211996 2578 manager.go:337] "Starting Device Plugin manager" Jul 06 20:05:49.212098 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:49.212033 2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 20:05:49.212098 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.212045 2578 server.go:87] "Starting device plugin registration server" Jul 06 20:05:49.212280 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.212268 2578 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 20:05:49.212345 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.212282 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 20:05:49.212388 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.212356 2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 20:05:49.212440 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.212423 2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 20:05:49.212440 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.212433 2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 20:05:49.213006 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:49.212989 2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 20:05:49.213074 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:49.213043 2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-218.ec2.internal\" not found" Jul 06 20:05:49.290970 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.290935 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 20:05:49.292265 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.292248 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 20:05:49.292312 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.292283 2578 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 20:05:49.292312 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.292305 2578 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 20:05:49.292391 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:49.292346 2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 20:05:49.294728 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.294704 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 20:05:49.312858 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.312827 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:05:49.313852 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.313831 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-218.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:05:49.313956 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.313869 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-218.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:05:49.313956 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.313885 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-218.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:05:49.313956 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.313916 2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-218.ec2.internal" Jul 06 20:05:49.324152 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.324134 2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-218.ec2.internal" Jul 06 20:05:49.324219 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:49.324155 2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-218.ec2.internal\": node \"ip-10-0-131-218.ec2.internal\" not found" Jul 06 20:05:49.351913 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:49.351863 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-218.ec2.internal\" not found" Jul 06 20:05:49.392770 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.392747 2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-218.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-218.ec2.internal"] Jul 06 20:05:49.392872 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.392811 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:05:49.394265 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.394250 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-218.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:05:49.394339 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.394280 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-218.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:05:49.394339 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.394289 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-218.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:05:49.395820 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.395808 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:05:49.395953 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.395940 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-218.ec2.internal" Jul 06 20:05:49.396003 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.395965 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:05:49.396472 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.396445 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-218.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:05:49.396541 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.396476 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-218.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:05:49.396541 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.396490 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-218.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:05:49.396541 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.396500 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-218.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:05:49.396541 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.396513 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-218.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:05:49.396673 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.396504 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-218.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:05:49.397772 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.397752 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-218.ec2.internal" Jul 06 20:05:49.397857 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.397780 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:05:49.398424 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.398407 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-218.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:05:49.398535 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.398432 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-218.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:05:49.398535 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.398448 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-218.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:05:49.413005 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:49.412983 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-218.ec2.internal\" not found" node="ip-10-0-131-218.ec2.internal" Jul 06 20:05:49.416764 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:49.416747 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-218.ec2.internal\" not found" node="ip-10-0-131-218.ec2.internal" Jul 06 20:05:49.452391 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:49.452361 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-218.ec2.internal\" not found" Jul 06 20:05:49.456744 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.456726 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8c77d522b11ae15bf067473201a7e234-config\") pod \"kube-apiserver-proxy-ip-10-0-131-218.ec2.internal\" (UID: \"8c77d522b11ae15bf067473201a7e234\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-218.ec2.internal" Jul 06 20:05:49.456815 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.456777 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9ef0663ed142b0a85050706048e46477-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-218.ec2.internal\" (UID: \"9ef0663ed142b0a85050706048e46477\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-218.ec2.internal" Jul 06 20:05:49.456815 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.456794 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ef0663ed142b0a85050706048e46477-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-218.ec2.internal\" (UID: \"9ef0663ed142b0a85050706048e46477\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-218.ec2.internal" Jul 06 20:05:49.553472 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:49.553427 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-218.ec2.internal\" not found" Jul 06 20:05:49.557809 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.557789 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9ef0663ed142b0a85050706048e46477-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-218.ec2.internal\" (UID: \"9ef0663ed142b0a85050706048e46477\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-218.ec2.internal" Jul 06 20:05:49.557881 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.557853 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9ef0663ed142b0a85050706048e46477-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-218.ec2.internal\" (UID: \"9ef0663ed142b0a85050706048e46477\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-218.ec2.internal" Jul 06 20:05:49.557918 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.557880 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ef0663ed142b0a85050706048e46477-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-218.ec2.internal\" (UID: \"9ef0663ed142b0a85050706048e46477\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-218.ec2.internal" Jul 06 20:05:49.557918 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.557905 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8c77d522b11ae15bf067473201a7e234-config\") pod \"kube-apiserver-proxy-ip-10-0-131-218.ec2.internal\" (UID: \"8c77d522b11ae15bf067473201a7e234\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-218.ec2.internal" Jul 06 20:05:49.557974 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.557931 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ef0663ed142b0a85050706048e46477-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-218.ec2.internal\" (UID: \"9ef0663ed142b0a85050706048e46477\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-218.ec2.internal" Jul 06 20:05:49.557974 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.557937 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8c77d522b11ae15bf067473201a7e234-config\") pod \"kube-apiserver-proxy-ip-10-0-131-218.ec2.internal\" (UID: \"8c77d522b11ae15bf067473201a7e234\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-218.ec2.internal" Jul 06 20:05:49.653945 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:49.653871 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-218.ec2.internal\" not found" Jul 06 20:05:49.715391 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.715354 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-218.ec2.internal" Jul 06 20:05:49.720110 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:49.719987 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-218.ec2.internal" Jul 06 20:05:49.754884 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:49.754850 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-218.ec2.internal\" not found" Jul 06 20:05:49.855395 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:49.855349 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-218.ec2.internal\" not found" Jul 06 20:05:49.955875 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:49.955807 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-218.ec2.internal\" not found" Jul 06 20:05:50.056950 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:50.056920 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-218.ec2.internal\" not found" Jul 06 20:05:50.101596 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:50.101569 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 20:05:50.102170 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:50.101730 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 20:05:50.157660 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:50.157633 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-218.ec2.internal\" not found" Jul 06 20:05:50.188592 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:50.188523 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 20:00:49 +0000 UTC" deadline="2028-02-04 05:46:25.424676168 +0000 UTC" Jul 06 20:05:50.188592 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:50.188575 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13857h40m35.236104578s" Jul 06 20:05:50.225009 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:50.224901 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef0663ed142b0a85050706048e46477.slice/crio-0bf3255ca680888f841e14f603c2e8ff743db600201cb6330340067cb9f92405 WatchSource:0}: Error finding container 0bf3255ca680888f841e14f603c2e8ff743db600201cb6330340067cb9f92405: Status 404 returned error can't find the container with id 0bf3255ca680888f841e14f603c2e8ff743db600201cb6330340067cb9f92405 Jul 06 20:05:50.228810 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:50.228793 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 20:05:50.246518 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:50.246497 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:05:50.247417 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:50.247401 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-218.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:05:50.247496 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:50.247431 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-218.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:05:50.247496 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:50.247442 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-218.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:05:50.258804 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:50.258783 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-218.ec2.internal\" not found" Jul 06 20:05:50.295109 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:50.295061 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-218.ec2.internal" event={"ID":"8c77d522b11ae15bf067473201a7e234","Type":"ContainerStarted","Data":"9fda96c99c560b07365fb2f27a519518e8755fb879fe93cb933f34768e44165c"} Jul 06 20:05:50.296267 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:50.296243 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-218.ec2.internal" event={"ID":"9ef0663ed142b0a85050706048e46477","Type":"ContainerStarted","Data":"0bf3255ca680888f841e14f603c2e8ff743db600201cb6330340067cb9f92405"} Jul 06 20:05:50.317310 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:50.317290 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 20:05:50.354806 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:50.354756 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-218.ec2.internal" Jul 06 20:05:50.388363 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:50.388337 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 20:05:50.390104 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:50.390087 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-218.ec2.internal" Jul 06 20:05:50.397019 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:50.397002 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 20:05:50.418355 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:50.418336 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 20:05:50.514035 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:50.513942 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 20:05:51.136954 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.136926 2578 apiserver.go:52] "Watching apiserver" Jul 06 20:05:51.153082 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.153041 2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 20:05:51.155301 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.155265 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-42g4g","openshift-dns/node-resolver-j4d6m","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-218.ec2.internal","openshift-network-diagnostics/network-check-target-mbz65","openshift-image-registry/node-ca-r6xsf","openshift-multus/multus-additional-cni-plugins-kgswg","openshift-multus/multus-kpft4","openshift-multus/network-metrics-daemon-5m47w","openshift-network-operator/iptables-alerter-h7j8j","openshift-ovn-kubernetes/ovnkube-node-dnfgp","kube-system/global-pull-secret-syncer-jmp82","kube-system/konnectivity-agent-5wjbn","kube-system/kube-apiserver-proxy-ip-10-0-131-218.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7dxp"] Jul 06 20:05:51.155425 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.155277 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 20:05:51.157002 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.156975 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.158160 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.158134 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:05:51.158276 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:51.158239 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmp82" podUID="4ef938dd-1047-4283-a4f5-d0d0abb4eda0" Jul 06 20:05:51.159659 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.159641 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7dxp" Jul 06 20:05:51.161478 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.161441 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbz65" Jul 06 20:05:51.161586 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:51.161515 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbz65" podUID="e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e" Jul 06 20:05:51.163321 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.162953 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r6xsf" Jul 06 20:05:51.163321 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.162999 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 20:05:51.163321 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.163054 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 20:05:51.163321 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.163003 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 20:05:51.164504 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.164487 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kgswg" Jul 06 20:05:51.165792 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.165775 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h7j8j" Jul 06 20:05:51.166268 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.166244 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 20:05:51.166342 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.166307 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2dt99\"" Jul 06 20:05:51.166409 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.166385 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 20:05:51.166669 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.166650 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 20:05:51.167273 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.167199 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5m47w" Jul 06 20:05:51.168616 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:51.167567 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5m47w" podUID="d7f13799-7290-425e-be8d-b0bef20ea3c4" Jul 06 20:05:51.168616 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.167835 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 20:05:51.170358 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.170335 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 20:05:51.171054 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.170654 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 20:05:51.171054 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.170799 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7pxfd\"" Jul 06 20:05:51.171294 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.171271 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j4d6m" Jul 06 20:05:51.171763 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.171627 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wf7sp\"" Jul 06 20:05:51.172584 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.172565 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-d9v86\"" Jul 06 20:05:51.173140 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.173119 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.174067 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.174043 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 20:05:51.174163 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.174127 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 20:05:51.174288 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.174271 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 20:05:51.174357 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.174279 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xr74p\"" Jul 06 20:05:51.174410 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.174359 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 20:05:51.174771 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.174751 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.177516 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.177481 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5wjbn" Jul 06 20:05:51.178851 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.178829 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 20:05:51.178950 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.178839 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 20:05:51.179015 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.178971 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 20:05:51.179015 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.179006 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 20:05:51.179123 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.179039 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 20:05:51.180385 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.180094 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 20:05:51.180385 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.180114 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 20:05:51.180385 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.180133 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 20:05:51.180385 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.180166 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-glx27\"" Jul 06 20:05:51.180385 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.180186 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 20:05:51.180385 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.180333 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2fsmw\"" Jul 06 20:05:51.180772 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.180545 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 20:05:51.188317 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.188299 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 20:05:51.193455 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.193436 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 20:05:51.193711 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.193688 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-g847p\"" Jul 06 20:05:51.196847 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.196825 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 20:05:51.196951 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.196898 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 20:05:51.197008 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.196947 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-nlvmv\"" Jul 06 20:05:51.198762 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.198745 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 20:05:51.220954 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.220934 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mjl7k" Jul 06 20:05:51.235950 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.235924 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mjl7k" Jul 06 20:05:51.256145 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.256119 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 20:05:51.268081 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.268052 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-var-lib-openvswitch\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.268199 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.268096 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-host-cni-bin\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.268199 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.268120 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-system-cni-dir\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.268199 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.268141 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hh7n\" (UniqueName: \"kubernetes.io/projected/87026d58-c6e6-4aea-8926-5b1a8cb6d71a-kube-api-access-4hh7n\") pod \"multus-additional-cni-plugins-kgswg\" (UID: \"87026d58-c6e6-4aea-8926-5b1a8cb6d71a\") " pod="openshift-multus/multus-additional-cni-plugins-kgswg" Jul 06 20:05:51.268366 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.268202 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2a3ceee0-1606-499a-800e-f8ce32884427-device-dir\") pod \"aws-ebs-csi-driver-node-k7dxp\" (UID: \"2a3ceee0-1606-499a-800e-f8ce32884427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7dxp" Jul 06 20:05:51.268366 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.268252 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-host-kubelet\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.268366 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.268291 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-host-run-multus-certs\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.268366 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.268350 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4ef938dd-1047-4283-a4f5-d0d0abb4eda0-dbus\") pod \"global-pull-secret-syncer-jmp82\" (UID: \"4ef938dd-1047-4283-a4f5-d0d0abb4eda0\") " pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:05:51.268660 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.268392 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt56s\" (UniqueName: \"kubernetes.io/projected/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-kube-api-access-xt56s\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.268660 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.268412 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/404225de-b80e-4320-9e75-f93046c00433-etc-modprobe-d\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.268660 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.268497 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/404225de-b80e-4320-9e75-f93046c00433-etc-sysctl-d\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.268660 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.268541 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-cnibin\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.268660 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.268583 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-os-release\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.268660 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.268607 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-host-slash\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.268660 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.268631 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-run-ovn\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.268660 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.268657 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-host-cni-netd\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.269050 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.268684 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-ovnkube-script-lib\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.269050 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.268713 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/404225de-b80e-4320-9e75-f93046c00433-lib-modules\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.269050 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.268752 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/87026d58-c6e6-4aea-8926-5b1a8cb6d71a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kgswg\" (UID: \"87026d58-c6e6-4aea-8926-5b1a8cb6d71a\") " pod="openshift-multus/multus-additional-cni-plugins-kgswg" Jul 06 20:05:51.269050 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.268776 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qghk\" (UniqueName: \"kubernetes.io/projected/0d2af31a-954a-4a78-861b-ddaa83e8df1a-kube-api-access-7qghk\") pod \"iptables-alerter-h7j8j\" (UID: \"0d2af31a-954a-4a78-861b-ddaa83e8df1a\") " pod="openshift-network-operator/iptables-alerter-h7j8j" Jul 06 20:05:51.269050 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.268803 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/404225de-b80e-4320-9e75-f93046c00433-etc-kubernetes\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.269050 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.268822 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqx4z\" (UniqueName: \"kubernetes.io/projected/e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e-kube-api-access-dqx4z\") pod \"network-check-target-mbz65\" (UID: \"e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e\") " pod="openshift-network-diagnostics/network-check-target-mbz65" Jul 06 20:05:51.269050 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.268847 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-systemd-units\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.269050 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.268879 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/404225de-b80e-4320-9e75-f93046c00433-etc-sysctl-conf\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.269050 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.268935 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2a3ceee0-1606-499a-800e-f8ce32884427-etc-selinux\") pod \"aws-ebs-csi-driver-node-k7dxp\" (UID: \"2a3ceee0-1606-499a-800e-f8ce32884427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7dxp" Jul 06 20:05:51.269050 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.268965 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.269050 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.268996 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-host-run-ovn-kubernetes\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.269050 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.269020 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-env-overrides\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.269050 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.269043 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4ef938dd-1047-4283-a4f5-d0d0abb4eda0-kubelet-config\") pod \"global-pull-secret-syncer-jmp82\" (UID: \"4ef938dd-1047-4283-a4f5-d0d0abb4eda0\") " pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:05:51.269639 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.269065 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-host-var-lib-cni-bin\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.269639 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.269125 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/82df8708-d324-4b4d-8c74-1a65b6461394-multus-daemon-config\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.269639 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.269151 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-etc-kubernetes\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.269639 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.269177 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/87026d58-c6e6-4aea-8926-5b1a8cb6d71a-os-release\") pod \"multus-additional-cni-plugins-kgswg\" (UID: \"87026d58-c6e6-4aea-8926-5b1a8cb6d71a\") " pod="openshift-multus/multus-additional-cni-plugins-kgswg" Jul 06 20:05:51.269639 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.269206 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/87026d58-c6e6-4aea-8926-5b1a8cb6d71a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kgswg\" (UID: \"87026d58-c6e6-4aea-8926-5b1a8cb6d71a\") " pod="openshift-multus/multus-additional-cni-plugins-kgswg" Jul 06 20:05:51.269639 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.269223 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/24dd4f5b-0cda-4bb0-b776-72712fb30f3c-agent-certs\") pod \"konnectivity-agent-5wjbn\" (UID: \"24dd4f5b-0cda-4bb0-b776-72712fb30f3c\") " pod="kube-system/konnectivity-agent-5wjbn" Jul 06 20:05:51.269639 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.269259 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2a3ceee0-1606-499a-800e-f8ce32884427-registration-dir\") pod \"aws-ebs-csi-driver-node-k7dxp\" (UID: \"2a3ceee0-1606-499a-800e-f8ce32884427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7dxp" Jul 06 20:05:51.269639 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.269296 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-log-socket\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.269639 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.269320 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/404225de-b80e-4320-9e75-f93046c00433-run\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.269639 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.269353 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-multus-socket-dir-parent\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.269639 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.269378 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-hostroot\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.269639 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.269401 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2a3ceee0-1606-499a-800e-f8ce32884427-socket-dir\") pod \"aws-ebs-csi-driver-node-k7dxp\" (UID: \"2a3ceee0-1606-499a-800e-f8ce32884427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7dxp" Jul 06 20:05:51.269639 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.269452 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xqrp\" (UniqueName: \"kubernetes.io/projected/4e950de4-fc27-492d-ab51-69255ef36a84-kube-api-access-7xqrp\") pod \"node-ca-r6xsf\" (UID: \"4e950de4-fc27-492d-ab51-69255ef36a84\") " pod="openshift-image-registry/node-ca-r6xsf" Jul 06 20:05:51.269639 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.269529 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f0b6b2dc-73c6-453c-9941-8738e48b4976-hosts-file\") pod \"node-resolver-j4d6m\" (UID: \"f0b6b2dc-73c6-453c-9941-8738e48b4976\") " pod="openshift-dns/node-resolver-j4d6m" Jul 06 20:05:51.269639 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.269557 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/404225de-b80e-4320-9e75-f93046c00433-var-lib-kubelet\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.269639 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.269600 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e950de4-fc27-492d-ab51-69255ef36a84-host\") pod \"node-ca-r6xsf\" (UID: \"4e950de4-fc27-492d-ab51-69255ef36a84\") " pod="openshift-image-registry/node-ca-r6xsf" Jul 06 20:05:51.270370 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.269627 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-etc-openvswitch\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.270370 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.269643 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82df8708-d324-4b4d-8c74-1a65b6461394-cni-binary-copy\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.270370 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.269689 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-host-var-lib-cni-multus\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.270370 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.269713 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f13799-7290-425e-be8d-b0bef20ea3c4-metrics-certs\") pod \"network-metrics-daemon-5m47w\" (UID: \"d7f13799-7290-425e-be8d-b0bef20ea3c4\") " pod="openshift-multus/network-metrics-daemon-5m47w" Jul 06 20:05:51.270370 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.269746 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2a3ceee0-1606-499a-800e-f8ce32884427-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k7dxp\" (UID: \"2a3ceee0-1606-499a-800e-f8ce32884427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7dxp" Jul 06 20:05:51.270370 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.269840 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0d2af31a-954a-4a78-861b-ddaa83e8df1a-iptables-alerter-script\") pod \"iptables-alerter-h7j8j\" (UID: \"0d2af31a-954a-4a78-861b-ddaa83e8df1a\") " pod="openshift-network-operator/iptables-alerter-h7j8j" Jul 06 20:05:51.270370 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.269880 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-run-openvswitch\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.270370 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.269907 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/404225de-b80e-4320-9e75-f93046c00433-etc-systemd\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.270370 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.269940 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/404225de-b80e-4320-9e75-f93046c00433-sys\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.270370 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.269968 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/404225de-b80e-4320-9e75-f93046c00433-host\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.270370 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.270006 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sgqn\" (UniqueName: \"kubernetes.io/projected/404225de-b80e-4320-9e75-f93046c00433-kube-api-access-4sgqn\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.270370 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.270034 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87026d58-c6e6-4aea-8926-5b1a8cb6d71a-cnibin\") pod \"multus-additional-cni-plugins-kgswg\" (UID: \"87026d58-c6e6-4aea-8926-5b1a8cb6d71a\") " pod="openshift-multus/multus-additional-cni-plugins-kgswg" Jul 06 20:05:51.270370 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.270059 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-host-run-netns\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.270370 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.270082 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv4w5\" (UniqueName: \"kubernetes.io/projected/82df8708-d324-4b4d-8c74-1a65b6461394-kube-api-access-xv4w5\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.270370 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.270115 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ef938dd-1047-4283-a4f5-d0d0abb4eda0-original-pull-secret\") pod \"global-pull-secret-syncer-jmp82\" (UID: \"4ef938dd-1047-4283-a4f5-d0d0abb4eda0\") " pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:05:51.270370 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.270157 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87026d58-c6e6-4aea-8926-5b1a8cb6d71a-system-cni-dir\") pod \"multus-additional-cni-plugins-kgswg\" (UID: \"87026d58-c6e6-4aea-8926-5b1a8cb6d71a\") " pod="openshift-multus/multus-additional-cni-plugins-kgswg" Jul 06 20:05:51.271132 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.270192 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87026d58-c6e6-4aea-8926-5b1a8cb6d71a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kgswg\" (UID: \"87026d58-c6e6-4aea-8926-5b1a8cb6d71a\") " pod="openshift-multus/multus-additional-cni-plugins-kgswg" Jul 06 20:05:51.271132 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.270228 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d2af31a-954a-4a78-861b-ddaa83e8df1a-host-slash\") pod \"iptables-alerter-h7j8j\" (UID: \"0d2af31a-954a-4a78-861b-ddaa83e8df1a\") " pod="openshift-network-operator/iptables-alerter-h7j8j" Jul 06 20:05:51.271132 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.270265 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-host-run-netns\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.271132 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.270300 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-ovn-node-metrics-cert\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.271132 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.270323 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/404225de-b80e-4320-9e75-f93046c00433-tmp\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.271132 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.270339 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-host-run-k8s-cni-cncf-io\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.271132 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.270355 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9d9z\" (UniqueName: \"kubernetes.io/projected/d7f13799-7290-425e-be8d-b0bef20ea3c4-kube-api-access-v9d9z\") pod \"network-metrics-daemon-5m47w\" (UID: \"d7f13799-7290-425e-be8d-b0bef20ea3c4\") " pod="openshift-multus/network-metrics-daemon-5m47w" Jul 06 20:05:51.271132 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.270374 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/24dd4f5b-0cda-4bb0-b776-72712fb30f3c-konnectivity-ca\") pod \"konnectivity-agent-5wjbn\" (UID: \"24dd4f5b-0cda-4bb0-b776-72712fb30f3c\") " pod="kube-system/konnectivity-agent-5wjbn" Jul 06 20:05:51.271132 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.270393 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2a3ceee0-1606-499a-800e-f8ce32884427-sys-fs\") pod \"aws-ebs-csi-driver-node-k7dxp\" (UID: \"2a3ceee0-1606-499a-800e-f8ce32884427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7dxp" Jul 06 20:05:51.271132 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.270407 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4e950de4-fc27-492d-ab51-69255ef36a84-serviceca\") pod \"node-ca-r6xsf\" (UID: \"4e950de4-fc27-492d-ab51-69255ef36a84\") " pod="openshift-image-registry/node-ca-r6xsf" Jul 06 20:05:51.271132 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.270436 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67jxb\" (UniqueName: \"kubernetes.io/projected/f0b6b2dc-73c6-453c-9941-8738e48b4976-kube-api-access-67jxb\") pod \"node-resolver-j4d6m\" (UID: \"f0b6b2dc-73c6-453c-9941-8738e48b4976\") " pod="openshift-dns/node-resolver-j4d6m" Jul 06 20:05:51.271132 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.270499 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-run-systemd\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.271132 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.270536 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-node-log\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.271132 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.270581 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/404225de-b80e-4320-9e75-f93046c00433-etc-sysconfig\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.271132 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.270605 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-multus-conf-dir\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.271132 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.270630 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87026d58-c6e6-4aea-8926-5b1a8cb6d71a-cni-binary-copy\") pod \"multus-additional-cni-plugins-kgswg\" (UID: \"87026d58-c6e6-4aea-8926-5b1a8cb6d71a\") " pod="openshift-multus/multus-additional-cni-plugins-kgswg" Jul 06 20:05:51.271786 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.270657 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thdsg\" (UniqueName: \"kubernetes.io/projected/2a3ceee0-1606-499a-800e-f8ce32884427-kube-api-access-thdsg\") pod \"aws-ebs-csi-driver-node-k7dxp\" (UID: \"2a3ceee0-1606-499a-800e-f8ce32884427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7dxp" Jul 06 20:05:51.271786 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.270708 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0b6b2dc-73c6-453c-9941-8738e48b4976-tmp-dir\") pod \"node-resolver-j4d6m\" (UID: \"f0b6b2dc-73c6-453c-9941-8738e48b4976\") " pod="openshift-dns/node-resolver-j4d6m" Jul 06 20:05:51.271786 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.270754 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-ovnkube-config\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.271786 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.270776 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/404225de-b80e-4320-9e75-f93046c00433-etc-tuned\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.271786 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.270796 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-multus-cni-dir\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.271786 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.270821 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-host-var-lib-kubelet\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.371548 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.371506 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.371760 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.371562 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-host-run-ovn-kubernetes\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.371760 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.371591 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-env-overrides\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.371760 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.371616 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4ef938dd-1047-4283-a4f5-d0d0abb4eda0-kubelet-config\") pod \"global-pull-secret-syncer-jmp82\" (UID: \"4ef938dd-1047-4283-a4f5-d0d0abb4eda0\") " pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:05:51.371760 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.371637 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-host-var-lib-cni-bin\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.371760 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.371634 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.371760 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.371647 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-host-run-ovn-kubernetes\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.371760 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.371659 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/82df8708-d324-4b4d-8c74-1a65b6461394-multus-daemon-config\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.371760 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.371722 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-etc-kubernetes\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.371760 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.371740 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4ef938dd-1047-4283-a4f5-d0d0abb4eda0-kubelet-config\") pod \"global-pull-secret-syncer-jmp82\" (UID: \"4ef938dd-1047-4283-a4f5-d0d0abb4eda0\") " pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:05:51.371760 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.371752 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/87026d58-c6e6-4aea-8926-5b1a8cb6d71a-os-release\") pod \"multus-additional-cni-plugins-kgswg\" (UID: \"87026d58-c6e6-4aea-8926-5b1a8cb6d71a\") " pod="openshift-multus/multus-additional-cni-plugins-kgswg" Jul 06 20:05:51.372268 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.371783 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-etc-kubernetes\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.372268 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.371743 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-host-var-lib-cni-bin\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.372268 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.371782 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/87026d58-c6e6-4aea-8926-5b1a8cb6d71a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kgswg\" (UID: \"87026d58-c6e6-4aea-8926-5b1a8cb6d71a\") " pod="openshift-multus/multus-additional-cni-plugins-kgswg" Jul 06 20:05:51.372268 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.371818 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/87026d58-c6e6-4aea-8926-5b1a8cb6d71a-os-release\") pod \"multus-additional-cni-plugins-kgswg\" (UID: \"87026d58-c6e6-4aea-8926-5b1a8cb6d71a\") " pod="openshift-multus/multus-additional-cni-plugins-kgswg" Jul 06 20:05:51.372268 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.371873 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/24dd4f5b-0cda-4bb0-b776-72712fb30f3c-agent-certs\") pod \"konnectivity-agent-5wjbn\" (UID: \"24dd4f5b-0cda-4bb0-b776-72712fb30f3c\") " pod="kube-system/konnectivity-agent-5wjbn" Jul 06 20:05:51.372268 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.371900 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2a3ceee0-1606-499a-800e-f8ce32884427-registration-dir\") pod \"aws-ebs-csi-driver-node-k7dxp\" (UID: \"2a3ceee0-1606-499a-800e-f8ce32884427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7dxp" Jul 06 20:05:51.372268 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.371948 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-log-socket\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.372268 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.371973 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/404225de-b80e-4320-9e75-f93046c00433-run\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.372268 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.371999 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-multus-socket-dir-parent\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.372268 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372025 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-hostroot\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.372268 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372052 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2a3ceee0-1606-499a-800e-f8ce32884427-socket-dir\") pod \"aws-ebs-csi-driver-node-k7dxp\" (UID: \"2a3ceee0-1606-499a-800e-f8ce32884427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7dxp" Jul 06 20:05:51.372268 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372078 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xqrp\" (UniqueName: \"kubernetes.io/projected/4e950de4-fc27-492d-ab51-69255ef36a84-kube-api-access-7xqrp\") pod \"node-ca-r6xsf\" (UID: \"4e950de4-fc27-492d-ab51-69255ef36a84\") " pod="openshift-image-registry/node-ca-r6xsf" Jul 06 20:05:51.372268 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372104 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f0b6b2dc-73c6-453c-9941-8738e48b4976-hosts-file\") pod \"node-resolver-j4d6m\" (UID: \"f0b6b2dc-73c6-453c-9941-8738e48b4976\") " pod="openshift-dns/node-resolver-j4d6m" Jul 06 20:05:51.372268 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372134 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/404225de-b80e-4320-9e75-f93046c00433-var-lib-kubelet\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.372268 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372159 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e950de4-fc27-492d-ab51-69255ef36a84-host\") pod \"node-ca-r6xsf\" (UID: \"4e950de4-fc27-492d-ab51-69255ef36a84\") " pod="openshift-image-registry/node-ca-r6xsf" Jul 06 20:05:51.372268 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372191 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-etc-openvswitch\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.372268 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372216 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82df8708-d324-4b4d-8c74-1a65b6461394-cni-binary-copy\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.373159 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372242 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-host-var-lib-cni-multus\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.373159 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372246 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-env-overrides\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.373159 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372289 2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 20:05:51.373159 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372337 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f13799-7290-425e-be8d-b0bef20ea3c4-metrics-certs\") pod \"network-metrics-daemon-5m47w\" (UID: \"d7f13799-7290-425e-be8d-b0bef20ea3c4\") " pod="openshift-multus/network-metrics-daemon-5m47w" Jul 06 20:05:51.373159 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372374 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2a3ceee0-1606-499a-800e-f8ce32884427-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k7dxp\" (UID: \"2a3ceee0-1606-499a-800e-f8ce32884427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7dxp" Jul 06 20:05:51.373159 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372373 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/82df8708-d324-4b4d-8c74-1a65b6461394-multus-daemon-config\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.373159 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372388 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2a3ceee0-1606-499a-800e-f8ce32884427-socket-dir\") pod \"aws-ebs-csi-driver-node-k7dxp\" (UID: \"2a3ceee0-1606-499a-800e-f8ce32884427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7dxp" Jul 06 20:05:51.373159 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372395 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0d2af31a-954a-4a78-861b-ddaa83e8df1a-iptables-alerter-script\") pod \"iptables-alerter-h7j8j\" (UID: \"0d2af31a-954a-4a78-861b-ddaa83e8df1a\") " pod="openshift-network-operator/iptables-alerter-h7j8j" Jul 06 20:05:51.373159 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372435 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-run-openvswitch\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.373159 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372430 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/87026d58-c6e6-4aea-8926-5b1a8cb6d71a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kgswg\" (UID: \"87026d58-c6e6-4aea-8926-5b1a8cb6d71a\") " pod="openshift-multus/multus-additional-cni-plugins-kgswg" Jul 06 20:05:51.373159 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372460 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/404225de-b80e-4320-9e75-f93046c00433-etc-systemd\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.373159 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372486 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/404225de-b80e-4320-9e75-f93046c00433-sys\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.373159 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372505 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-log-socket\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.373159 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372511 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/404225de-b80e-4320-9e75-f93046c00433-host\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.373159 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372552 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f0b6b2dc-73c6-453c-9941-8738e48b4976-hosts-file\") pod \"node-resolver-j4d6m\" (UID: \"f0b6b2dc-73c6-453c-9941-8738e48b4976\") " pod="openshift-dns/node-resolver-j4d6m" Jul 06 20:05:51.373159 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372570 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sgqn\" (UniqueName: \"kubernetes.io/projected/404225de-b80e-4320-9e75-f93046c00433-kube-api-access-4sgqn\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.373159 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372577 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-run-openvswitch\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.373159 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372582 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/404225de-b80e-4320-9e75-f93046c00433-run\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.374188 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372442 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e950de4-fc27-492d-ab51-69255ef36a84-host\") pod \"node-ca-r6xsf\" (UID: \"4e950de4-fc27-492d-ab51-69255ef36a84\") " pod="openshift-image-registry/node-ca-r6xsf" Jul 06 20:05:51.374188 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372597 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87026d58-c6e6-4aea-8926-5b1a8cb6d71a-cnibin\") pod \"multus-additional-cni-plugins-kgswg\" (UID: \"87026d58-c6e6-4aea-8926-5b1a8cb6d71a\") " pod="openshift-multus/multus-additional-cni-plugins-kgswg" Jul 06 20:05:51.374188 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372610 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/404225de-b80e-4320-9e75-f93046c00433-host\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.374188 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372608 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-hostroot\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.374188 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372629 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87026d58-c6e6-4aea-8926-5b1a8cb6d71a-cnibin\") pod \"multus-additional-cni-plugins-kgswg\" (UID: \"87026d58-c6e6-4aea-8926-5b1a8cb6d71a\") " pod="openshift-multus/multus-additional-cni-plugins-kgswg" Jul 06 20:05:51.374188 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372642 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/404225de-b80e-4320-9e75-f93046c00433-etc-systemd\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.374188 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372655 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-multus-socket-dir-parent\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.374188 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372667 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/404225de-b80e-4320-9e75-f93046c00433-var-lib-kubelet\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.374188 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372684 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-host-run-netns\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.374188 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372696 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/404225de-b80e-4320-9e75-f93046c00433-sys\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.374188 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372706 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-etc-openvswitch\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.374188 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372710 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv4w5\" (UniqueName: \"kubernetes.io/projected/82df8708-d324-4b4d-8c74-1a65b6461394-kube-api-access-xv4w5\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.374188 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372740 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-host-run-netns\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.374188 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372739 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ef938dd-1047-4283-a4f5-d0d0abb4eda0-original-pull-secret\") pod \"global-pull-secret-syncer-jmp82\" (UID: \"4ef938dd-1047-4283-a4f5-d0d0abb4eda0\") " pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:05:51.374188 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372760 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2a3ceee0-1606-499a-800e-f8ce32884427-registration-dir\") pod \"aws-ebs-csi-driver-node-k7dxp\" (UID: \"2a3ceee0-1606-499a-800e-f8ce32884427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7dxp" Jul 06 20:05:51.374188 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372750 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2a3ceee0-1606-499a-800e-f8ce32884427-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k7dxp\" (UID: \"2a3ceee0-1606-499a-800e-f8ce32884427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7dxp" Jul 06 20:05:51.374188 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372775 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87026d58-c6e6-4aea-8926-5b1a8cb6d71a-system-cni-dir\") pod \"multus-additional-cni-plugins-kgswg\" (UID: \"87026d58-c6e6-4aea-8926-5b1a8cb6d71a\") " pod="openshift-multus/multus-additional-cni-plugins-kgswg" Jul 06 20:05:51.374188 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372799 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87026d58-c6e6-4aea-8926-5b1a8cb6d71a-system-cni-dir\") pod \"multus-additional-cni-plugins-kgswg\" (UID: \"87026d58-c6e6-4aea-8926-5b1a8cb6d71a\") " pod="openshift-multus/multus-additional-cni-plugins-kgswg" Jul 06 20:05:51.375077 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:51.372697 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:05:51.375077 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372815 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87026d58-c6e6-4aea-8926-5b1a8cb6d71a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kgswg\" (UID: \"87026d58-c6e6-4aea-8926-5b1a8cb6d71a\") " pod="openshift-multus/multus-additional-cni-plugins-kgswg" Jul 06 20:05:51.375077 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372844 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d2af31a-954a-4a78-861b-ddaa83e8df1a-host-slash\") pod \"iptables-alerter-h7j8j\" (UID: \"0d2af31a-954a-4a78-861b-ddaa83e8df1a\") " pod="openshift-network-operator/iptables-alerter-h7j8j" Jul 06 20:05:51.375077 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:51.372864 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7f13799-7290-425e-be8d-b0bef20ea3c4-metrics-certs podName:d7f13799-7290-425e-be8d-b0bef20ea3c4 nodeName:}" failed. No retries permitted until 2026-07-06 20:05:51.872844344 +0000 UTC m=+3.083992543 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7f13799-7290-425e-be8d-b0bef20ea3c4-metrics-certs") pod "network-metrics-daemon-5m47w" (UID: "d7f13799-7290-425e-be8d-b0bef20ea3c4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:05:51.375077 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372879 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d2af31a-954a-4a78-861b-ddaa83e8df1a-host-slash\") pod \"iptables-alerter-h7j8j\" (UID: \"0d2af31a-954a-4a78-861b-ddaa83e8df1a\") " pod="openshift-network-operator/iptables-alerter-h7j8j" Jul 06 20:05:51.375077 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372891 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-host-run-netns\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.375077 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:51.372803 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:05:51.375077 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372930 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-host-run-netns\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.375077 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:51.372939 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ef938dd-1047-4283-a4f5-d0d0abb4eda0-original-pull-secret podName:4ef938dd-1047-4283-a4f5-d0d0abb4eda0 nodeName:}" failed. No retries permitted until 2026-07-06 20:05:51.872928356 +0000 UTC m=+3.084076553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4ef938dd-1047-4283-a4f5-d0d0abb4eda0-original-pull-secret") pod "global-pull-secret-syncer-jmp82" (UID: "4ef938dd-1047-4283-a4f5-d0d0abb4eda0") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:05:51.375077 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372299 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-host-var-lib-cni-multus\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.375077 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372954 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87026d58-c6e6-4aea-8926-5b1a8cb6d71a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kgswg\" (UID: \"87026d58-c6e6-4aea-8926-5b1a8cb6d71a\") " pod="openshift-multus/multus-additional-cni-plugins-kgswg" Jul 06 20:05:51.375077 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372959 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-ovn-node-metrics-cert\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.375077 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.372993 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/404225de-b80e-4320-9e75-f93046c00433-tmp\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.375077 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373019 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-host-run-k8s-cni-cncf-io\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.375077 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373045 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9d9z\" (UniqueName: \"kubernetes.io/projected/d7f13799-7290-425e-be8d-b0bef20ea3c4-kube-api-access-v9d9z\") pod \"network-metrics-daemon-5m47w\" (UID: \"d7f13799-7290-425e-be8d-b0bef20ea3c4\") " pod="openshift-multus/network-metrics-daemon-5m47w" Jul 06 20:05:51.375077 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373074 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/24dd4f5b-0cda-4bb0-b776-72712fb30f3c-konnectivity-ca\") pod \"konnectivity-agent-5wjbn\" (UID: \"24dd4f5b-0cda-4bb0-b776-72712fb30f3c\") " pod="kube-system/konnectivity-agent-5wjbn" Jul 06 20:05:51.375077 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373088 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-host-run-k8s-cni-cncf-io\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.376005 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373099 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2a3ceee0-1606-499a-800e-f8ce32884427-sys-fs\") pod \"aws-ebs-csi-driver-node-k7dxp\" (UID: \"2a3ceee0-1606-499a-800e-f8ce32884427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7dxp" Jul 06 20:05:51.376005 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373141 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4e950de4-fc27-492d-ab51-69255ef36a84-serviceca\") pod \"node-ca-r6xsf\" (UID: \"4e950de4-fc27-492d-ab51-69255ef36a84\") " pod="openshift-image-registry/node-ca-r6xsf" Jul 06 20:05:51.376005 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373147 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2a3ceee0-1606-499a-800e-f8ce32884427-sys-fs\") pod \"aws-ebs-csi-driver-node-k7dxp\" (UID: \"2a3ceee0-1606-499a-800e-f8ce32884427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7dxp" Jul 06 20:05:51.376005 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373169 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67jxb\" (UniqueName: \"kubernetes.io/projected/f0b6b2dc-73c6-453c-9941-8738e48b4976-kube-api-access-67jxb\") pod \"node-resolver-j4d6m\" (UID: \"f0b6b2dc-73c6-453c-9941-8738e48b4976\") " pod="openshift-dns/node-resolver-j4d6m" Jul 06 20:05:51.376005 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373195 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-run-systemd\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.376005 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373227 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-node-log\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.376005 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373252 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/404225de-b80e-4320-9e75-f93046c00433-etc-sysconfig\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.376005 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373278 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-multus-conf-dir\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.376005 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373305 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87026d58-c6e6-4aea-8926-5b1a8cb6d71a-cni-binary-copy\") pod \"multus-additional-cni-plugins-kgswg\" (UID: \"87026d58-c6e6-4aea-8926-5b1a8cb6d71a\") " pod="openshift-multus/multus-additional-cni-plugins-kgswg" Jul 06 20:05:51.376005 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373342 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thdsg\" (UniqueName: \"kubernetes.io/projected/2a3ceee0-1606-499a-800e-f8ce32884427-kube-api-access-thdsg\") pod \"aws-ebs-csi-driver-node-k7dxp\" (UID: \"2a3ceee0-1606-499a-800e-f8ce32884427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7dxp" Jul 06 20:05:51.376005 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373345 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82df8708-d324-4b4d-8c74-1a65b6461394-cni-binary-copy\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.376005 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373373 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0b6b2dc-73c6-453c-9941-8738e48b4976-tmp-dir\") pod \"node-resolver-j4d6m\" (UID: \"f0b6b2dc-73c6-453c-9941-8738e48b4976\") " pod="openshift-dns/node-resolver-j4d6m" Jul 06 20:05:51.376005 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373392 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-node-log\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.376005 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373401 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-run-systemd\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.376005 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373429 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-ovnkube-config\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.376005 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373458 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/404225de-b80e-4320-9e75-f93046c00433-etc-tuned\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.376005 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373483 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-multus-cni-dir\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.376005 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373477 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-multus-conf-dir\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.376838 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373531 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-host-var-lib-kubelet\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.376838 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373570 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/404225de-b80e-4320-9e75-f93046c00433-etc-sysconfig\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.376838 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373584 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-var-lib-openvswitch\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.376838 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373611 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-host-cni-bin\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.376838 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373639 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-system-cni-dir\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.376838 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373674 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hh7n\" (UniqueName: \"kubernetes.io/projected/87026d58-c6e6-4aea-8926-5b1a8cb6d71a-kube-api-access-4hh7n\") pod \"multus-additional-cni-plugins-kgswg\" (UID: \"87026d58-c6e6-4aea-8926-5b1a8cb6d71a\") " pod="openshift-multus/multus-additional-cni-plugins-kgswg" Jul 06 20:05:51.376838 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373703 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2a3ceee0-1606-499a-800e-f8ce32884427-device-dir\") pod \"aws-ebs-csi-driver-node-k7dxp\" (UID: \"2a3ceee0-1606-499a-800e-f8ce32884427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7dxp" Jul 06 20:05:51.376838 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373728 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-host-kubelet\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.376838 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373752 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-host-run-multus-certs\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.376838 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373765 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4e950de4-fc27-492d-ab51-69255ef36a84-serviceca\") pod \"node-ca-r6xsf\" (UID: \"4e950de4-fc27-492d-ab51-69255ef36a84\") " pod="openshift-image-registry/node-ca-r6xsf" Jul 06 20:05:51.376838 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373777 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4ef938dd-1047-4283-a4f5-d0d0abb4eda0-dbus\") pod \"global-pull-secret-syncer-jmp82\" (UID: \"4ef938dd-1047-4283-a4f5-d0d0abb4eda0\") " pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:05:51.376838 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373815 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt56s\" (UniqueName: \"kubernetes.io/projected/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-kube-api-access-xt56s\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.376838 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373843 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/404225de-b80e-4320-9e75-f93046c00433-etc-modprobe-d\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.376838 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373868 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/404225de-b80e-4320-9e75-f93046c00433-etc-sysctl-d\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.376838 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373896 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-cnibin\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.376838 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373923 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-os-release\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.376838 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373956 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-host-slash\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.376838 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373981 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-run-ovn\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.377507 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.374011 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-host-cni-netd\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.377507 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.374026 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-system-cni-dir\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.377507 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.374038 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-ovnkube-script-lib\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.377507 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.374039 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87026d58-c6e6-4aea-8926-5b1a8cb6d71a-cni-binary-copy\") pod \"multus-additional-cni-plugins-kgswg\" (UID: \"87026d58-c6e6-4aea-8926-5b1a8cb6d71a\") " pod="openshift-multus/multus-additional-cni-plugins-kgswg" Jul 06 20:05:51.377507 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.373341 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0d2af31a-954a-4a78-861b-ddaa83e8df1a-iptables-alerter-script\") pod \"iptables-alerter-h7j8j\" (UID: \"0d2af31a-954a-4a78-861b-ddaa83e8df1a\") " pod="openshift-network-operator/iptables-alerter-h7j8j" Jul 06 20:05:51.377507 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.374048 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-ovnkube-config\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.377507 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.374064 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/404225de-b80e-4320-9e75-f93046c00433-lib-modules\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.377507 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.374111 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-var-lib-openvswitch\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.377507 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.374111 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/87026d58-c6e6-4aea-8926-5b1a8cb6d71a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kgswg\" (UID: \"87026d58-c6e6-4aea-8926-5b1a8cb6d71a\") " pod="openshift-multus/multus-additional-cni-plugins-kgswg" Jul 06 20:05:51.377507 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.374160 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/404225de-b80e-4320-9e75-f93046c00433-etc-sysctl-d\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.377507 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.374189 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qghk\" (UniqueName: \"kubernetes.io/projected/0d2af31a-954a-4a78-861b-ddaa83e8df1a-kube-api-access-7qghk\") pod \"iptables-alerter-h7j8j\" (UID: \"0d2af31a-954a-4a78-861b-ddaa83e8df1a\") " pod="openshift-network-operator/iptables-alerter-h7j8j" Jul 06 20:05:51.377507 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.374225 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/404225de-b80e-4320-9e75-f93046c00433-etc-kubernetes\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.377507 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.374252 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqx4z\" (UniqueName: \"kubernetes.io/projected/e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e-kube-api-access-dqx4z\") pod \"network-check-target-mbz65\" (UID: \"e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e\") " pod="openshift-network-diagnostics/network-check-target-mbz65" Jul 06 20:05:51.377507 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.374271 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-systemd-units\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.377507 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.374312 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/404225de-b80e-4320-9e75-f93046c00433-etc-sysctl-conf\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.377507 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.374332 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2a3ceee0-1606-499a-800e-f8ce32884427-etc-selinux\") pod \"aws-ebs-csi-driver-node-k7dxp\" (UID: \"2a3ceee0-1606-499a-800e-f8ce32884427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7dxp" Jul 06 20:05:51.377507 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.374402 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2a3ceee0-1606-499a-800e-f8ce32884427-device-dir\") pod \"aws-ebs-csi-driver-node-k7dxp\" (UID: \"2a3ceee0-1606-499a-800e-f8ce32884427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7dxp" Jul 06 20:05:51.378157 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.374455 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-host-kubelet\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.378157 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.374479 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2a3ceee0-1606-499a-800e-f8ce32884427-etc-selinux\") pod \"aws-ebs-csi-driver-node-k7dxp\" (UID: \"2a3ceee0-1606-499a-800e-f8ce32884427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7dxp" Jul 06 20:05:51.378157 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.374502 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-host-run-multus-certs\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.378157 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.374559 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-cnibin\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.378157 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.374581 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-multus-cni-dir\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.378157 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.374626 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-os-release\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.378157 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.374670 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/404225de-b80e-4320-9e75-f93046c00433-etc-modprobe-d\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.378157 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.374675 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-host-slash\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.378157 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.374751 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-run-ovn\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.378157 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.374800 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-host-cni-netd\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.378157 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.374857 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/24dd4f5b-0cda-4bb0-b776-72712fb30f3c-konnectivity-ca\") pod \"konnectivity-agent-5wjbn\" (UID: \"24dd4f5b-0cda-4bb0-b776-72712fb30f3c\") " pod="kube-system/konnectivity-agent-5wjbn" Jul 06 20:05:51.378157 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.374918 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0b6b2dc-73c6-453c-9941-8738e48b4976-tmp-dir\") pod \"node-resolver-j4d6m\" (UID: \"f0b6b2dc-73c6-453c-9941-8738e48b4976\") " pod="openshift-dns/node-resolver-j4d6m" Jul 06 20:05:51.378157 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.374946 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/404225de-b80e-4320-9e75-f93046c00433-etc-kubernetes\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.378157 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.374962 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82df8708-d324-4b4d-8c74-1a65b6461394-host-var-lib-kubelet\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.378157 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.374921 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-host-cni-bin\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.378157 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.375028 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-systemd-units\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.378157 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.375076 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/404225de-b80e-4320-9e75-f93046c00433-lib-modules\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.378157 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.375134 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/404225de-b80e-4320-9e75-f93046c00433-etc-sysctl-conf\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.379017 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.375183 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4ef938dd-1047-4283-a4f5-d0d0abb4eda0-dbus\") pod \"global-pull-secret-syncer-jmp82\" (UID: \"4ef938dd-1047-4283-a4f5-d0d0abb4eda0\") " pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:05:51.379017 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.375324 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-ovnkube-script-lib\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.379017 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.375750 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/87026d58-c6e6-4aea-8926-5b1a8cb6d71a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kgswg\" (UID: \"87026d58-c6e6-4aea-8926-5b1a8cb6d71a\") " pod="openshift-multus/multus-additional-cni-plugins-kgswg" Jul 06 20:05:51.379017 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.376806 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/404225de-b80e-4320-9e75-f93046c00433-etc-tuned\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.379017 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.376981 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-ovn-node-metrics-cert\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.379017 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.377569 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/404225de-b80e-4320-9e75-f93046c00433-tmp\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.379017 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.378641 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/24dd4f5b-0cda-4bb0-b776-72712fb30f3c-agent-certs\") pod \"konnectivity-agent-5wjbn\" (UID: \"24dd4f5b-0cda-4bb0-b776-72712fb30f3c\") " pod="kube-system/konnectivity-agent-5wjbn" Jul 06 20:05:51.394595 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.394526 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sgqn\" (UniqueName: \"kubernetes.io/projected/404225de-b80e-4320-9e75-f93046c00433-kube-api-access-4sgqn\") pod \"tuned-42g4g\" (UID: \"404225de-b80e-4320-9e75-f93046c00433\") " pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.396486 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.396459 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv4w5\" (UniqueName: \"kubernetes.io/projected/82df8708-d324-4b4d-8c74-1a65b6461394-kube-api-access-xv4w5\") pod \"multus-kpft4\" (UID: \"82df8708-d324-4b4d-8c74-1a65b6461394\") " pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.397237 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.397209 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67jxb\" (UniqueName: \"kubernetes.io/projected/f0b6b2dc-73c6-453c-9941-8738e48b4976-kube-api-access-67jxb\") pod \"node-resolver-j4d6m\" (UID: \"f0b6b2dc-73c6-453c-9941-8738e48b4976\") " pod="openshift-dns/node-resolver-j4d6m" Jul 06 20:05:51.397524 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.397495 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9d9z\" (UniqueName: \"kubernetes.io/projected/d7f13799-7290-425e-be8d-b0bef20ea3c4-kube-api-access-v9d9z\") pod \"network-metrics-daemon-5m47w\" (UID: \"d7f13799-7290-425e-be8d-b0bef20ea3c4\") " pod="openshift-multus/network-metrics-daemon-5m47w" Jul 06 20:05:51.398543 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:51.398520 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:05:51.398645 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:51.398551 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:05:51.398645 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:51.398566 2578 projected.go:196] Error preparing data for projected volume kube-api-access-dqx4z for pod openshift-network-diagnostics/network-check-target-mbz65: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:05:51.398645 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:51.398627 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e-kube-api-access-dqx4z podName:e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e nodeName:}" failed. No retries permitted until 2026-07-06 20:05:51.898609678 +0000 UTC m=+3.109757878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dqx4z" (UniqueName: "kubernetes.io/projected/e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e-kube-api-access-dqx4z") pod "network-check-target-mbz65" (UID: "e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:05:51.398920 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.398897 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xqrp\" (UniqueName: \"kubernetes.io/projected/4e950de4-fc27-492d-ab51-69255ef36a84-kube-api-access-7xqrp\") pod \"node-ca-r6xsf\" (UID: \"4e950de4-fc27-492d-ab51-69255ef36a84\") " pod="openshift-image-registry/node-ca-r6xsf" Jul 06 20:05:51.401117 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.401091 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thdsg\" (UniqueName: \"kubernetes.io/projected/2a3ceee0-1606-499a-800e-f8ce32884427-kube-api-access-thdsg\") pod \"aws-ebs-csi-driver-node-k7dxp\" (UID: \"2a3ceee0-1606-499a-800e-f8ce32884427\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7dxp" Jul 06 20:05:51.402094 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.402075 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hh7n\" (UniqueName: \"kubernetes.io/projected/87026d58-c6e6-4aea-8926-5b1a8cb6d71a-kube-api-access-4hh7n\") pod \"multus-additional-cni-plugins-kgswg\" (UID: \"87026d58-c6e6-4aea-8926-5b1a8cb6d71a\") " pod="openshift-multus/multus-additional-cni-plugins-kgswg" Jul 06 20:05:51.402609 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.402584 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qghk\" (UniqueName: \"kubernetes.io/projected/0d2af31a-954a-4a78-861b-ddaa83e8df1a-kube-api-access-7qghk\") pod \"iptables-alerter-h7j8j\" (UID: \"0d2af31a-954a-4a78-861b-ddaa83e8df1a\") " pod="openshift-network-operator/iptables-alerter-h7j8j" Jul 06 20:05:51.402609 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.402604 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt56s\" (UniqueName: \"kubernetes.io/projected/0f4c3a8a-30c4-428e-88a6-dcf086ef8615-kube-api-access-xt56s\") pod \"ovnkube-node-dnfgp\" (UID: \"0f4c3a8a-30c4-428e-88a6-dcf086ef8615\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.467837 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.467800 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kpft4" Jul 06 20:05:51.479892 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.479862 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7dxp" Jul 06 20:05:51.486634 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.486611 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r6xsf" Jul 06 20:05:51.492269 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.492252 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kgswg" Jul 06 20:05:51.501579 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.501523 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h7j8j" Jul 06 20:05:51.509543 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.509505 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j4d6m" Jul 06 20:05:51.516331 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.516314 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:05:51.523646 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.523617 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-42g4g" Jul 06 20:05:51.528688 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.528580 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5wjbn" Jul 06 20:05:51.546036 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.546009 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 20:05:51.878918 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.878816 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f13799-7290-425e-be8d-b0bef20ea3c4-metrics-certs\") pod \"network-metrics-daemon-5m47w\" (UID: \"d7f13799-7290-425e-be8d-b0bef20ea3c4\") " pod="openshift-multus/network-metrics-daemon-5m47w" Jul 06 20:05:51.878918 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.878882 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ef938dd-1047-4283-a4f5-d0d0abb4eda0-original-pull-secret\") pod \"global-pull-secret-syncer-jmp82\" (UID: \"4ef938dd-1047-4283-a4f5-d0d0abb4eda0\") " pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:05:51.879142 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:51.879058 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:05:51.879142 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:51.879129 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ef938dd-1047-4283-a4f5-d0d0abb4eda0-original-pull-secret podName:4ef938dd-1047-4283-a4f5-d0d0abb4eda0 nodeName:}" failed. No retries permitted until 2026-07-06 20:05:52.879108747 +0000 UTC m=+4.090256959 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4ef938dd-1047-4283-a4f5-d0d0abb4eda0-original-pull-secret") pod "global-pull-secret-syncer-jmp82" (UID: "4ef938dd-1047-4283-a4f5-d0d0abb4eda0") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:05:51.879367 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:51.879332 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:05:51.879480 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:51.879430 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7f13799-7290-425e-be8d-b0bef20ea3c4-metrics-certs podName:d7f13799-7290-425e-be8d-b0bef20ea3c4 nodeName:}" failed. No retries permitted until 2026-07-06 20:05:52.879409665 +0000 UTC m=+4.090557879 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7f13799-7290-425e-be8d-b0bef20ea3c4-metrics-certs") pod "network-metrics-daemon-5m47w" (UID: "d7f13799-7290-425e-be8d-b0bef20ea3c4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:05:51.979828 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:51.979787 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqx4z\" (UniqueName: \"kubernetes.io/projected/e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e-kube-api-access-dqx4z\") pod \"network-check-target-mbz65\" (UID: \"e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e\") " pod="openshift-network-diagnostics/network-check-target-mbz65" Jul 06 20:05:51.980022 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:51.979937 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:05:51.980022 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:51.979955 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:05:51.980022 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:51.979965 2578 projected.go:196] Error preparing data for projected volume kube-api-access-dqx4z for pod openshift-network-diagnostics/network-check-target-mbz65: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:05:51.980199 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:51.980026 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e-kube-api-access-dqx4z podName:e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e nodeName:}" failed. No retries permitted until 2026-07-06 20:05:52.98001232 +0000 UTC m=+4.191160522 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dqx4z" (UniqueName: "kubernetes.io/projected/e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e-kube-api-access-dqx4z") pod "network-check-target-mbz65" (UID: "e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:05:52.239059 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:52.238981 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 20:00:51 +0000 UTC" deadline="2028-03-09 08:24:15.960666378 +0000 UTC" Jul 06 20:05:52.239059 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:52.239036 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14676h18m23.721633506s" Jul 06 20:05:52.243902 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:52.243868 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87026d58_c6e6_4aea_8926_5b1a8cb6d71a.slice/crio-7e58f45b95d03560ae95584634736293415b327b0ffb6aaa856ead83b0ec55e8 WatchSource:0}: Error finding container 7e58f45b95d03560ae95584634736293415b327b0ffb6aaa856ead83b0ec55e8: Status 404 returned error can't find the container with id 7e58f45b95d03560ae95584634736293415b327b0ffb6aaa856ead83b0ec55e8 Jul 06 20:05:52.247604 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:52.247446 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f4c3a8a_30c4_428e_88a6_dcf086ef8615.slice/crio-ed80d1c5c057b8b6bdced0ea9ce5c179b864bda387d26ed767f168c942b3de36 WatchSource:0}: Error finding container ed80d1c5c057b8b6bdced0ea9ce5c179b864bda387d26ed767f168c942b3de36: Status 404 returned error can't find the container with id ed80d1c5c057b8b6bdced0ea9ce5c179b864bda387d26ed767f168c942b3de36 Jul 06 20:05:52.251678 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:52.251641 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e950de4_fc27_492d_ab51_69255ef36a84.slice/crio-2fca4f460109235e685c60116707f2f7c767846b2e629f0fdb562c2aa0a3f05f WatchSource:0}: Error finding container 2fca4f460109235e685c60116707f2f7c767846b2e629f0fdb562c2aa0a3f05f: Status 404 returned error can't find the container with id 2fca4f460109235e685c60116707f2f7c767846b2e629f0fdb562c2aa0a3f05f Jul 06 20:05:52.252265 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:52.252236 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d2af31a_954a_4a78_861b_ddaa83e8df1a.slice/crio-34e8175d3bf170d4248df7b27c4d29a7ecc3ea6b909df4bc6da93ead2c5dc744 WatchSource:0}: Error finding container 34e8175d3bf170d4248df7b27c4d29a7ecc3ea6b909df4bc6da93ead2c5dc744: Status 404 returned error can't find the container with id 34e8175d3bf170d4248df7b27c4d29a7ecc3ea6b909df4bc6da93ead2c5dc744 Jul 06 20:05:52.254163 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:52.253861 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a3ceee0_1606_499a_800e_f8ce32884427.slice/crio-e13db76be69d03974f39a4a973ceab6e7b30a97825445c7fa9bb83ecc65c1ee7 WatchSource:0}: Error finding container e13db76be69d03974f39a4a973ceab6e7b30a97825445c7fa9bb83ecc65c1ee7: Status 404 returned error can't find the container with id e13db76be69d03974f39a4a973ceab6e7b30a97825445c7fa9bb83ecc65c1ee7 Jul 06 20:05:52.255553 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:52.255419 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b6b2dc_73c6_453c_9941_8738e48b4976.slice/crio-fe759598708cb3038b89c3dadcea6d65a2b74358833032c40be08a3355d2ce28 WatchSource:0}: Error finding container fe759598708cb3038b89c3dadcea6d65a2b74358833032c40be08a3355d2ce28: Status 404 returned error can't find the container with id fe759598708cb3038b89c3dadcea6d65a2b74358833032c40be08a3355d2ce28 Jul 06 20:05:52.255888 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:52.255855 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82df8708_d324_4b4d_8c74_1a65b6461394.slice/crio-33f6d0b94e8a7645a6ac25257c9eef5698f6d74a0155ccfccaf4aada156ce86b WatchSource:0}: Error finding container 33f6d0b94e8a7645a6ac25257c9eef5698f6d74a0155ccfccaf4aada156ce86b: Status 404 returned error can't find the container with id 33f6d0b94e8a7645a6ac25257c9eef5698f6d74a0155ccfccaf4aada156ce86b Jul 06 20:05:52.257521 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:52.257468 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24dd4f5b_0cda_4bb0_b776_72712fb30f3c.slice/crio-9d1c38c19c0c6608126f14f1d1266c1ba1fe3af9a764f283d22d35d1c88796d8 WatchSource:0}: Error finding container 9d1c38c19c0c6608126f14f1d1266c1ba1fe3af9a764f283d22d35d1c88796d8: Status 404 returned error can't find the container with id 9d1c38c19c0c6608126f14f1d1266c1ba1fe3af9a764f283d22d35d1c88796d8 Jul 06 20:05:52.261607 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:05:52.259962 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404225de_b80e_4320_9e75_f93046c00433.slice/crio-fc20b281b22a016d52ebb5313c05f4bc9301e7710f1653623871d115a8163f60 WatchSource:0}: Error finding container fc20b281b22a016d52ebb5313c05f4bc9301e7710f1653623871d115a8163f60: Status 404 returned error can't find the container with id fc20b281b22a016d52ebb5313c05f4bc9301e7710f1653623871d115a8163f60 Jul 06 20:05:52.293202 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:52.293058 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5m47w" Jul 06 20:05:52.293313 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:52.293106 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:05:52.293313 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:52.293292 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5m47w" podUID="d7f13799-7290-425e-be8d-b0bef20ea3c4" Jul 06 20:05:52.293426 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:52.293396 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmp82" podUID="4ef938dd-1047-4283-a4f5-d0d0abb4eda0" Jul 06 20:05:52.300136 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:52.300109 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-42g4g" event={"ID":"404225de-b80e-4320-9e75-f93046c00433","Type":"ContainerStarted","Data":"fc20b281b22a016d52ebb5313c05f4bc9301e7710f1653623871d115a8163f60"} Jul 06 20:05:52.301208 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:52.301185 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r6xsf" event={"ID":"4e950de4-fc27-492d-ab51-69255ef36a84","Type":"ContainerStarted","Data":"2fca4f460109235e685c60116707f2f7c767846b2e629f0fdb562c2aa0a3f05f"} Jul 06 20:05:52.303713 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:52.303685 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" event={"ID":"0f4c3a8a-30c4-428e-88a6-dcf086ef8615","Type":"ContainerStarted","Data":"ed80d1c5c057b8b6bdced0ea9ce5c179b864bda387d26ed767f168c942b3de36"} Jul 06 20:05:52.307111 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:52.307085 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kgswg" event={"ID":"87026d58-c6e6-4aea-8926-5b1a8cb6d71a","Type":"ContainerStarted","Data":"7e58f45b95d03560ae95584634736293415b327b0ffb6aaa856ead83b0ec55e8"} Jul 06 20:05:52.311491 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:52.311466 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5wjbn" event={"ID":"24dd4f5b-0cda-4bb0-b776-72712fb30f3c","Type":"ContainerStarted","Data":"9d1c38c19c0c6608126f14f1d1266c1ba1fe3af9a764f283d22d35d1c88796d8"} Jul 06 20:05:52.312622 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:52.312602 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j4d6m" event={"ID":"f0b6b2dc-73c6-453c-9941-8738e48b4976","Type":"ContainerStarted","Data":"fe759598708cb3038b89c3dadcea6d65a2b74358833032c40be08a3355d2ce28"} Jul 06 20:05:52.313792 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:52.313772 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7dxp" event={"ID":"2a3ceee0-1606-499a-800e-f8ce32884427","Type":"ContainerStarted","Data":"e13db76be69d03974f39a4a973ceab6e7b30a97825445c7fa9bb83ecc65c1ee7"} Jul 06 20:05:52.314969 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:52.314938 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h7j8j" event={"ID":"0d2af31a-954a-4a78-861b-ddaa83e8df1a","Type":"ContainerStarted","Data":"34e8175d3bf170d4248df7b27c4d29a7ecc3ea6b909df4bc6da93ead2c5dc744"} Jul 06 20:05:52.316304 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:52.316282 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kpft4" event={"ID":"82df8708-d324-4b4d-8c74-1a65b6461394","Type":"ContainerStarted","Data":"33f6d0b94e8a7645a6ac25257c9eef5698f6d74a0155ccfccaf4aada156ce86b"} Jul 06 20:05:52.887234 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:52.887147 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f13799-7290-425e-be8d-b0bef20ea3c4-metrics-certs\") pod \"network-metrics-daemon-5m47w\" (UID: \"d7f13799-7290-425e-be8d-b0bef20ea3c4\") " pod="openshift-multus/network-metrics-daemon-5m47w" Jul 06 20:05:52.887234 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:52.887215 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ef938dd-1047-4283-a4f5-d0d0abb4eda0-original-pull-secret\") pod \"global-pull-secret-syncer-jmp82\" (UID: \"4ef938dd-1047-4283-a4f5-d0d0abb4eda0\") " pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:05:52.887414 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:52.887345 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:05:52.887414 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:52.887353 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:05:52.887414 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:52.887411 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ef938dd-1047-4283-a4f5-d0d0abb4eda0-original-pull-secret podName:4ef938dd-1047-4283-a4f5-d0d0abb4eda0 nodeName:}" failed. No retries permitted until 2026-07-06 20:05:54.887392909 +0000 UTC m=+6.098541112 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4ef938dd-1047-4283-a4f5-d0d0abb4eda0-original-pull-secret") pod "global-pull-secret-syncer-jmp82" (UID: "4ef938dd-1047-4283-a4f5-d0d0abb4eda0") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:05:52.887560 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:52.887427 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7f13799-7290-425e-be8d-b0bef20ea3c4-metrics-certs podName:d7f13799-7290-425e-be8d-b0bef20ea3c4 nodeName:}" failed. No retries permitted until 2026-07-06 20:05:54.887420279 +0000 UTC m=+6.098568477 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7f13799-7290-425e-be8d-b0bef20ea3c4-metrics-certs") pod "network-metrics-daemon-5m47w" (UID: "d7f13799-7290-425e-be8d-b0bef20ea3c4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:05:52.988642 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:52.988597 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqx4z\" (UniqueName: \"kubernetes.io/projected/e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e-kube-api-access-dqx4z\") pod \"network-check-target-mbz65\" (UID: \"e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e\") " pod="openshift-network-diagnostics/network-check-target-mbz65" Jul 06 20:05:52.988825 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:52.988797 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:05:52.988825 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:52.988821 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:05:52.988945 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:52.988834 2578 projected.go:196] Error preparing data for projected volume kube-api-access-dqx4z for pod openshift-network-diagnostics/network-check-target-mbz65: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:05:52.988945 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:52.988903 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e-kube-api-access-dqx4z podName:e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e nodeName:}" failed. No retries permitted until 2026-07-06 20:05:54.988884315 +0000 UTC m=+6.200032530 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dqx4z" (UniqueName: "kubernetes.io/projected/e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e-kube-api-access-dqx4z") pod "network-check-target-mbz65" (UID: "e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:05:53.240353 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:53.239438 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 20:00:51 +0000 UTC" deadline="2028-01-01 06:48:05.992566092 +0000 UTC" Jul 06 20:05:53.240353 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:53.239495 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13042h42m12.753076093s" Jul 06 20:05:53.293850 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:53.293363 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbz65" Jul 06 20:05:53.293850 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:53.293547 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbz65" podUID="e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e" Jul 06 20:05:53.342008 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:53.341962 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-218.ec2.internal" event={"ID":"8c77d522b11ae15bf067473201a7e234","Type":"ContainerStarted","Data":"827f03835f910f6948effd75b62e0fa9d38a39cee1a310915a7a6a0e80b229b4"} Jul 06 20:05:53.357146 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:53.357108 2578 generic.go:356] "Generic (PLEG): container finished" podID="9ef0663ed142b0a85050706048e46477" containerID="49ccd2c2fa62e2bc698d749591608d82591b6c8c051923dd115aa3523dbbe412" exitCode=0 Jul 06 20:05:53.357322 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:53.357173 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-218.ec2.internal" event={"ID":"9ef0663ed142b0a85050706048e46477","Type":"ContainerDied","Data":"49ccd2c2fa62e2bc698d749591608d82591b6c8c051923dd115aa3523dbbe412"} Jul 06 20:05:53.388833 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:53.388774 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-218.ec2.internal" podStartSLOduration=2.388744463 podStartE2EDuration="2.388744463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:05:53.365672546 +0000 UTC m=+4.576820768" watchObservedRunningTime="2026-07-06 20:05:53.388744463 +0000 UTC m=+4.599892687" Jul 06 20:05:54.293489 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:54.293444 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5m47w" Jul 06 20:05:54.294002 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:54.293495 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:05:54.294002 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:54.293619 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5m47w" podUID="d7f13799-7290-425e-be8d-b0bef20ea3c4" Jul 06 20:05:54.294002 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:54.293718 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmp82" podUID="4ef938dd-1047-4283-a4f5-d0d0abb4eda0" Jul 06 20:05:54.372607 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:54.372576 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-218.ec2.internal" event={"ID":"9ef0663ed142b0a85050706048e46477","Type":"ContainerStarted","Data":"ba22a981aba11c5e662f152377acc709cb780c747e741d1a121e5288237d8bee"} Jul 06 20:05:54.916931 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:54.916889 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ef938dd-1047-4283-a4f5-d0d0abb4eda0-original-pull-secret\") pod \"global-pull-secret-syncer-jmp82\" (UID: \"4ef938dd-1047-4283-a4f5-d0d0abb4eda0\") " pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:05:54.917150 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:54.917003 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f13799-7290-425e-be8d-b0bef20ea3c4-metrics-certs\") pod \"network-metrics-daemon-5m47w\" (UID: \"d7f13799-7290-425e-be8d-b0bef20ea3c4\") " pod="openshift-multus/network-metrics-daemon-5m47w" Jul 06 20:05:54.917226 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:54.917145 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:05:54.917226 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:54.917213 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7f13799-7290-425e-be8d-b0bef20ea3c4-metrics-certs podName:d7f13799-7290-425e-be8d-b0bef20ea3c4 nodeName:}" failed. No retries permitted until 2026-07-06 20:05:58.917194596 +0000 UTC m=+10.128342796 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7f13799-7290-425e-be8d-b0bef20ea3c4-metrics-certs") pod "network-metrics-daemon-5m47w" (UID: "d7f13799-7290-425e-be8d-b0bef20ea3c4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:05:54.917400 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:54.917375 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:05:54.917523 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:54.917457 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ef938dd-1047-4283-a4f5-d0d0abb4eda0-original-pull-secret podName:4ef938dd-1047-4283-a4f5-d0d0abb4eda0 nodeName:}" failed. No retries permitted until 2026-07-06 20:05:58.91743634 +0000 UTC m=+10.128584543 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4ef938dd-1047-4283-a4f5-d0d0abb4eda0-original-pull-secret") pod "global-pull-secret-syncer-jmp82" (UID: "4ef938dd-1047-4283-a4f5-d0d0abb4eda0") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:05:55.017913 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:55.017868 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqx4z\" (UniqueName: \"kubernetes.io/projected/e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e-kube-api-access-dqx4z\") pod \"network-check-target-mbz65\" (UID: \"e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e\") " pod="openshift-network-diagnostics/network-check-target-mbz65" Jul 06 20:05:55.018124 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:55.018106 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:05:55.018188 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:55.018132 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:05:55.018188 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:55.018146 2578 projected.go:196] Error preparing data for projected volume kube-api-access-dqx4z for pod openshift-network-diagnostics/network-check-target-mbz65: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:05:55.018293 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:55.018212 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e-kube-api-access-dqx4z podName:e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e nodeName:}" failed. No retries permitted until 2026-07-06 20:05:59.018194216 +0000 UTC m=+10.229342435 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dqx4z" (UniqueName: "kubernetes.io/projected/e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e-kube-api-access-dqx4z") pod "network-check-target-mbz65" (UID: "e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:05:55.296270 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:55.295639 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbz65" Jul 06 20:05:55.296270 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:55.295780 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbz65" podUID="e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e" Jul 06 20:05:56.293362 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:56.293325 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5m47w" Jul 06 20:05:56.293578 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:56.293452 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5m47w" podUID="d7f13799-7290-425e-be8d-b0bef20ea3c4" Jul 06 20:05:56.294001 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:56.293325 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:05:56.294001 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:56.293938 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmp82" podUID="4ef938dd-1047-4283-a4f5-d0d0abb4eda0" Jul 06 20:05:57.295992 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:57.295909 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbz65" Jul 06 20:05:57.296446 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:57.296116 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbz65" podUID="e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e" Jul 06 20:05:58.292932 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:58.292895 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:05:58.293143 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:58.293024 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmp82" podUID="4ef938dd-1047-4283-a4f5-d0d0abb4eda0" Jul 06 20:05:58.293397 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:58.292853 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5m47w" Jul 06 20:05:58.293552 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:58.293510 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5m47w" podUID="d7f13799-7290-425e-be8d-b0bef20ea3c4" Jul 06 20:05:58.955948 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:58.955890 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f13799-7290-425e-be8d-b0bef20ea3c4-metrics-certs\") pod \"network-metrics-daemon-5m47w\" (UID: \"d7f13799-7290-425e-be8d-b0bef20ea3c4\") " pod="openshift-multus/network-metrics-daemon-5m47w" Jul 06 20:05:58.956409 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:58.955955 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ef938dd-1047-4283-a4f5-d0d0abb4eda0-original-pull-secret\") pod \"global-pull-secret-syncer-jmp82\" (UID: \"4ef938dd-1047-4283-a4f5-d0d0abb4eda0\") " pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:05:58.956409 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:58.956128 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:05:58.956409 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:58.956195 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ef938dd-1047-4283-a4f5-d0d0abb4eda0-original-pull-secret podName:4ef938dd-1047-4283-a4f5-d0d0abb4eda0 nodeName:}" failed. No retries permitted until 2026-07-06 20:06:06.956174333 +0000 UTC m=+18.167322534 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4ef938dd-1047-4283-a4f5-d0d0abb4eda0-original-pull-secret") pod "global-pull-secret-syncer-jmp82" (UID: "4ef938dd-1047-4283-a4f5-d0d0abb4eda0") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:05:58.956593 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:58.956577 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:05:58.956628 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:58.956621 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7f13799-7290-425e-be8d-b0bef20ea3c4-metrics-certs podName:d7f13799-7290-425e-be8d-b0bef20ea3c4 nodeName:}" failed. No retries permitted until 2026-07-06 20:06:06.956609046 +0000 UTC m=+18.167757243 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7f13799-7290-425e-be8d-b0bef20ea3c4-metrics-certs") pod "network-metrics-daemon-5m47w" (UID: "d7f13799-7290-425e-be8d-b0bef20ea3c4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:05:59.056752 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:59.056707 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqx4z\" (UniqueName: \"kubernetes.io/projected/e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e-kube-api-access-dqx4z\") pod \"network-check-target-mbz65\" (UID: \"e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e\") " pod="openshift-network-diagnostics/network-check-target-mbz65" Jul 06 20:05:59.056964 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:59.056921 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:05:59.056964 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:59.056949 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:05:59.056964 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:59.056965 2578 projected.go:196] Error preparing data for projected volume kube-api-access-dqx4z for pod openshift-network-diagnostics/network-check-target-mbz65: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:05:59.057144 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:59.057040 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e-kube-api-access-dqx4z podName:e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e nodeName:}" failed. No retries permitted until 2026-07-06 20:06:07.05701949 +0000 UTC m=+18.268167706 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dqx4z" (UniqueName: "kubernetes.io/projected/e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e-kube-api-access-dqx4z") pod "network-check-target-mbz65" (UID: "e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:05:59.294311 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:05:59.294227 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbz65" Jul 06 20:05:59.294508 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:05:59.294354 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbz65" podUID="e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e" Jul 06 20:06:00.293699 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:00.293658 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5m47w" Jul 06 20:06:00.294219 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:00.293706 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:06:00.294219 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:00.293898 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5m47w" podUID="d7f13799-7290-425e-be8d-b0bef20ea3c4" Jul 06 20:06:00.294219 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:00.293982 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmp82" podUID="4ef938dd-1047-4283-a4f5-d0d0abb4eda0" Jul 06 20:06:01.293680 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:01.293644 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbz65" Jul 06 20:06:01.293882 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:01.293780 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbz65" podUID="e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e" Jul 06 20:06:02.293042 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:02.293007 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:06:02.293234 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:02.293007 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5m47w" Jul 06 20:06:02.293234 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:02.293143 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmp82" podUID="4ef938dd-1047-4283-a4f5-d0d0abb4eda0" Jul 06 20:06:02.293234 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:02.293207 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5m47w" podUID="d7f13799-7290-425e-be8d-b0bef20ea3c4" Jul 06 20:06:03.293090 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:03.293045 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbz65" Jul 06 20:06:03.293553 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:03.293192 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbz65" podUID="e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e" Jul 06 20:06:04.292600 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:04.292557 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5m47w" Jul 06 20:06:04.292801 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:04.292610 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:06:04.292801 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:04.292717 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5m47w" podUID="d7f13799-7290-425e-be8d-b0bef20ea3c4" Jul 06 20:06:04.292883 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:04.292820 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmp82" podUID="4ef938dd-1047-4283-a4f5-d0d0abb4eda0" Jul 06 20:06:05.292730 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:05.292687 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbz65" Jul 06 20:06:05.293167 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:05.292820 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbz65" podUID="e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e" Jul 06 20:06:06.293176 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:06.293136 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:06:06.293176 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:06.293162 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5m47w" Jul 06 20:06:06.293697 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:06.293273 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmp82" podUID="4ef938dd-1047-4283-a4f5-d0d0abb4eda0" Jul 06 20:06:06.293697 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:06.293415 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5m47w" podUID="d7f13799-7290-425e-be8d-b0bef20ea3c4" Jul 06 20:06:07.019242 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:07.019205 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f13799-7290-425e-be8d-b0bef20ea3c4-metrics-certs\") pod \"network-metrics-daemon-5m47w\" (UID: \"d7f13799-7290-425e-be8d-b0bef20ea3c4\") " pod="openshift-multus/network-metrics-daemon-5m47w" Jul 06 20:06:07.019242 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:07.019246 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ef938dd-1047-4283-a4f5-d0d0abb4eda0-original-pull-secret\") pod \"global-pull-secret-syncer-jmp82\" (UID: \"4ef938dd-1047-4283-a4f5-d0d0abb4eda0\") " pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:06:07.019449 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:07.019377 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:06:07.019449 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:07.019384 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:06:07.019449 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:07.019428 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ef938dd-1047-4283-a4f5-d0d0abb4eda0-original-pull-secret podName:4ef938dd-1047-4283-a4f5-d0d0abb4eda0 nodeName:}" failed. No retries permitted until 2026-07-06 20:06:23.019415273 +0000 UTC m=+34.230563469 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4ef938dd-1047-4283-a4f5-d0d0abb4eda0-original-pull-secret") pod "global-pull-secret-syncer-jmp82" (UID: "4ef938dd-1047-4283-a4f5-d0d0abb4eda0") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:06:07.019582 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:07.019471 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7f13799-7290-425e-be8d-b0bef20ea3c4-metrics-certs podName:d7f13799-7290-425e-be8d-b0bef20ea3c4 nodeName:}" failed. No retries permitted until 2026-07-06 20:06:23.019434434 +0000 UTC m=+34.230582632 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7f13799-7290-425e-be8d-b0bef20ea3c4-metrics-certs") pod "network-metrics-daemon-5m47w" (UID: "d7f13799-7290-425e-be8d-b0bef20ea3c4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:06:07.120585 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:07.120547 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqx4z\" (UniqueName: \"kubernetes.io/projected/e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e-kube-api-access-dqx4z\") pod \"network-check-target-mbz65\" (UID: \"e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e\") " pod="openshift-network-diagnostics/network-check-target-mbz65" Jul 06 20:06:07.120769 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:07.120737 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:06:07.120769 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:07.120767 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:06:07.120957 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:07.120778 2578 projected.go:196] Error preparing data for projected volume kube-api-access-dqx4z for pod openshift-network-diagnostics/network-check-target-mbz65: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:06:07.120957 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:07.120837 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e-kube-api-access-dqx4z podName:e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e nodeName:}" failed. No retries permitted until 2026-07-06 20:06:23.120821656 +0000 UTC m=+34.331969857 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dqx4z" (UniqueName: "kubernetes.io/projected/e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e-kube-api-access-dqx4z") pod "network-check-target-mbz65" (UID: "e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:06:07.292878 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:07.292777 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbz65" Jul 06 20:06:07.293041 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:07.292899 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbz65" podUID="e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e" Jul 06 20:06:08.293483 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:08.293438 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5m47w" Jul 06 20:06:08.293483 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:08.293488 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:06:08.293981 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:08.293584 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5m47w" podUID="d7f13799-7290-425e-be8d-b0bef20ea3c4" Jul 06 20:06:08.293981 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:08.293722 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmp82" podUID="4ef938dd-1047-4283-a4f5-d0d0abb4eda0" Jul 06 20:06:09.296699 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:09.296664 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbz65" Jul 06 20:06:09.297137 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:09.296763 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbz65" podUID="e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e" Jul 06 20:06:10.293185 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:10.293148 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:06:10.293383 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:10.293260 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmp82" podUID="4ef938dd-1047-4283-a4f5-d0d0abb4eda0" Jul 06 20:06:10.293383 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:10.293289 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5m47w" Jul 06 20:06:10.293383 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:10.293362 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5m47w" podUID="d7f13799-7290-425e-be8d-b0bef20ea3c4" Jul 06 20:06:11.293025 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:11.292973 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbz65" Jul 06 20:06:11.293599 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:11.293107 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbz65" podUID="e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e" Jul 06 20:06:12.293426 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:12.293140 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5m47w" Jul 06 20:06:12.294092 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:12.293190 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:06:12.294092 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:12.293592 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5m47w" podUID="d7f13799-7290-425e-be8d-b0bef20ea3c4" Jul 06 20:06:12.294092 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:12.293697 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmp82" podUID="4ef938dd-1047-4283-a4f5-d0d0abb4eda0" Jul 06 20:06:12.410397 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:12.410281 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kpft4" event={"ID":"82df8708-d324-4b4d-8c74-1a65b6461394","Type":"ContainerStarted","Data":"ba2505c16a757f73293ad3554f9d54f034e82e834b9dd8854281118cea419c53"} Jul 06 20:06:12.411978 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:12.411948 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-42g4g" event={"ID":"404225de-b80e-4320-9e75-f93046c00433","Type":"ContainerStarted","Data":"dba6032421e425bd8122eaebf83838ed92ff0475e721a514bcd48fd6731fe8c0"} Jul 06 20:06:12.413524 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:12.413503 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r6xsf" event={"ID":"4e950de4-fc27-492d-ab51-69255ef36a84","Type":"ContainerStarted","Data":"cdda4f19f53f404b9496e1305a1ea395e8c7bbc08d6ea32aeaac682f5b1a7d6e"} Jul 06 20:06:12.416800 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:12.416779 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dnfgp_0f4c3a8a-30c4-428e-88a6-dcf086ef8615/ovn-acl-logging/0.log" Jul 06 20:06:12.417143 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:12.417121 2578 generic.go:356] "Generic (PLEG): container finished" podID="0f4c3a8a-30c4-428e-88a6-dcf086ef8615" containerID="f16dc81e56980e5e7d38bdf4a8f7ffa279ae1dff4018e29392f141b0c596aca0" exitCode=1 Jul 06 20:06:12.417238 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:12.417190 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" event={"ID":"0f4c3a8a-30c4-428e-88a6-dcf086ef8615","Type":"ContainerStarted","Data":"ea4a3dd394abaadb04bef7f055652775956bd96b270293e7e8740150e6ef0900"} Jul 06 20:06:12.417299 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:12.417285 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" event={"ID":"0f4c3a8a-30c4-428e-88a6-dcf086ef8615","Type":"ContainerStarted","Data":"dbd26f2b4d620f5ff856715226465f66781af27f5e82e2db0b20668aa317f87c"} Jul 06 20:06:12.417356 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:12.417339 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" event={"ID":"0f4c3a8a-30c4-428e-88a6-dcf086ef8615","Type":"ContainerStarted","Data":"d33e6bd7aad07685eb23811f3357b824e373468d47f5304698a8639bd3c6bfe6"} Jul 06 20:06:12.417404 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:12.417388 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" event={"ID":"0f4c3a8a-30c4-428e-88a6-dcf086ef8615","Type":"ContainerStarted","Data":"c1d608af88c3ab4395dbe86f30803b65af65cc46fede2e74afa06be5f0f3d41f"} Jul 06 20:06:12.417477 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:12.417439 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" event={"ID":"0f4c3a8a-30c4-428e-88a6-dcf086ef8615","Type":"ContainerDied","Data":"f16dc81e56980e5e7d38bdf4a8f7ffa279ae1dff4018e29392f141b0c596aca0"} Jul 06 20:06:12.417537 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:12.417517 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" event={"ID":"0f4c3a8a-30c4-428e-88a6-dcf086ef8615","Type":"ContainerStarted","Data":"51a16ed5973307214876e11902633cc3e8f3ddac46cfdbd52af3d348ea70856a"} Jul 06 20:06:12.419167 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:12.419143 2578 generic.go:356] "Generic (PLEG): container finished" podID="87026d58-c6e6-4aea-8926-5b1a8cb6d71a" containerID="62845d9752f6f4f615ac0461bd311155391b26d329f101c2fe32f69b63ac6e09" exitCode=0 Jul 06 20:06:12.419415 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:12.419313 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kgswg" event={"ID":"87026d58-c6e6-4aea-8926-5b1a8cb6d71a","Type":"ContainerDied","Data":"62845d9752f6f4f615ac0461bd311155391b26d329f101c2fe32f69b63ac6e09"} Jul 06 20:06:12.420918 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:12.420897 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5wjbn" event={"ID":"24dd4f5b-0cda-4bb0-b776-72712fb30f3c","Type":"ContainerStarted","Data":"4b13e9698209e1b8263fdd07765e46a9c8161a202ad7df6e6fe1774b268b64bc"} Jul 06 20:06:12.423075 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:12.423021 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j4d6m" event={"ID":"f0b6b2dc-73c6-453c-9941-8738e48b4976","Type":"ContainerStarted","Data":"2eb083e38456bcef004ede617dbf3633d84e38052f0e7e4c2d8cd3124a7796a4"} Jul 06 20:06:12.425199 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:12.425175 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7dxp" event={"ID":"2a3ceee0-1606-499a-800e-f8ce32884427","Type":"ContainerStarted","Data":"40079be3252b773022867b5af971227ae2f008f3cdecca66abb554c53f38b015"} Jul 06 20:06:12.435862 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:12.435807 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-218.ec2.internal" podStartSLOduration=21.435791306 podStartE2EDuration="21.435791306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:05:54.413576166 +0000 UTC m=+5.624724384" watchObservedRunningTime="2026-07-06 20:06:12.435791306 +0000 UTC m=+23.646939526" Jul 06 20:06:12.461735 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:12.461689 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-kpft4" podStartSLOduration=3.170245232 podStartE2EDuration="22.461674197s" totalImagesPullingTime="19.291428965s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:05:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:06:12.435729875 +0000 UTC m=+23.646878094" watchObservedRunningTime="2026-07-06 20:06:12.461674197 +0000 UTC m=+23.672822414" Jul 06 20:06:12.498327 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:12.498260 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-r6xsf" podStartSLOduration=3.22613258 podStartE2EDuration="22.498241902s" totalImagesPullingTime="19.272109322s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:05:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:06:12.461980361 +0000 UTC m=+23.673128593" watchObservedRunningTime="2026-07-06 20:06:12.498241902 +0000 UTC m=+23.709390162" Jul 06 20:06:12.554761 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:12.554705 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-42g4g" podStartSLOduration=3.290554504 podStartE2EDuration="22.5546891s" totalImagesPullingTime="19.264134596s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:05:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:06:12.554617691 +0000 UTC m=+23.765765910" watchObservedRunningTime="2026-07-06 20:06:12.5546891 +0000 UTC m=+23.765837319" Jul 06 20:06:12.554921 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:12.554810 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-j4d6m" podStartSLOduration=3.287330806 podStartE2EDuration="22.554801239s" totalImagesPullingTime="19.267470433s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:05:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:06:12.528875901 +0000 UTC m=+23.740024127" watchObservedRunningTime="2026-07-06 20:06:12.554801239 +0000 UTC m=+23.765949456" Jul 06 20:06:13.040489 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:13.040437 2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 20:06:13.228924 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:13.228778 2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T20:06:13.040476258Z","UUID":"839bc66c-b1eb-40b0-a674-16b33d0bcaa7","Handler":null,"Name":"","Endpoint":""} Jul 06 20:06:13.230574 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:13.230547 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 20:06:13.230699 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:13.230595 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 20:06:13.293154 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:13.293127 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbz65" Jul 06 20:06:13.293299 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:13.293259 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbz65" podUID="e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e" Jul 06 20:06:13.430150 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:13.430106 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7dxp" event={"ID":"2a3ceee0-1606-499a-800e-f8ce32884427","Type":"ContainerStarted","Data":"85015071069d4c8e3527eb16f01f6a1ea4abe578ff397669fc8455208eb0a4c5"} Jul 06 20:06:13.431793 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:13.431766 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h7j8j" event={"ID":"0d2af31a-954a-4a78-861b-ddaa83e8df1a","Type":"ContainerStarted","Data":"c4f218f7fce0cdc95b80debb72970bd6713f33bfea32b1eb9c01ae777f1f9677"} Jul 06 20:06:13.458509 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:13.458448 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5wjbn" podStartSLOduration=4.196159572 podStartE2EDuration="23.458433134s" totalImagesPullingTime="19.262273562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:05:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:06:12.586658363 +0000 UTC m=+23.797806581" watchObservedRunningTime="2026-07-06 20:06:13.458433134 +0000 UTC m=+24.669581351" Jul 06 20:06:14.293179 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:14.293099 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5m47w" Jul 06 20:06:14.293179 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:14.293138 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:06:14.293393 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:14.293223 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5m47w" podUID="d7f13799-7290-425e-be8d-b0bef20ea3c4" Jul 06 20:06:14.293393 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:14.293329 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmp82" podUID="4ef938dd-1047-4283-a4f5-d0d0abb4eda0" Jul 06 20:06:14.437406 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:14.437385 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dnfgp_0f4c3a8a-30c4-428e-88a6-dcf086ef8615/ovn-acl-logging/0.log" Jul 06 20:06:14.437826 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:14.437768 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" event={"ID":"0f4c3a8a-30c4-428e-88a6-dcf086ef8615","Type":"ContainerStarted","Data":"6d6bc211d4dfb0505aaef52dc0f1163e89fb52b27451b4c9508709e616bd2a10"} Jul 06 20:06:14.439861 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:14.439837 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7dxp" event={"ID":"2a3ceee0-1606-499a-800e-f8ce32884427","Type":"ContainerStarted","Data":"5075c6d99fb5513472baa2c1e885c4eaab5eedbabba0a2ffb4e596bcf9136f43"} Jul 06 20:06:14.490163 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:14.490106 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-h7j8j" podStartSLOduration=5.232695528 podStartE2EDuration="24.490085941s" totalImagesPullingTime="19.257390413s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:05:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:06:13.458372357 +0000 UTC m=+24.669520575" watchObservedRunningTime="2026-07-06 20:06:14.490085941 +0000 UTC m=+25.701234156" Jul 06 20:06:14.490434 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:14.490395 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k7dxp" podStartSLOduration=3.0926791160000002 podStartE2EDuration="24.490388228s" totalImagesPullingTime="21.397709112s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:05:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:06:14.489481575 +0000 UTC m=+25.700629793" watchObservedRunningTime="2026-07-06 20:06:14.490388228 +0000 UTC m=+25.701536448" Jul 06 20:06:15.293003 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:15.292785 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbz65" Jul 06 20:06:15.293221 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:15.293101 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbz65" podUID="e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e" Jul 06 20:06:15.451826 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:15.451791 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5wjbn" Jul 06 20:06:15.452399 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:15.452381 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5wjbn" Jul 06 20:06:16.293550 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:16.293516 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:06:16.293550 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:16.293554 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5m47w" Jul 06 20:06:16.293798 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:16.293647 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmp82" podUID="4ef938dd-1047-4283-a4f5-d0d0abb4eda0" Jul 06 20:06:16.293798 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:16.293745 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5m47w" podUID="d7f13799-7290-425e-be8d-b0bef20ea3c4" Jul 06 20:06:17.293617 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:17.293534 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbz65" Jul 06 20:06:17.294252 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:17.293669 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbz65" podUID="e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e" Jul 06 20:06:17.957717 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:17.957479 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5wjbn" Jul 06 20:06:17.957877 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:17.957746 2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 06 20:06:17.958181 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:17.958165 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5wjbn" Jul 06 20:06:18.293294 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:18.293204 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5m47w" Jul 06 20:06:18.293294 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:18.293222 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:06:18.293541 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:18.293312 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5m47w" podUID="d7f13799-7290-425e-be8d-b0bef20ea3c4" Jul 06 20:06:18.293541 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:18.293469 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmp82" podUID="4ef938dd-1047-4283-a4f5-d0d0abb4eda0" Jul 06 20:06:18.452013 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:18.451982 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dnfgp_0f4c3a8a-30c4-428e-88a6-dcf086ef8615/ovn-acl-logging/0.log" Jul 06 20:06:18.452627 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:18.452331 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" event={"ID":"0f4c3a8a-30c4-428e-88a6-dcf086ef8615","Type":"ContainerStarted","Data":"6d208eb9582137277f29343eba129217cb7c520046386ac3fcfc4d7a2e601348"} Jul 06 20:06:18.452679 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:18.452652 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:06:18.452768 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:18.452737 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:06:18.452888 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:18.452804 2578 scope.go:117] "RemoveContainer" containerID="f16dc81e56980e5e7d38bdf4a8f7ffa279ae1dff4018e29392f141b0c596aca0" Jul 06 20:06:18.454394 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:18.454371 2578 generic.go:356] "Generic (PLEG): container finished" podID="87026d58-c6e6-4aea-8926-5b1a8cb6d71a" containerID="77d153c3fdd62ccddb424f911d6347146f72c76c740b771dd5e925d6373e441e" exitCode=0 Jul 06 20:06:18.454505 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:18.454478 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kgswg" event={"ID":"87026d58-c6e6-4aea-8926-5b1a8cb6d71a","Type":"ContainerDied","Data":"77d153c3fdd62ccddb424f911d6347146f72c76c740b771dd5e925d6373e441e"} Jul 06 20:06:18.469043 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:18.469018 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:06:19.294318 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:19.294221 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbz65" Jul 06 20:06:19.294372 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:19.294315 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbz65" podUID="e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e" Jul 06 20:06:19.460725 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:19.460651 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dnfgp_0f4c3a8a-30c4-428e-88a6-dcf086ef8615/ovn-acl-logging/0.log" Jul 06 20:06:19.461104 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:19.460998 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" event={"ID":"0f4c3a8a-30c4-428e-88a6-dcf086ef8615","Type":"ContainerStarted","Data":"c8bbeac4c3820da94e236ee37b070205e559a4369e4551e8c0e436dd034a4571"} Jul 06 20:06:19.461333 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:19.461310 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:06:19.463153 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:19.463129 2578 generic.go:356] "Generic (PLEG): container finished" podID="87026d58-c6e6-4aea-8926-5b1a8cb6d71a" containerID="a5960a3dbcc00599753ba0f59fbe27cbe3fb45caf5de06ce504f3da1225d39fa" exitCode=0 Jul 06 20:06:19.463230 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:19.463175 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kgswg" event={"ID":"87026d58-c6e6-4aea-8926-5b1a8cb6d71a","Type":"ContainerDied","Data":"a5960a3dbcc00599753ba0f59fbe27cbe3fb45caf5de06ce504f3da1225d39fa"} Jul 06 20:06:19.476896 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:19.476876 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:06:19.512290 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:19.512248 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" podStartSLOduration=10.186747419 podStartE2EDuration="29.512234559s" totalImagesPullingTime="19.32548714s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:05:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:06:19.510608062 +0000 UTC m=+30.721756280" watchObservedRunningTime="2026-07-06 20:06:19.512234559 +0000 UTC m=+30.723382777" Jul 06 20:06:20.208878 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:20.208712 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jmp82"] Jul 06 20:06:20.209024 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:20.209012 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:06:20.209136 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:20.209117 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmp82" podUID="4ef938dd-1047-4283-a4f5-d0d0abb4eda0" Jul 06 20:06:20.219170 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:20.219141 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5m47w"] Jul 06 20:06:20.219269 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:20.219252 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5m47w" Jul 06 20:06:20.219360 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:20.219342 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5m47w" podUID="d7f13799-7290-425e-be8d-b0bef20ea3c4" Jul 06 20:06:20.219742 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:20.219693 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mbz65"] Jul 06 20:06:20.219809 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:20.219790 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbz65" Jul 06 20:06:20.219883 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:20.219866 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbz65" podUID="e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e" Jul 06 20:06:20.468157 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:20.468123 2578 generic.go:356] "Generic (PLEG): container finished" podID="87026d58-c6e6-4aea-8926-5b1a8cb6d71a" containerID="b330167f18215040c7cd56d09acd3652ab7aa19b8307e51fc25d52bc174367cb" exitCode=0 Jul 06 20:06:20.468823 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:20.468191 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kgswg" event={"ID":"87026d58-c6e6-4aea-8926-5b1a8cb6d71a","Type":"ContainerDied","Data":"b330167f18215040c7cd56d09acd3652ab7aa19b8307e51fc25d52bc174367cb"} Jul 06 20:06:21.293565 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:21.293525 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:06:21.293729 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:21.293646 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmp82" podUID="4ef938dd-1047-4283-a4f5-d0d0abb4eda0" Jul 06 20:06:22.292637 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:22.292604 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5m47w" Jul 06 20:06:22.292637 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:22.292602 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbz65" Jul 06 20:06:22.293373 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:22.292757 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5m47w" podUID="d7f13799-7290-425e-be8d-b0bef20ea3c4" Jul 06 20:06:22.293373 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:22.292824 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbz65" podUID="e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e" Jul 06 20:06:23.051141 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:23.051103 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f13799-7290-425e-be8d-b0bef20ea3c4-metrics-certs\") pod \"network-metrics-daemon-5m47w\" (UID: \"d7f13799-7290-425e-be8d-b0bef20ea3c4\") " pod="openshift-multus/network-metrics-daemon-5m47w" Jul 06 20:06:23.051141 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:23.051143 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ef938dd-1047-4283-a4f5-d0d0abb4eda0-original-pull-secret\") pod \"global-pull-secret-syncer-jmp82\" (UID: \"4ef938dd-1047-4283-a4f5-d0d0abb4eda0\") " pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:06:23.051337 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:23.051267 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:06:23.051337 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:23.051268 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:06:23.051337 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:23.051334 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7f13799-7290-425e-be8d-b0bef20ea3c4-metrics-certs podName:d7f13799-7290-425e-be8d-b0bef20ea3c4 nodeName:}" failed. No retries permitted until 2026-07-06 20:06:55.051313618 +0000 UTC m=+66.262461814 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7f13799-7290-425e-be8d-b0bef20ea3c4-metrics-certs") pod "network-metrics-daemon-5m47w" (UID: "d7f13799-7290-425e-be8d-b0bef20ea3c4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:06:23.051495 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:23.051353 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ef938dd-1047-4283-a4f5-d0d0abb4eda0-original-pull-secret podName:4ef938dd-1047-4283-a4f5-d0d0abb4eda0 nodeName:}" failed. No retries permitted until 2026-07-06 20:06:55.051344823 +0000 UTC m=+66.262493020 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4ef938dd-1047-4283-a4f5-d0d0abb4eda0-original-pull-secret") pod "global-pull-secret-syncer-jmp82" (UID: "4ef938dd-1047-4283-a4f5-d0d0abb4eda0") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:06:23.152358 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:23.152324 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqx4z\" (UniqueName: \"kubernetes.io/projected/e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e-kube-api-access-dqx4z\") pod \"network-check-target-mbz65\" (UID: \"e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e\") " pod="openshift-network-diagnostics/network-check-target-mbz65" Jul 06 20:06:23.152565 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:23.152509 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:06:23.152565 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:23.152527 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:06:23.152565 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:23.152535 2578 projected.go:196] Error preparing data for projected volume kube-api-access-dqx4z for pod openshift-network-diagnostics/network-check-target-mbz65: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:06:23.152689 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:23.152596 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e-kube-api-access-dqx4z podName:e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e nodeName:}" failed. No retries permitted until 2026-07-06 20:06:55.152577533 +0000 UTC m=+66.363725764 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dqx4z" (UniqueName: "kubernetes.io/projected/e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e-kube-api-access-dqx4z") pod "network-check-target-mbz65" (UID: "e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:06:23.293511 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:23.293473 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:06:23.294023 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:23.293612 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jmp82" podUID="4ef938dd-1047-4283-a4f5-d0d0abb4eda0" Jul 06 20:06:24.293099 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.293069 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbz65" Jul 06 20:06:24.293222 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.293072 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5m47w" Jul 06 20:06:24.293222 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:24.293205 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbz65" podUID="e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e" Jul 06 20:06:24.293331 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:24.293290 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5m47w" podUID="d7f13799-7290-425e-be8d-b0bef20ea3c4" Jul 06 20:06:24.396684 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.396647 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:24.498618 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.498576 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:24.543072 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.542999 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-218.ec2.internal" event="NodeReady" Jul 06 20:06:24.543213 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.543116 2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 20:06:24.600021 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.599984 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d4b6bdbbc-fr554"] Jul 06 20:06:24.604168 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.604138 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:24.618336 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.618314 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pwlxq"] Jul 06 20:06:24.618471 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.618442 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" Jul 06 20:06:24.622410 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.622392 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 20:06:24.623134 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.623117 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gmdh4\"" Jul 06 20:06:24.623487 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.623472 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 20:06:24.623577 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.623479 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 20:06:24.629730 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.629708 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 20:06:24.630158 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.630140 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9vmsh"] Jul 06 20:06:24.630287 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.630269 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pwlxq" Jul 06 20:06:24.632759 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.632743 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 20:06:24.633054 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.633042 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5vcsc\"" Jul 06 20:06:24.633100 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.633083 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 20:06:24.644959 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.644938 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9vmsh" Jul 06 20:06:24.645060 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.644942 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pwlxq"] Jul 06 20:06:24.645060 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.645006 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9vmsh"] Jul 06 20:06:24.645060 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.645022 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d4b6bdbbc-fr554"] Jul 06 20:06:24.647773 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.647752 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 20:06:24.647880 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.647752 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 20:06:24.648085 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.648070 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 20:06:24.648673 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.648648 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kwpnh\"" Jul 06 20:06:24.762848 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.762813 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blvtm\" (UniqueName: \"kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-kube-api-access-blvtm\") pod \"image-registry-5d4b6bdbbc-fr554\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" Jul 06 20:06:24.762848 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.762851 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/71661bcb-74d1-48e3-afa1-4acec1af79ae-config-volume\") pod \"dns-default-pwlxq\" (UID: \"71661bcb-74d1-48e3-afa1-4acec1af79ae\") " pod="openshift-dns/dns-default-pwlxq" Jul 06 20:06:24.763039 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.762945 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-bound-sa-token\") pod \"image-registry-5d4b6bdbbc-fr554\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" Jul 06 20:06:24.763039 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.762980 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba49337f-cb8d-4c73-9a76-7945951eac79-cert\") pod \"ingress-canary-9vmsh\" (UID: \"ba49337f-cb8d-4c73-9a76-7945951eac79\") " pod="openshift-ingress-canary/ingress-canary-9vmsh" Jul 06 20:06:24.763039 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.763026 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71661bcb-74d1-48e3-afa1-4acec1af79ae-tmp-dir\") pod \"dns-default-pwlxq\" (UID: \"71661bcb-74d1-48e3-afa1-4acec1af79ae\") " pod="openshift-dns/dns-default-pwlxq" Jul 06 20:06:24.763131 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.763042 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-registry-tls\") pod \"image-registry-5d4b6bdbbc-fr554\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" Jul 06 20:06:24.763131 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.763063 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71661bcb-74d1-48e3-afa1-4acec1af79ae-metrics-tls\") pod \"dns-default-pwlxq\" (UID: \"71661bcb-74d1-48e3-afa1-4acec1af79ae\") " pod="openshift-dns/dns-default-pwlxq" Jul 06 20:06:24.763131 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.763098 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/11be38ab-74ab-495f-b752-786d02385ee2-ca-trust-extracted\") pod \"image-registry-5d4b6bdbbc-fr554\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" Jul 06 20:06:24.763131 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.763119 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/11be38ab-74ab-495f-b752-786d02385ee2-installation-pull-secrets\") pod \"image-registry-5d4b6bdbbc-fr554\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" Jul 06 20:06:24.763246 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.763144 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11be38ab-74ab-495f-b752-786d02385ee2-trusted-ca\") pod \"image-registry-5d4b6bdbbc-fr554\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" Jul 06 20:06:24.763246 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.763179 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjzhl\" (UniqueName: \"kubernetes.io/projected/71661bcb-74d1-48e3-afa1-4acec1af79ae-kube-api-access-vjzhl\") pod \"dns-default-pwlxq\" (UID: \"71661bcb-74d1-48e3-afa1-4acec1af79ae\") " pod="openshift-dns/dns-default-pwlxq" Jul 06 20:06:24.763246 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.763220 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/11be38ab-74ab-495f-b752-786d02385ee2-registry-certificates\") pod \"image-registry-5d4b6bdbbc-fr554\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" Jul 06 20:06:24.763339 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.763258 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/11be38ab-74ab-495f-b752-786d02385ee2-image-registry-private-configuration\") pod \"image-registry-5d4b6bdbbc-fr554\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" Jul 06 20:06:24.763339 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.763280 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6c7d\" (UniqueName: \"kubernetes.io/projected/ba49337f-cb8d-4c73-9a76-7945951eac79-kube-api-access-m6c7d\") pod \"ingress-canary-9vmsh\" (UID: \"ba49337f-cb8d-4c73-9a76-7945951eac79\") " pod="openshift-ingress-canary/ingress-canary-9vmsh" Jul 06 20:06:24.864435 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.864408 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-bound-sa-token\") pod \"image-registry-5d4b6bdbbc-fr554\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" Jul 06 20:06:24.864582 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.864441 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba49337f-cb8d-4c73-9a76-7945951eac79-cert\") pod \"ingress-canary-9vmsh\" (UID: \"ba49337f-cb8d-4c73-9a76-7945951eac79\") " pod="openshift-ingress-canary/ingress-canary-9vmsh" Jul 06 20:06:24.864582 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.864497 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71661bcb-74d1-48e3-afa1-4acec1af79ae-tmp-dir\") pod \"dns-default-pwlxq\" (UID: \"71661bcb-74d1-48e3-afa1-4acec1af79ae\") " pod="openshift-dns/dns-default-pwlxq" Jul 06 20:06:24.864582 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.864520 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-registry-tls\") pod \"image-registry-5d4b6bdbbc-fr554\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" Jul 06 20:06:24.864582 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.864551 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71661bcb-74d1-48e3-afa1-4acec1af79ae-metrics-tls\") pod \"dns-default-pwlxq\" (UID: \"71661bcb-74d1-48e3-afa1-4acec1af79ae\") " pod="openshift-dns/dns-default-pwlxq" Jul 06 20:06:24.864582 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.864573 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/11be38ab-74ab-495f-b752-786d02385ee2-ca-trust-extracted\") pod \"image-registry-5d4b6bdbbc-fr554\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" Jul 06 20:06:24.864877 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.864595 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/11be38ab-74ab-495f-b752-786d02385ee2-installation-pull-secrets\") pod \"image-registry-5d4b6bdbbc-fr554\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" Jul 06 20:06:24.864877 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.864621 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11be38ab-74ab-495f-b752-786d02385ee2-trusted-ca\") pod \"image-registry-5d4b6bdbbc-fr554\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" Jul 06 20:06:24.864877 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.864654 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjzhl\" (UniqueName: \"kubernetes.io/projected/71661bcb-74d1-48e3-afa1-4acec1af79ae-kube-api-access-vjzhl\") pod \"dns-default-pwlxq\" (UID: \"71661bcb-74d1-48e3-afa1-4acec1af79ae\") " pod="openshift-dns/dns-default-pwlxq" Jul 06 20:06:24.864877 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:24.864661 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 20:06:24.864877 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:24.864681 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d4b6bdbbc-fr554: secret "image-registry-tls" not found Jul 06 20:06:24.864877 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:24.864745 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-registry-tls podName:11be38ab-74ab-495f-b752-786d02385ee2 nodeName:}" failed. No retries permitted until 2026-07-06 20:06:25.364723542 +0000 UTC m=+36.575871755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-registry-tls") pod "image-registry-5d4b6bdbbc-fr554" (UID: "11be38ab-74ab-495f-b752-786d02385ee2") : secret "image-registry-tls" not found Jul 06 20:06:24.865178 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.864902 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71661bcb-74d1-48e3-afa1-4acec1af79ae-tmp-dir\") pod \"dns-default-pwlxq\" (UID: \"71661bcb-74d1-48e3-afa1-4acec1af79ae\") " pod="openshift-dns/dns-default-pwlxq" Jul 06 20:06:24.865178 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.864681 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/11be38ab-74ab-495f-b752-786d02385ee2-registry-certificates\") pod \"image-registry-5d4b6bdbbc-fr554\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" Jul 06 20:06:24.865178 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.864979 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/11be38ab-74ab-495f-b752-786d02385ee2-image-registry-private-configuration\") pod \"image-registry-5d4b6bdbbc-fr554\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" Jul 06 20:06:24.865178 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.865013 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6c7d\" (UniqueName: \"kubernetes.io/projected/ba49337f-cb8d-4c73-9a76-7945951eac79-kube-api-access-m6c7d\") pod \"ingress-canary-9vmsh\" (UID: \"ba49337f-cb8d-4c73-9a76-7945951eac79\") " pod="openshift-ingress-canary/ingress-canary-9vmsh" Jul 06 20:06:24.865178 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.865023 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/11be38ab-74ab-495f-b752-786d02385ee2-ca-trust-extracted\") pod \"image-registry-5d4b6bdbbc-fr554\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" Jul 06 20:06:24.865178 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.865044 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blvtm\" (UniqueName: \"kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-kube-api-access-blvtm\") pod \"image-registry-5d4b6bdbbc-fr554\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" Jul 06 20:06:24.865178 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.865077 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/71661bcb-74d1-48e3-afa1-4acec1af79ae-config-volume\") pod \"dns-default-pwlxq\" (UID: \"71661bcb-74d1-48e3-afa1-4acec1af79ae\") " pod="openshift-dns/dns-default-pwlxq" Jul 06 20:06:24.865535 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.865381 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/11be38ab-74ab-495f-b752-786d02385ee2-registry-certificates\") pod \"image-registry-5d4b6bdbbc-fr554\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" Jul 06 20:06:24.865535 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:24.865397 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 20:06:24.865535 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:24.865487 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71661bcb-74d1-48e3-afa1-4acec1af79ae-metrics-tls podName:71661bcb-74d1-48e3-afa1-4acec1af79ae nodeName:}" failed. No retries permitted until 2026-07-06 20:06:25.365437282 +0000 UTC m=+36.576585497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71661bcb-74d1-48e3-afa1-4acec1af79ae-metrics-tls") pod "dns-default-pwlxq" (UID: "71661bcb-74d1-48e3-afa1-4acec1af79ae") : secret "dns-default-metrics-tls" not found Jul 06 20:06:24.865667 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.865647 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/71661bcb-74d1-48e3-afa1-4acec1af79ae-config-volume\") pod \"dns-default-pwlxq\" (UID: \"71661bcb-74d1-48e3-afa1-4acec1af79ae\") " pod="openshift-dns/dns-default-pwlxq" Jul 06 20:06:24.866008 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.865760 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11be38ab-74ab-495f-b752-786d02385ee2-trusted-ca\") pod \"image-registry-5d4b6bdbbc-fr554\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" Jul 06 20:06:24.868822 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.868791 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/11be38ab-74ab-495f-b752-786d02385ee2-image-registry-private-configuration\") pod \"image-registry-5d4b6bdbbc-fr554\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" Jul 06 20:06:24.869049 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.869019 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/11be38ab-74ab-495f-b752-786d02385ee2-installation-pull-secrets\") pod \"image-registry-5d4b6bdbbc-fr554\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" Jul 06 20:06:24.869089 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.869060 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba49337f-cb8d-4c73-9a76-7945951eac79-cert\") pod \"ingress-canary-9vmsh\" (UID: \"ba49337f-cb8d-4c73-9a76-7945951eac79\") " pod="openshift-ingress-canary/ingress-canary-9vmsh" Jul 06 20:06:24.883242 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.883212 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-bound-sa-token\") pod \"image-registry-5d4b6bdbbc-fr554\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" Jul 06 20:06:24.883364 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.883289 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjzhl\" (UniqueName: \"kubernetes.io/projected/71661bcb-74d1-48e3-afa1-4acec1af79ae-kube-api-access-vjzhl\") pod \"dns-default-pwlxq\" (UID: \"71661bcb-74d1-48e3-afa1-4acec1af79ae\") " pod="openshift-dns/dns-default-pwlxq" Jul 06 20:06:24.885159 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.885128 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blvtm\" (UniqueName: \"kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-kube-api-access-blvtm\") pod \"image-registry-5d4b6bdbbc-fr554\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" Jul 06 20:06:24.886891 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.886872 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6c7d\" (UniqueName: \"kubernetes.io/projected/ba49337f-cb8d-4c73-9a76-7945951eac79-kube-api-access-m6c7d\") pod \"ingress-canary-9vmsh\" (UID: \"ba49337f-cb8d-4c73-9a76-7945951eac79\") " pod="openshift-ingress-canary/ingress-canary-9vmsh" Jul 06 20:06:24.953080 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:24.953036 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9vmsh" Jul 06 20:06:25.151246 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:25.151189 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9vmsh"] Jul 06 20:06:25.154216 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:06:25.154184 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba49337f_cb8d_4c73_9a76_7945951eac79.slice/crio-b53e43e9b9ccfefd5b692960db53818aee4f9e010652de57467c6b0d054e868f WatchSource:0}: Error finding container b53e43e9b9ccfefd5b692960db53818aee4f9e010652de57467c6b0d054e868f: Status 404 returned error can't find the container with id b53e43e9b9ccfefd5b692960db53818aee4f9e010652de57467c6b0d054e868f Jul 06 20:06:25.293042 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:25.293010 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:06:25.296179 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:25.296158 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 20:06:25.369750 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:25.369676 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-registry-tls\") pod \"image-registry-5d4b6bdbbc-fr554\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" Jul 06 20:06:25.369750 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:25.369711 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71661bcb-74d1-48e3-afa1-4acec1af79ae-metrics-tls\") pod \"dns-default-pwlxq\" (UID: \"71661bcb-74d1-48e3-afa1-4acec1af79ae\") " pod="openshift-dns/dns-default-pwlxq" Jul 06 20:06:25.369905 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:25.369830 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 20:06:25.369905 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:25.369841 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 20:06:25.369905 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:25.369857 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d4b6bdbbc-fr554: secret "image-registry-tls" not found Jul 06 20:06:25.369905 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:25.369877 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71661bcb-74d1-48e3-afa1-4acec1af79ae-metrics-tls podName:71661bcb-74d1-48e3-afa1-4acec1af79ae nodeName:}" failed. No retries permitted until 2026-07-06 20:06:26.369865188 +0000 UTC m=+37.581013387 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71661bcb-74d1-48e3-afa1-4acec1af79ae-metrics-tls") pod "dns-default-pwlxq" (UID: "71661bcb-74d1-48e3-afa1-4acec1af79ae") : secret "dns-default-metrics-tls" not found Jul 06 20:06:25.369905 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:25.369901 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-registry-tls podName:11be38ab-74ab-495f-b752-786d02385ee2 nodeName:}" failed. No retries permitted until 2026-07-06 20:06:26.369889563 +0000 UTC m=+37.581037759 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-registry-tls") pod "image-registry-5d4b6bdbbc-fr554" (UID: "11be38ab-74ab-495f-b752-786d02385ee2") : secret "image-registry-tls" not found Jul 06 20:06:25.480585 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:25.480549 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9vmsh" event={"ID":"ba49337f-cb8d-4c73-9a76-7945951eac79","Type":"ContainerStarted","Data":"b53e43e9b9ccfefd5b692960db53818aee4f9e010652de57467c6b0d054e868f"} Jul 06 20:06:25.483308 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:25.483286 2578 generic.go:356] "Generic (PLEG): container finished" podID="87026d58-c6e6-4aea-8926-5b1a8cb6d71a" containerID="331f9444901991ea79568dd5edfc2925c2abe0664b7a9ca84693fb2ae7915b17" exitCode=0 Jul 06 20:06:25.483414 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:25.483319 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kgswg" event={"ID":"87026d58-c6e6-4aea-8926-5b1a8cb6d71a","Type":"ContainerDied","Data":"331f9444901991ea79568dd5edfc2925c2abe0664b7a9ca84693fb2ae7915b17"} Jul 06 20:06:26.293592 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:26.293559 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5m47w" Jul 06 20:06:26.293780 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:26.293759 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbz65" Jul 06 20:06:26.302069 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:26.302046 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vntfk\"" Jul 06 20:06:26.302227 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:26.302212 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 20:06:26.302648 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:26.302526 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mt2s9\"" Jul 06 20:06:26.302648 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:26.302561 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 20:06:26.302648 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:26.302572 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 20:06:26.378778 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:26.378736 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-registry-tls\") pod \"image-registry-5d4b6bdbbc-fr554\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" Jul 06 20:06:26.378778 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:26.378772 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71661bcb-74d1-48e3-afa1-4acec1af79ae-metrics-tls\") pod \"dns-default-pwlxq\" (UID: \"71661bcb-74d1-48e3-afa1-4acec1af79ae\") " pod="openshift-dns/dns-default-pwlxq" Jul 06 20:06:26.379011 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:26.378887 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 20:06:26.379011 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:26.378891 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 20:06:26.379011 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:26.378908 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d4b6bdbbc-fr554: secret "image-registry-tls" not found Jul 06 20:06:26.379011 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:26.378957 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71661bcb-74d1-48e3-afa1-4acec1af79ae-metrics-tls podName:71661bcb-74d1-48e3-afa1-4acec1af79ae nodeName:}" failed. No retries permitted until 2026-07-06 20:06:28.378939409 +0000 UTC m=+39.590087605 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71661bcb-74d1-48e3-afa1-4acec1af79ae-metrics-tls") pod "dns-default-pwlxq" (UID: "71661bcb-74d1-48e3-afa1-4acec1af79ae") : secret "dns-default-metrics-tls" not found Jul 06 20:06:26.379011 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:26.378970 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-registry-tls podName:11be38ab-74ab-495f-b752-786d02385ee2 nodeName:}" failed. No retries permitted until 2026-07-06 20:06:28.378964148 +0000 UTC m=+39.590112344 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-registry-tls") pod "image-registry-5d4b6bdbbc-fr554" (UID: "11be38ab-74ab-495f-b752-786d02385ee2") : secret "image-registry-tls" not found Jul 06 20:06:26.488912 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:26.488883 2578 generic.go:356] "Generic (PLEG): container finished" podID="87026d58-c6e6-4aea-8926-5b1a8cb6d71a" containerID="40e570ee67089eba2b57426ff88377a604589bec24f7f9b072f7a182e1769dff" exitCode=0 Jul 06 20:06:26.489307 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:26.488935 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kgswg" event={"ID":"87026d58-c6e6-4aea-8926-5b1a8cb6d71a","Type":"ContainerDied","Data":"40e570ee67089eba2b57426ff88377a604589bec24f7f9b072f7a182e1769dff"} Jul 06 20:06:26.710280 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:26.710117 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:26.790123 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:26.790082 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:27.015210 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:27.015174 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:27.110386 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:27.110352 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:27.205127 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:27.205087 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:27.433000 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:27.432956 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:27.494989 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:27.494958 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kgswg" event={"ID":"87026d58-c6e6-4aea-8926-5b1a8cb6d71a","Type":"ContainerStarted","Data":"f9e8204a3c5f5b40236ccb43488e89efc4a1f716b46063210f3856d3c33de426"} Jul 06 20:06:27.536745 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:27.536688 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kgswg" podStartSLOduration=7.531841047 podStartE2EDuration="37.536668163s" totalImagesPullingTime="30.004827116s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:05:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:06:27.535328582 +0000 UTC m=+38.746476814" watchObservedRunningTime="2026-07-06 20:06:27.536668163 +0000 UTC m=+38.747816390" Jul 06 20:06:27.551885 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:27.551840 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:27.949852 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:27.949808 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:28.025775 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:28.025742 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:28.343771 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:28.343723 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:28.406785 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:28.406749 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-registry-tls\") pod \"image-registry-5d4b6bdbbc-fr554\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" Jul 06 20:06:28.406785 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:28.406785 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71661bcb-74d1-48e3-afa1-4acec1af79ae-metrics-tls\") pod \"dns-default-pwlxq\" (UID: \"71661bcb-74d1-48e3-afa1-4acec1af79ae\") " pod="openshift-dns/dns-default-pwlxq" Jul 06 20:06:28.407022 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:28.406897 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 20:06:28.407022 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:28.406913 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 20:06:28.407022 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:28.406931 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d4b6bdbbc-fr554: secret "image-registry-tls" not found Jul 06 20:06:28.407022 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:28.406956 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71661bcb-74d1-48e3-afa1-4acec1af79ae-metrics-tls podName:71661bcb-74d1-48e3-afa1-4acec1af79ae nodeName:}" failed. No retries permitted until 2026-07-06 20:06:32.406942402 +0000 UTC m=+43.618090598 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71661bcb-74d1-48e3-afa1-4acec1af79ae-metrics-tls") pod "dns-default-pwlxq" (UID: "71661bcb-74d1-48e3-afa1-4acec1af79ae") : secret "dns-default-metrics-tls" not found Jul 06 20:06:28.407022 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:28.406986 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-registry-tls podName:11be38ab-74ab-495f-b752-786d02385ee2 nodeName:}" failed. No retries permitted until 2026-07-06 20:06:32.406969425 +0000 UTC m=+43.618117622 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-registry-tls") pod "image-registry-5d4b6bdbbc-fr554" (UID: "11be38ab-74ab-495f-b752-786d02385ee2") : secret "image-registry-tls" not found Jul 06 20:06:28.412273 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:28.412248 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:28.499309 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:28.499220 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9vmsh" event={"ID":"ba49337f-cb8d-4c73-9a76-7945951eac79","Type":"ContainerStarted","Data":"38331de1f5cd138bb4ce5788175d60355c66f0fc7c9a11850efab684385aec94"} Jul 06 20:06:29.509633 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:29.509607 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:29.592984 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:29.592950 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:29.860564 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:29.860487 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:29.961882 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:29.961840 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:32.434388 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:32.434345 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-registry-tls\") pod \"image-registry-5d4b6bdbbc-fr554\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" Jul 06 20:06:32.434388 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:32.434391 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71661bcb-74d1-48e3-afa1-4acec1af79ae-metrics-tls\") pod \"dns-default-pwlxq\" (UID: \"71661bcb-74d1-48e3-afa1-4acec1af79ae\") " pod="openshift-dns/dns-default-pwlxq" Jul 06 20:06:32.434836 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:32.434513 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 20:06:32.434836 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:32.434519 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 20:06:32.434836 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:32.434537 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d4b6bdbbc-fr554: secret "image-registry-tls" not found Jul 06 20:06:32.434836 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:32.434564 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71661bcb-74d1-48e3-afa1-4acec1af79ae-metrics-tls podName:71661bcb-74d1-48e3-afa1-4acec1af79ae nodeName:}" failed. No retries permitted until 2026-07-06 20:06:40.434552096 +0000 UTC m=+51.645700292 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71661bcb-74d1-48e3-afa1-4acec1af79ae-metrics-tls") pod "dns-default-pwlxq" (UID: "71661bcb-74d1-48e3-afa1-4acec1af79ae") : secret "dns-default-metrics-tls" not found Jul 06 20:06:32.434836 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:32.434582 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-registry-tls podName:11be38ab-74ab-495f-b752-786d02385ee2 nodeName:}" failed. No retries permitted until 2026-07-06 20:06:40.434569435 +0000 UTC m=+51.645717631 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-registry-tls") pod "image-registry-5d4b6bdbbc-fr554" (UID: "11be38ab-74ab-495f-b752-786d02385ee2") : secret "image-registry-tls" not found Jul 06 20:06:33.140415 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:33.140369 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:33.588220 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:33.588174 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:33.697662 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:33.697631 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:33.727931 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:33.727886 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9vmsh" podStartSLOduration=6.736122709 podStartE2EDuration="9.727871821s" totalImagesPullingTime="2.991749112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:06:28.525785891 +0000 UTC m=+39.736934111" watchObservedRunningTime="2026-07-06 20:06:33.727871821 +0000 UTC m=+44.939020038" Jul 06 20:06:33.728622 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:33.728603 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-s2vnx"] Jul 06 20:06:33.750841 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:33.750816 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s2vnx" Jul 06 20:06:33.754028 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:33.754008 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 20:06:33.754028 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:33.754017 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 20:06:33.754028 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:33.754030 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 20:06:33.754494 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:33.754477 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 20:06:33.754570 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:33.754477 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nkxmb\"" Jul 06 20:06:33.771200 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:33.771180 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s2vnx"] Jul 06 20:06:33.802540 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:33.802503 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:33.847924 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:33.847858 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c2e525b8-01f4-4894-9604-e72935da6856-crio-socket\") pod \"insights-runtime-extractor-s2vnx\" (UID: \"c2e525b8-01f4-4894-9604-e72935da6856\") " pod="openshift-insights/insights-runtime-extractor-s2vnx" Jul 06 20:06:33.847924 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:33.847922 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c2e525b8-01f4-4894-9604-e72935da6856-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s2vnx\" (UID: \"c2e525b8-01f4-4894-9604-e72935da6856\") " pod="openshift-insights/insights-runtime-extractor-s2vnx" Jul 06 20:06:33.848048 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:33.847967 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c2e525b8-01f4-4894-9604-e72935da6856-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s2vnx\" (UID: \"c2e525b8-01f4-4894-9604-e72935da6856\") " pod="openshift-insights/insights-runtime-extractor-s2vnx" Jul 06 20:06:33.848048 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:33.847984 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2e525b8-01f4-4894-9604-e72935da6856-data-volume\") pod \"insights-runtime-extractor-s2vnx\" (UID: \"c2e525b8-01f4-4894-9604-e72935da6856\") " pod="openshift-insights/insights-runtime-extractor-s2vnx" Jul 06 20:06:33.848048 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:33.848000 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrzs6\" (UniqueName: \"kubernetes.io/projected/c2e525b8-01f4-4894-9604-e72935da6856-kube-api-access-wrzs6\") pod \"insights-runtime-extractor-s2vnx\" (UID: \"c2e525b8-01f4-4894-9604-e72935da6856\") " pod="openshift-insights/insights-runtime-extractor-s2vnx" Jul 06 20:06:33.948476 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:33.948419 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c2e525b8-01f4-4894-9604-e72935da6856-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s2vnx\" (UID: \"c2e525b8-01f4-4894-9604-e72935da6856\") " pod="openshift-insights/insights-runtime-extractor-s2vnx" Jul 06 20:06:33.948641 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:33.948495 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c2e525b8-01f4-4894-9604-e72935da6856-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s2vnx\" (UID: \"c2e525b8-01f4-4894-9604-e72935da6856\") " pod="openshift-insights/insights-runtime-extractor-s2vnx" Jul 06 20:06:33.948641 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:33.948524 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2e525b8-01f4-4894-9604-e72935da6856-data-volume\") pod \"insights-runtime-extractor-s2vnx\" (UID: \"c2e525b8-01f4-4894-9604-e72935da6856\") " pod="openshift-insights/insights-runtime-extractor-s2vnx" Jul 06 20:06:33.948641 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:33.948540 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrzs6\" (UniqueName: \"kubernetes.io/projected/c2e525b8-01f4-4894-9604-e72935da6856-kube-api-access-wrzs6\") pod \"insights-runtime-extractor-s2vnx\" (UID: \"c2e525b8-01f4-4894-9604-e72935da6856\") " pod="openshift-insights/insights-runtime-extractor-s2vnx" Jul 06 20:06:33.948641 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:33.948581 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 20:06:33.948641 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:33.948622 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 20:06:33.948838 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:33.948664 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2e525b8-01f4-4894-9604-e72935da6856-insights-runtime-extractor-tls podName:c2e525b8-01f4-4894-9604-e72935da6856 nodeName:}" failed. No retries permitted until 2026-07-06 20:06:34.44864263 +0000 UTC m=+45.659790849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c2e525b8-01f4-4894-9604-e72935da6856-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s2vnx" (UID: "c2e525b8-01f4-4894-9604-e72935da6856") : secret "insights-runtime-extractor-tls" not found Jul 06 20:06:33.948838 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:33.948588 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c2e525b8-01f4-4894-9604-e72935da6856-crio-socket\") pod \"insights-runtime-extractor-s2vnx\" (UID: \"c2e525b8-01f4-4894-9604-e72935da6856\") " pod="openshift-insights/insights-runtime-extractor-s2vnx" Jul 06 20:06:33.948838 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:33.948741 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c2e525b8-01f4-4894-9604-e72935da6856-kube-rbac-proxy-cm podName:c2e525b8-01f4-4894-9604-e72935da6856 nodeName:}" failed. No retries permitted until 2026-07-06 20:06:34.448723006 +0000 UTC m=+45.659871203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c2e525b8-01f4-4894-9604-e72935da6856-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s2vnx" (UID: "c2e525b8-01f4-4894-9604-e72935da6856") : configmap "kube-rbac-proxy" not found Jul 06 20:06:33.948838 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:33.948750 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c2e525b8-01f4-4894-9604-e72935da6856-crio-socket\") pod \"insights-runtime-extractor-s2vnx\" (UID: \"c2e525b8-01f4-4894-9604-e72935da6856\") " pod="openshift-insights/insights-runtime-extractor-s2vnx" Jul 06 20:06:33.948975 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:33.948842 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2e525b8-01f4-4894-9604-e72935da6856-data-volume\") pod \"insights-runtime-extractor-s2vnx\" (UID: \"c2e525b8-01f4-4894-9604-e72935da6856\") " pod="openshift-insights/insights-runtime-extractor-s2vnx" Jul 06 20:06:33.963300 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:33.963278 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrzs6\" (UniqueName: \"kubernetes.io/projected/c2e525b8-01f4-4894-9604-e72935da6856-kube-api-access-wrzs6\") pod \"insights-runtime-extractor-s2vnx\" (UID: \"c2e525b8-01f4-4894-9604-e72935da6856\") " pod="openshift-insights/insights-runtime-extractor-s2vnx" Jul 06 20:06:34.262936 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:34.262901 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:34.343840 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:34.343811 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:34.451269 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:34.451231 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c2e525b8-01f4-4894-9604-e72935da6856-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s2vnx\" (UID: \"c2e525b8-01f4-4894-9604-e72935da6856\") " pod="openshift-insights/insights-runtime-extractor-s2vnx" Jul 06 20:06:34.451491 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:34.451291 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c2e525b8-01f4-4894-9604-e72935da6856-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s2vnx\" (UID: \"c2e525b8-01f4-4894-9604-e72935da6856\") " pod="openshift-insights/insights-runtime-extractor-s2vnx" Jul 06 20:06:34.451491 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:34.451362 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 20:06:34.451491 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:34.451407 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 20:06:34.451491 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:34.451412 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c2e525b8-01f4-4894-9604-e72935da6856-kube-rbac-proxy-cm podName:c2e525b8-01f4-4894-9604-e72935da6856 nodeName:}" failed. No retries permitted until 2026-07-06 20:06:35.451400099 +0000 UTC m=+46.662548294 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c2e525b8-01f4-4894-9604-e72935da6856-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s2vnx" (UID: "c2e525b8-01f4-4894-9604-e72935da6856") : configmap "kube-rbac-proxy" not found Jul 06 20:06:34.451657 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:34.451520 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2e525b8-01f4-4894-9604-e72935da6856-insights-runtime-extractor-tls podName:c2e525b8-01f4-4894-9604-e72935da6856 nodeName:}" failed. No retries permitted until 2026-07-06 20:06:35.451501765 +0000 UTC m=+46.662649968 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c2e525b8-01f4-4894-9604-e72935da6856-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s2vnx" (UID: "c2e525b8-01f4-4894-9604-e72935da6856") : secret "insights-runtime-extractor-tls" not found Jul 06 20:06:34.463329 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:34.463301 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:34.555669 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:34.555576 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:34.650066 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:34.650025 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:35.459213 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:35.459178 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c2e525b8-01f4-4894-9604-e72935da6856-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s2vnx\" (UID: \"c2e525b8-01f4-4894-9604-e72935da6856\") " pod="openshift-insights/insights-runtime-extractor-s2vnx" Jul 06 20:06:35.459402 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:35.459258 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c2e525b8-01f4-4894-9604-e72935da6856-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s2vnx\" (UID: \"c2e525b8-01f4-4894-9604-e72935da6856\") " pod="openshift-insights/insights-runtime-extractor-s2vnx" Jul 06 20:06:35.459402 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:35.459335 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 20:06:35.459402 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:35.459345 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 20:06:35.459402 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:35.459402 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c2e525b8-01f4-4894-9604-e72935da6856-kube-rbac-proxy-cm podName:c2e525b8-01f4-4894-9604-e72935da6856 nodeName:}" failed. No retries permitted until 2026-07-06 20:06:37.459383875 +0000 UTC m=+48.670532076 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c2e525b8-01f4-4894-9604-e72935da6856-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s2vnx" (UID: "c2e525b8-01f4-4894-9604-e72935da6856") : configmap "kube-rbac-proxy" not found Jul 06 20:06:35.459673 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:35.459421 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2e525b8-01f4-4894-9604-e72935da6856-insights-runtime-extractor-tls podName:c2e525b8-01f4-4894-9604-e72935da6856 nodeName:}" failed. No retries permitted until 2026-07-06 20:06:37.459411825 +0000 UTC m=+48.670560034 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c2e525b8-01f4-4894-9604-e72935da6856-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s2vnx" (UID: "c2e525b8-01f4-4894-9604-e72935da6856") : secret "insights-runtime-extractor-tls" not found Jul 06 20:06:35.693995 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:35.693966 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j4d6m_f0b6b2dc-73c6-453c-9941-8738e48b4976/dns-node-resolver/0.log" Jul 06 20:06:35.872846 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:35.872756 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:35.939356 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:35.939322 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:36.703581 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:36.703556 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r6xsf_4e950de4-fc27-492d-ab51-69255ef36a84/node-ca/0.log" Jul 06 20:06:36.967278 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:36.967198 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:37.476789 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:37.476758 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c2e525b8-01f4-4894-9604-e72935da6856-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s2vnx\" (UID: \"c2e525b8-01f4-4894-9604-e72935da6856\") " pod="openshift-insights/insights-runtime-extractor-s2vnx" Jul 06 20:06:37.476960 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:37.476827 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c2e525b8-01f4-4894-9604-e72935da6856-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s2vnx\" (UID: \"c2e525b8-01f4-4894-9604-e72935da6856\") " pod="openshift-insights/insights-runtime-extractor-s2vnx" Jul 06 20:06:37.476960 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:37.476928 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 20:06:37.477092 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:37.476986 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2e525b8-01f4-4894-9604-e72935da6856-insights-runtime-extractor-tls podName:c2e525b8-01f4-4894-9604-e72935da6856 nodeName:}" failed. No retries permitted until 2026-07-06 20:06:41.476971691 +0000 UTC m=+52.688119886 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c2e525b8-01f4-4894-9604-e72935da6856-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s2vnx" (UID: "c2e525b8-01f4-4894-9604-e72935da6856") : secret "insights-runtime-extractor-tls" not found Jul 06 20:06:37.477092 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:37.476989 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 20:06:37.477092 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:37.477071 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c2e525b8-01f4-4894-9604-e72935da6856-kube-rbac-proxy-cm podName:c2e525b8-01f4-4894-9604-e72935da6856 nodeName:}" failed. No retries permitted until 2026-07-06 20:06:41.477050627 +0000 UTC m=+52.688198843 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c2e525b8-01f4-4894-9604-e72935da6856-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s2vnx" (UID: "c2e525b8-01f4-4894-9604-e72935da6856") : configmap "kube-rbac-proxy" not found Jul 06 20:06:37.697564 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:37.697537 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9vmsh_ba49337f-cb8d-4c73-9a76-7945951eac79/serve-healthcheck-canary/0.log" Jul 06 20:06:37.917579 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:37.917540 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:39.302783 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:39.302750 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:39.373637 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:39.373605 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:39.446258 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:39.446225 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:40.346133 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:40.346091 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:40.499635 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:40.499602 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-registry-tls\") pod \"image-registry-5d4b6bdbbc-fr554\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" Jul 06 20:06:40.499744 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:40.499639 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71661bcb-74d1-48e3-afa1-4acec1af79ae-metrics-tls\") pod \"dns-default-pwlxq\" (UID: \"71661bcb-74d1-48e3-afa1-4acec1af79ae\") " pod="openshift-dns/dns-default-pwlxq" Jul 06 20:06:40.499784 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:40.499755 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 20:06:40.499784 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:40.499772 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d4b6bdbbc-fr554: secret "image-registry-tls" not found Jul 06 20:06:40.499784 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:40.499775 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 20:06:40.499871 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:40.499825 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71661bcb-74d1-48e3-afa1-4acec1af79ae-metrics-tls podName:71661bcb-74d1-48e3-afa1-4acec1af79ae nodeName:}" failed. No retries permitted until 2026-07-06 20:06:56.499811812 +0000 UTC m=+67.710960008 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71661bcb-74d1-48e3-afa1-4acec1af79ae-metrics-tls") pod "dns-default-pwlxq" (UID: "71661bcb-74d1-48e3-afa1-4acec1af79ae") : secret "dns-default-metrics-tls" not found Jul 06 20:06:40.499871 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:40.499838 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-registry-tls podName:11be38ab-74ab-495f-b752-786d02385ee2 nodeName:}" failed. No retries permitted until 2026-07-06 20:06:56.499832007 +0000 UTC m=+67.710980203 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-registry-tls") pod "image-registry-5d4b6bdbbc-fr554" (UID: "11be38ab-74ab-495f-b752-786d02385ee2") : secret "image-registry-tls" not found Jul 06 20:06:40.611276 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:40.611184 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:40.857050 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:40.857022 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:40.955540 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:40.955505 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:41.070684 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:41.070652 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:41.158243 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:41.158210 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:41.507554 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:41.507515 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c2e525b8-01f4-4894-9604-e72935da6856-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s2vnx\" (UID: \"c2e525b8-01f4-4894-9604-e72935da6856\") " pod="openshift-insights/insights-runtime-extractor-s2vnx" Jul 06 20:06:41.507966 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:41.507575 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c2e525b8-01f4-4894-9604-e72935da6856-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s2vnx\" (UID: \"c2e525b8-01f4-4894-9604-e72935da6856\") " pod="openshift-insights/insights-runtime-extractor-s2vnx" Jul 06 20:06:41.507966 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:41.507667 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 20:06:41.507966 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:41.507730 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2e525b8-01f4-4894-9604-e72935da6856-insights-runtime-extractor-tls podName:c2e525b8-01f4-4894-9604-e72935da6856 nodeName:}" failed. No retries permitted until 2026-07-06 20:06:49.507716164 +0000 UTC m=+60.718864360 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c2e525b8-01f4-4894-9604-e72935da6856-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s2vnx" (UID: "c2e525b8-01f4-4894-9604-e72935da6856") : secret "insights-runtime-extractor-tls" not found Jul 06 20:06:41.508084 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:41.508032 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c2e525b8-01f4-4894-9604-e72935da6856-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s2vnx\" (UID: \"c2e525b8-01f4-4894-9604-e72935da6856\") " pod="openshift-insights/insights-runtime-extractor-s2vnx" Jul 06 20:06:41.844282 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:41.844187 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:42.262772 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:42.262738 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:42.640821 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:42.640730 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:43.058219 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:43.058182 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:43.163545 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:43.163504 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:43.437118 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:43.437083 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:43.820408 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:43.820325 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:43.893064 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:43.893029 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:44.050182 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:44.050154 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:44.191301 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:44.191274 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:44.285694 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:44.285657 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:44.451378 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:44.451293 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:44.858152 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:44.858059 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:44.951395 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:44.951358 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:45.446398 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:45.446364 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:46.039717 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:46.039677 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:46.171216 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:46.171179 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:46.274469 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:46.274414 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:46.333715 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:46.333632 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:46.443611 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:46.443583 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:46.643518 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:46.643490 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:46.745289 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:46.745254 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:47.479117 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:47.479076 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:47.842657 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:47.842560 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:48.269191 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:48.269158 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:48.562242 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:48.562160 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:48.667894 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:48.667855 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:48.784974 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:48.784942 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:48.894424 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:48.894390 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:49.021247 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:49.021218 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:49.163431 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:49.163350 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:49.268562 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:49.268532 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:49.343392 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:49.343354 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:49.570502 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:49.570386 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c2e525b8-01f4-4894-9604-e72935da6856-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s2vnx\" (UID: \"c2e525b8-01f4-4894-9604-e72935da6856\") " pod="openshift-insights/insights-runtime-extractor-s2vnx" Jul 06 20:06:49.572804 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:49.572775 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c2e525b8-01f4-4894-9604-e72935da6856-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s2vnx\" (UID: \"c2e525b8-01f4-4894-9604-e72935da6856\") " pod="openshift-insights/insights-runtime-extractor-s2vnx" Jul 06 20:06:49.664768 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:49.664733 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nkxmb\"" Jul 06 20:06:49.670811 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:49.670792 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s2vnx" Jul 06 20:06:49.844121 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:49.844034 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s2vnx"] Jul 06 20:06:49.850987 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:06:49.850959 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2e525b8_01f4_4894_9604_e72935da6856.slice/crio-1b964166045619801c325d35ccfe178220aec4765228d05593e3e3b3cf707f9a WatchSource:0}: Error finding container 1b964166045619801c325d35ccfe178220aec4765228d05593e3e3b3cf707f9a: Status 404 returned error can't find the container with id 1b964166045619801c325d35ccfe178220aec4765228d05593e3e3b3cf707f9a Jul 06 20:06:49.868166 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:49.868139 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:50.174848 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:50.174807 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:50.414553 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:50.414520 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:50.511700 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:50.511592 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:50.549387 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:50.549304 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s2vnx" event={"ID":"c2e525b8-01f4-4894-9604-e72935da6856","Type":"ContainerStarted","Data":"1b964166045619801c325d35ccfe178220aec4765228d05593e3e3b3cf707f9a"} Jul 06 20:06:50.662963 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:50.662923 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:50.849342 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:50.849254 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:51.090779 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:51.090743 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:51.459640 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:51.459595 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:51.483798 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:51.483771 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dnfgp" Jul 06 20:06:54.361121 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:54.361089 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:54.432750 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:54.432672 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:54.561612 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:54.561585 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s2vnx" event={"ID":"c2e525b8-01f4-4894-9604-e72935da6856","Type":"ContainerStarted","Data":"9ebad063882b8b4da527af6d8ff9a5ade24ef17badfaccdcb7bd5537dd2b6e93"} Jul 06 20:06:54.561776 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:54.561643 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s2vnx" event={"ID":"c2e525b8-01f4-4894-9604-e72935da6856","Type":"ContainerStarted","Data":"38d68d6620b2d2e52079e84c5aa6063495a2730ba55def341d5517e1e21d8350"} Jul 06 20:06:54.993712 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:54.993677 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65f96cbc-gffz8"] Jul 06 20:06:54.997247 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:54.997224 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d7877775d-62h55"] Jul 06 20:06:54.997406 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:54.997385 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65f96cbc-gffz8" Jul 06 20:06:55.000627 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.000596 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 06 20:06:55.000748 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.000637 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 06 20:06:55.000748 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.000667 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d7877775d-62h55" Jul 06 20:06:55.000897 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.000883 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 06 20:06:55.001473 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.001439 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 06 20:06:55.001891 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.001872 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 06 20:06:55.002573 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.002554 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 06 20:06:55.003098 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.003078 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 06 20:06:55.003832 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.003812 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 06 20:06:55.019715 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.019648 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d7877775d-62h55"] Jul 06 20:06:55.020521 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.020496 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65f96cbc-gffz8"] Jul 06 20:06:55.026135 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.026113 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d4b6bdbbc-fr554"] Jul 06 20:06:55.026361 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:06:55.026336 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" podUID="11be38ab-74ab-495f-b752-786d02385ee2" Jul 06 20:06:55.114852 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.114817 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klmp2\" (UniqueName: \"kubernetes.io/projected/18f5aff3-4753-452a-a117-39e6a98fdd18-kube-api-access-klmp2\") pod \"cluster-proxy-proxy-agent-6b65f96cbc-gffz8\" (UID: \"18f5aff3-4753-452a-a117-39e6a98fdd18\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65f96cbc-gffz8" Jul 06 20:06:55.115031 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.114878 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/18f5aff3-4753-452a-a117-39e6a98fdd18-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b65f96cbc-gffz8\" (UID: \"18f5aff3-4753-452a-a117-39e6a98fdd18\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65f96cbc-gffz8" Jul 06 20:06:55.115031 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.114979 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/06fab944-3999-4d28-b75e-4110bbbc1170-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d7877775d-62h55\" (UID: \"06fab944-3999-4d28-b75e-4110bbbc1170\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d7877775d-62h55" Jul 06 20:06:55.115031 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.115004 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg57p\" (UniqueName: \"kubernetes.io/projected/06fab944-3999-4d28-b75e-4110bbbc1170-kube-api-access-wg57p\") pod \"klusterlet-addon-workmgr-6d7877775d-62h55\" (UID: \"06fab944-3999-4d28-b75e-4110bbbc1170\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d7877775d-62h55" Jul 06 20:06:55.115031 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.115022 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/18f5aff3-4753-452a-a117-39e6a98fdd18-ca\") pod \"cluster-proxy-proxy-agent-6b65f96cbc-gffz8\" (UID: \"18f5aff3-4753-452a-a117-39e6a98fdd18\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65f96cbc-gffz8" Jul 06 20:06:55.115200 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.115042 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/18f5aff3-4753-452a-a117-39e6a98fdd18-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b65f96cbc-gffz8\" (UID: \"18f5aff3-4753-452a-a117-39e6a98fdd18\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65f96cbc-gffz8" Jul 06 20:06:55.115200 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.115088 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f13799-7290-425e-be8d-b0bef20ea3c4-metrics-certs\") pod \"network-metrics-daemon-5m47w\" (UID: \"d7f13799-7290-425e-be8d-b0bef20ea3c4\") " pod="openshift-multus/network-metrics-daemon-5m47w" Jul 06 20:06:55.115200 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.115165 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ef938dd-1047-4283-a4f5-d0d0abb4eda0-original-pull-secret\") pod \"global-pull-secret-syncer-jmp82\" (UID: \"4ef938dd-1047-4283-a4f5-d0d0abb4eda0\") " pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:06:55.115344 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.115257 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/18f5aff3-4753-452a-a117-39e6a98fdd18-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b65f96cbc-gffz8\" (UID: \"18f5aff3-4753-452a-a117-39e6a98fdd18\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65f96cbc-gffz8" Jul 06 20:06:55.115344 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.115298 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06fab944-3999-4d28-b75e-4110bbbc1170-tmp\") pod \"klusterlet-addon-workmgr-6d7877775d-62h55\" (UID: \"06fab944-3999-4d28-b75e-4110bbbc1170\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d7877775d-62h55" Jul 06 20:06:55.115344 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.115317 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/18f5aff3-4753-452a-a117-39e6a98fdd18-hub\") pod \"cluster-proxy-proxy-agent-6b65f96cbc-gffz8\" (UID: \"18f5aff3-4753-452a-a117-39e6a98fdd18\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65f96cbc-gffz8" Jul 06 20:06:55.117977 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.117959 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 20:06:55.118736 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.118699 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 20:06:55.128335 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.128311 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4ef938dd-1047-4283-a4f5-d0d0abb4eda0-original-pull-secret\") pod \"global-pull-secret-syncer-jmp82\" (UID: \"4ef938dd-1047-4283-a4f5-d0d0abb4eda0\") " pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:06:55.128335 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.128322 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f13799-7290-425e-be8d-b0bef20ea3c4-metrics-certs\") pod \"network-metrics-daemon-5m47w\" (UID: \"d7f13799-7290-425e-be8d-b0bef20ea3c4\") " pod="openshift-multus/network-metrics-daemon-5m47w" Jul 06 20:06:55.138961 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.138882 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:55.216657 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.216621 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg57p\" (UniqueName: \"kubernetes.io/projected/06fab944-3999-4d28-b75e-4110bbbc1170-kube-api-access-wg57p\") pod \"klusterlet-addon-workmgr-6d7877775d-62h55\" (UID: \"06fab944-3999-4d28-b75e-4110bbbc1170\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d7877775d-62h55" Jul 06 20:06:55.216657 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.216670 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/18f5aff3-4753-452a-a117-39e6a98fdd18-ca\") pod \"cluster-proxy-proxy-agent-6b65f96cbc-gffz8\" (UID: \"18f5aff3-4753-452a-a117-39e6a98fdd18\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65f96cbc-gffz8" Jul 06 20:06:55.216907 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.216701 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/18f5aff3-4753-452a-a117-39e6a98fdd18-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b65f96cbc-gffz8\" (UID: \"18f5aff3-4753-452a-a117-39e6a98fdd18\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65f96cbc-gffz8" Jul 06 20:06:55.216962 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.216926 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/18f5aff3-4753-452a-a117-39e6a98fdd18-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b65f96cbc-gffz8\" (UID: \"18f5aff3-4753-452a-a117-39e6a98fdd18\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65f96cbc-gffz8" Jul 06 20:06:55.217010 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.216962 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06fab944-3999-4d28-b75e-4110bbbc1170-tmp\") pod \"klusterlet-addon-workmgr-6d7877775d-62h55\" (UID: \"06fab944-3999-4d28-b75e-4110bbbc1170\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d7877775d-62h55" Jul 06 20:06:55.217010 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.216988 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/18f5aff3-4753-452a-a117-39e6a98fdd18-hub\") pod \"cluster-proxy-proxy-agent-6b65f96cbc-gffz8\" (UID: \"18f5aff3-4753-452a-a117-39e6a98fdd18\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65f96cbc-gffz8" Jul 06 20:06:55.217703 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.217191 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klmp2\" (UniqueName: \"kubernetes.io/projected/18f5aff3-4753-452a-a117-39e6a98fdd18-kube-api-access-klmp2\") pod \"cluster-proxy-proxy-agent-6b65f96cbc-gffz8\" (UID: \"18f5aff3-4753-452a-a117-39e6a98fdd18\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65f96cbc-gffz8" Jul 06 20:06:55.217703 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.217237 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqx4z\" (UniqueName: \"kubernetes.io/projected/e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e-kube-api-access-dqx4z\") pod \"network-check-target-mbz65\" (UID: \"e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e\") " pod="openshift-network-diagnostics/network-check-target-mbz65" Jul 06 20:06:55.217703 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.217283 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/18f5aff3-4753-452a-a117-39e6a98fdd18-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b65f96cbc-gffz8\" (UID: \"18f5aff3-4753-452a-a117-39e6a98fdd18\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65f96cbc-gffz8" Jul 06 20:06:55.217703 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.217332 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/06fab944-3999-4d28-b75e-4110bbbc1170-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d7877775d-62h55\" (UID: \"06fab944-3999-4d28-b75e-4110bbbc1170\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d7877775d-62h55" Jul 06 20:06:55.217703 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.217353 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06fab944-3999-4d28-b75e-4110bbbc1170-tmp\") pod \"klusterlet-addon-workmgr-6d7877775d-62h55\" (UID: \"06fab944-3999-4d28-b75e-4110bbbc1170\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d7877775d-62h55" Jul 06 20:06:55.218210 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.218184 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/18f5aff3-4753-452a-a117-39e6a98fdd18-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b65f96cbc-gffz8\" (UID: \"18f5aff3-4753-452a-a117-39e6a98fdd18\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65f96cbc-gffz8" Jul 06 20:06:55.219813 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.219783 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/18f5aff3-4753-452a-a117-39e6a98fdd18-ca\") pod \"cluster-proxy-proxy-agent-6b65f96cbc-gffz8\" (UID: \"18f5aff3-4753-452a-a117-39e6a98fdd18\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65f96cbc-gffz8" Jul 06 20:06:55.220443 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.220419 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/06fab944-3999-4d28-b75e-4110bbbc1170-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d7877775d-62h55\" (UID: \"06fab944-3999-4d28-b75e-4110bbbc1170\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d7877775d-62h55" Jul 06 20:06:55.221196 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.221173 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/18f5aff3-4753-452a-a117-39e6a98fdd18-hub\") pod \"cluster-proxy-proxy-agent-6b65f96cbc-gffz8\" (UID: \"18f5aff3-4753-452a-a117-39e6a98fdd18\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65f96cbc-gffz8" Jul 06 20:06:55.221277 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.221217 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/18f5aff3-4753-452a-a117-39e6a98fdd18-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b65f96cbc-gffz8\" (UID: \"18f5aff3-4753-452a-a117-39e6a98fdd18\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65f96cbc-gffz8" Jul 06 20:06:55.221277 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.221218 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/18f5aff3-4753-452a-a117-39e6a98fdd18-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b65f96cbc-gffz8\" (UID: \"18f5aff3-4753-452a-a117-39e6a98fdd18\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65f96cbc-gffz8" Jul 06 20:06:55.230335 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.230311 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 20:06:55.230473 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.230369 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:55.242030 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.242009 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 20:06:55.242811 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.242788 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg57p\" (UniqueName: \"kubernetes.io/projected/06fab944-3999-4d28-b75e-4110bbbc1170-kube-api-access-wg57p\") pod \"klusterlet-addon-workmgr-6d7877775d-62h55\" (UID: \"06fab944-3999-4d28-b75e-4110bbbc1170\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d7877775d-62h55" Jul 06 20:06:55.242883 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.242822 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klmp2\" (UniqueName: \"kubernetes.io/projected/18f5aff3-4753-452a-a117-39e6a98fdd18-kube-api-access-klmp2\") pod \"cluster-proxy-proxy-agent-6b65f96cbc-gffz8\" (UID: \"18f5aff3-4753-452a-a117-39e6a98fdd18\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65f96cbc-gffz8" Jul 06 20:06:55.251131 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.251078 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqx4z\" (UniqueName: \"kubernetes.io/projected/e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e-kube-api-access-dqx4z\") pod \"network-check-target-mbz65\" (UID: \"e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e\") " pod="openshift-network-diagnostics/network-check-target-mbz65" Jul 06 20:06:55.301296 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.301263 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jmp82" Jul 06 20:06:55.320630 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.320598 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65f96cbc-gffz8" Jul 06 20:06:55.327766 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.327735 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d7877775d-62h55" Jul 06 20:06:55.333896 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.333854 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:55.426511 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.426485 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vntfk\"" Jul 06 20:06:55.426930 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.426807 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mt2s9\"" Jul 06 20:06:55.429606 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.429551 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5m47w" Jul 06 20:06:55.434558 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.434533 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbz65" Jul 06 20:06:55.523703 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.523586 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jmp82"] Jul 06 20:06:55.560315 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.560280 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65f96cbc-gffz8"] Jul 06 20:06:55.566256 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:06:55.566218 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18f5aff3_4753_452a_a117_39e6a98fdd18.slice/crio-a090d8ace7d10338b750b03833f851ca5683f524c10d78c82bc21720514911aa WatchSource:0}: Error finding container a090d8ace7d10338b750b03833f851ca5683f524c10d78c82bc21720514911aa: Status 404 returned error can't find the container with id a090d8ace7d10338b750b03833f851ca5683f524c10d78c82bc21720514911aa Jul 06 20:06:55.567601 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.567578 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d7877775d-62h55"] Jul 06 20:06:55.568971 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.568844 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" Jul 06 20:06:55.569069 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.569046 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jmp82" event={"ID":"4ef938dd-1047-4283-a4f5-d0d0abb4eda0","Type":"ContainerStarted","Data":"253a46e9575d42a481b8e0f266acd3836f1b467a5aa2944ff280ed9e46f165ef"} Jul 06 20:06:55.595702 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.595674 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" Jul 06 20:06:55.647954 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.647918 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mbz65"] Jul 06 20:06:55.651692 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:06:55.651574 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode26d5fb5_e4e3_42b2_bd13_ee6853bfad9e.slice/crio-16eadd7df94ee9085f582222a110a30ae52c673d58676b8a9098c70920a9c0ae WatchSource:0}: Error finding container 16eadd7df94ee9085f582222a110a30ae52c673d58676b8a9098c70920a9c0ae: Status 404 returned error can't find the container with id 16eadd7df94ee9085f582222a110a30ae52c673d58676b8a9098c70920a9c0ae Jul 06 20:06:55.672911 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.672887 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5m47w"] Jul 06 20:06:55.674206 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:06:55.674186 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f13799_7290_425e_be8d_b0bef20ea3c4.slice/crio-63b925df368b7af4363f578f74c4a94fc124475cb217c62038a46c72bc3dfe83 WatchSource:0}: Error finding container 63b925df368b7af4363f578f74c4a94fc124475cb217c62038a46c72bc3dfe83: Status 404 returned error can't find the container with id 63b925df368b7af4363f578f74c4a94fc124475cb217c62038a46c72bc3dfe83 Jul 06 20:06:55.723838 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.723808 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11be38ab-74ab-495f-b752-786d02385ee2-trusted-ca\") pod \"11be38ab-74ab-495f-b752-786d02385ee2\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " Jul 06 20:06:55.723838 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.723841 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/11be38ab-74ab-495f-b752-786d02385ee2-image-registry-private-configuration\") pod \"11be38ab-74ab-495f-b752-786d02385ee2\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " Jul 06 20:06:55.724041 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.723892 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blvtm\" (UniqueName: \"kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-kube-api-access-blvtm\") pod \"11be38ab-74ab-495f-b752-786d02385ee2\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " Jul 06 20:06:55.724041 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.723921 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/11be38ab-74ab-495f-b752-786d02385ee2-ca-trust-extracted\") pod \"11be38ab-74ab-495f-b752-786d02385ee2\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " Jul 06 20:06:55.724041 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.723962 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/11be38ab-74ab-495f-b752-786d02385ee2-installation-pull-secrets\") pod \"11be38ab-74ab-495f-b752-786d02385ee2\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " Jul 06 20:06:55.724041 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.723993 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-bound-sa-token\") pod \"11be38ab-74ab-495f-b752-786d02385ee2\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " Jul 06 20:06:55.724041 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.724033 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/11be38ab-74ab-495f-b752-786d02385ee2-registry-certificates\") pod \"11be38ab-74ab-495f-b752-786d02385ee2\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " Jul 06 20:06:55.724278 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.724226 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11be38ab-74ab-495f-b752-786d02385ee2-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "11be38ab-74ab-495f-b752-786d02385ee2" (UID: "11be38ab-74ab-495f-b752-786d02385ee2"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:06:55.724333 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.724277 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11be38ab-74ab-495f-b752-786d02385ee2-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "11be38ab-74ab-495f-b752-786d02385ee2" (UID: "11be38ab-74ab-495f-b752-786d02385ee2"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:06:55.724512 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.724490 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11be38ab-74ab-495f-b752-786d02385ee2-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "11be38ab-74ab-495f-b752-786d02385ee2" (UID: "11be38ab-74ab-495f-b752-786d02385ee2"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:06:55.726290 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.726268 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "11be38ab-74ab-495f-b752-786d02385ee2" (UID: "11be38ab-74ab-495f-b752-786d02385ee2"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:06:55.726368 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.726281 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-kube-api-access-blvtm" (OuterVolumeSpecName: "kube-api-access-blvtm") pod "11be38ab-74ab-495f-b752-786d02385ee2" (UID: "11be38ab-74ab-495f-b752-786d02385ee2"). InnerVolumeSpecName "kube-api-access-blvtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:06:55.726607 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.726589 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11be38ab-74ab-495f-b752-786d02385ee2-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "11be38ab-74ab-495f-b752-786d02385ee2" (UID: "11be38ab-74ab-495f-b752-786d02385ee2"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:06:55.726679 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.726605 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11be38ab-74ab-495f-b752-786d02385ee2-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "11be38ab-74ab-495f-b752-786d02385ee2" (UID: "11be38ab-74ab-495f-b752-786d02385ee2"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:06:55.825339 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.825255 2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/11be38ab-74ab-495f-b752-786d02385ee2-installation-pull-secrets\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:06:55.825339 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.825287 2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-bound-sa-token\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:06:55.825339 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.825300 2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/11be38ab-74ab-495f-b752-786d02385ee2-registry-certificates\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:06:55.825339 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.825311 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11be38ab-74ab-495f-b752-786d02385ee2-trusted-ca\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:06:55.825339 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.825324 2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/11be38ab-74ab-495f-b752-786d02385ee2-image-registry-private-configuration\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:06:55.825339 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.825336 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blvtm\" (UniqueName: \"kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-kube-api-access-blvtm\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:06:55.825683 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.825350 2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/11be38ab-74ab-495f-b752-786d02385ee2-ca-trust-extracted\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:06:55.986694 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:55.986656 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:56.078389 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:56.078308 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:56.176822 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:56.176783 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:56.266629 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:56.266591 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:56.532498 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:56.532444 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71661bcb-74d1-48e3-afa1-4acec1af79ae-metrics-tls\") pod \"dns-default-pwlxq\" (UID: \"71661bcb-74d1-48e3-afa1-4acec1af79ae\") " pod="openshift-dns/dns-default-pwlxq" Jul 06 20:06:56.532498 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:56.532501 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-registry-tls\") pod \"image-registry-5d4b6bdbbc-fr554\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" Jul 06 20:06:56.535437 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:56.535408 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71661bcb-74d1-48e3-afa1-4acec1af79ae-metrics-tls\") pod \"dns-default-pwlxq\" (UID: \"71661bcb-74d1-48e3-afa1-4acec1af79ae\") " pod="openshift-dns/dns-default-pwlxq" Jul 06 20:06:56.535584 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:56.535508 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-registry-tls\") pod \"image-registry-5d4b6bdbbc-fr554\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" Jul 06 20:06:56.573314 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:56.573233 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5m47w" event={"ID":"d7f13799-7290-425e-be8d-b0bef20ea3c4","Type":"ContainerStarted","Data":"63b925df368b7af4363f578f74c4a94fc124475cb217c62038a46c72bc3dfe83"} Jul 06 20:06:56.574444 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:56.574413 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d7877775d-62h55" event={"ID":"06fab944-3999-4d28-b75e-4110bbbc1170","Type":"ContainerStarted","Data":"8001060759b03e66b4e6ffcb6022f77de2cafda6809ecdf6ab0a4bb51fd29048"} Jul 06 20:06:56.575712 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:56.575683 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mbz65" event={"ID":"e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e","Type":"ContainerStarted","Data":"16eadd7df94ee9085f582222a110a30ae52c673d58676b8a9098c70920a9c0ae"} Jul 06 20:06:56.576875 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:56.576837 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65f96cbc-gffz8" event={"ID":"18f5aff3-4753-452a-a117-39e6a98fdd18","Type":"ContainerStarted","Data":"a090d8ace7d10338b750b03833f851ca5683f524c10d78c82bc21720514911aa"} Jul 06 20:06:56.576875 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:56.576857 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d4b6bdbbc-fr554" Jul 06 20:06:56.633485 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:56.633433 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-registry-tls\") pod \"11be38ab-74ab-495f-b752-786d02385ee2\" (UID: \"11be38ab-74ab-495f-b752-786d02385ee2\") " Jul 06 20:06:56.635738 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:56.635710 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "11be38ab-74ab-495f-b752-786d02385ee2" (UID: "11be38ab-74ab-495f-b752-786d02385ee2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:06:56.734770 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:56.734728 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11be38ab-74ab-495f-b752-786d02385ee2-registry-tls\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:06:56.742102 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:56.742076 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5vcsc\"" Jul 06 20:06:56.749407 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:56.749382 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pwlxq" Jul 06 20:06:56.932021 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:56.931983 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d4b6bdbbc-fr554"] Jul 06 20:06:56.935220 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:56.935183 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5d4b6bdbbc-fr554"] Jul 06 20:06:56.964761 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:56.964732 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pwlxq"] Jul 06 20:06:56.971408 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:06:56.971383 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71661bcb_74d1_48e3_afa1_4acec1af79ae.slice/crio-5af4889efdad16a4a80eced95261ec019679ef7533e30c0e4c8437e103806176 WatchSource:0}: Error finding container 5af4889efdad16a4a80eced95261ec019679ef7533e30c0e4c8437e103806176: Status 404 returned error can't find the container with id 5af4889efdad16a4a80eced95261ec019679ef7533e30c0e4c8437e103806176 Jul 06 20:06:57.299122 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:57.298908 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11be38ab-74ab-495f-b752-786d02385ee2" path="/var/lib/kubelet/pods/11be38ab-74ab-495f-b752-786d02385ee2/volumes" Jul 06 20:06:57.585697 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:57.584822 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s2vnx" event={"ID":"c2e525b8-01f4-4894-9604-e72935da6856","Type":"ContainerStarted","Data":"550f64edc139106fae82a171f9d4f9b2d193554101f6d00f0a583deb7293bac1"} Jul 06 20:06:57.590711 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:57.590648 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pwlxq" event={"ID":"71661bcb-74d1-48e3-afa1-4acec1af79ae","Type":"ContainerStarted","Data":"5af4889efdad16a4a80eced95261ec019679ef7533e30c0e4c8437e103806176"} Jul 06 20:06:57.610241 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:57.610044 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-s2vnx" podStartSLOduration=17.714002979 podStartE2EDuration="24.6100249s" totalImagesPullingTime="6.896021921s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:06:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:06:57.609287341 +0000 UTC m=+68.820435562" watchObservedRunningTime="2026-07-06 20:06:57.6100249 +0000 UTC m=+68.821173110" Jul 06 20:06:58.602181 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:58.602099 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5m47w" event={"ID":"d7f13799-7290-425e-be8d-b0bef20ea3c4","Type":"ContainerStarted","Data":"355d37f23476a4b949d65f6c6756bb118bcf0b0284a25ad597598662920d7b13"} Jul 06 20:06:58.602181 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:58.602162 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5m47w" event={"ID":"d7f13799-7290-425e-be8d-b0bef20ea3c4","Type":"ContainerStarted","Data":"fe60dd41dc715f9d551d827b8559a1d5ae755912a0874988967ee0214349c080"} Jul 06 20:06:58.752611 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:58.752540 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:58.824570 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:06:58.824523 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:03.169239 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:03.169169 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:03.251003 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:03.250962 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:03.359173 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:03.359097 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:03.437019 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:03.436936 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:04.626920 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:04.626875 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mbz65" event={"ID":"e26d5fb5-e4e3-42b2-bd13-ee6853bfad9e","Type":"ContainerStarted","Data":"4ad0e8ae575a7618a8945c0b51707eb4457f2f25f4fea84e93fbb00ddfe0cbbe"} Jul 06 20:07:04.627385 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:04.626995 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mbz65" Jul 06 20:07:04.629278 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:04.629251 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65f96cbc-gffz8" event={"ID":"18f5aff3-4753-452a-a117-39e6a98fdd18","Type":"ContainerStarted","Data":"a0f9720d164d1df5712c32b80b4fc5eb89ce3427807fca0ee9c7c0af6d1fb5d1"} Jul 06 20:07:04.629401 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:04.629290 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65f96cbc-gffz8" event={"ID":"18f5aff3-4753-452a-a117-39e6a98fdd18","Type":"ContainerStarted","Data":"4bf15750a05f9879405146ce7be6397ce312faed251d31f2dcb61f8166590135"} Jul 06 20:07:04.629401 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:04.629308 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65f96cbc-gffz8" event={"ID":"18f5aff3-4753-452a-a117-39e6a98fdd18","Type":"ContainerStarted","Data":"2dbc5af278a5ceabc80a9f08c55290d1928811c34beee1f3320c4bb09d92bad9"} Jul 06 20:07:04.631330 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:04.631306 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pwlxq" event={"ID":"71661bcb-74d1-48e3-afa1-4acec1af79ae","Type":"ContainerStarted","Data":"fad7565856970e3b045de95560757852d2f171b3130f54a668d1b024dd1caa69"} Jul 06 20:07:04.631428 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:04.631358 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pwlxq" Jul 06 20:07:04.631428 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:04.631381 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pwlxq" event={"ID":"71661bcb-74d1-48e3-afa1-4acec1af79ae","Type":"ContainerStarted","Data":"72597d18e08b63e421c964b3462a76e6674675f3d8d07a4edc0c4f7f5eed501f"} Jul 06 20:07:04.633009 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:04.632983 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jmp82" event={"ID":"4ef938dd-1047-4283-a4f5-d0d0abb4eda0","Type":"ContainerStarted","Data":"c9501683480f44e109b4e936674684776a40837bc72f4164777bf20165c54a9d"} Jul 06 20:07:04.651227 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:04.651183 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mbz65" podStartSLOduration=66.592003911 podStartE2EDuration="1m14.651167397s" totalImagesPullingTime="8.059163486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:05:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:07:04.64961469 +0000 UTC m=+75.860762908" watchObservedRunningTime="2026-07-06 20:07:04.651167397 +0000 UTC m=+75.862315617" Jul 06 20:07:04.651427 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:04.651404 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5m47w" podStartSLOduration=72.319973012 podStartE2EDuration="1m14.651397146s" totalImagesPullingTime="2.331424134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:05:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:06:58.627240072 +0000 UTC m=+69.838388290" watchObservedRunningTime="2026-07-06 20:07:04.651397146 +0000 UTC m=+75.862545392" Jul 06 20:07:04.674149 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:04.674101 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jmp82" podStartSLOduration=66.495408343 podStartE2EDuration="1m14.674081881s" totalImagesPullingTime="8.178673538s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:05:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:07:04.673947739 +0000 UTC m=+75.885095970" watchObservedRunningTime="2026-07-06 20:07:04.674081881 +0000 UTC m=+75.885230095" Jul 06 20:07:04.701470 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:04.701414 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pwlxq" podStartSLOduration=33.976041316999996 podStartE2EDuration="40.701400857s" totalImagesPullingTime="6.72535954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:07:04.70102569 +0000 UTC m=+75.912173909" watchObservedRunningTime="2026-07-06 20:07:04.701400857 +0000 UTC m=+75.912549076" Jul 06 20:07:04.733659 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:04.733542 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b65f96cbc-gffz8" podStartSLOduration=2.58811875 podStartE2EDuration="10.733520585s" totalImagesPullingTime="8.145401835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:06:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:07:04.732126839 +0000 UTC m=+75.943275059" watchObservedRunningTime="2026-07-06 20:07:04.733520585 +0000 UTC m=+75.944668806" Jul 06 20:07:06.207998 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:06.207963 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:06.310447 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:06.310403 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:06.640001 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:06.639921 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d7877775d-62h55" event={"ID":"06fab944-3999-4d28-b75e-4110bbbc1170","Type":"ContainerStarted","Data":"c92ff2ef7116723cb171fa7d7d6b63317ee7a282e19464f729eec2d97233369f"} Jul 06 20:07:06.640149 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:06.640128 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d7877775d-62h55" Jul 06 20:07:06.641717 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:06.641701 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d7877775d-62h55" Jul 06 20:07:06.662276 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:06.662230 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d7877775d-62h55" podStartSLOduration=2.6444003719999998 podStartE2EDuration="12.662216377s" totalImagesPullingTime="10.017816005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:06:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:07:06.661918564 +0000 UTC m=+77.873066783" watchObservedRunningTime="2026-07-06 20:07:06.662216377 +0000 UTC m=+77.873364595" Jul 06 20:07:07.374327 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:07.374285 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:07.464888 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:07.464854 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:07.536560 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:07.536526 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:07.642055 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:07.642027 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:07.702952 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:07.702917 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:08.596683 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:08.596639 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:08.697166 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:08.697134 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:09.364429 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:09.364380 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:09.469269 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:09.469151 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:09.632060 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:09.631968 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:12.576585 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:12.576544 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:12.677323 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:12.677285 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:14.582995 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:14.582958 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:14.637962 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:14.637934 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pwlxq" Jul 06 20:07:14.682785 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:14.682757 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:15.294239 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:15.294199 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:15.358230 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:15.358196 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:16.370098 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:16.370059 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:16.436125 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:16.436086 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:18.423741 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:18.423698 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:18.516676 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:18.516632 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:18.629846 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:18.629805 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:18.711538 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:18.711435 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:18.809437 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:18.809408 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:18.912178 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:18.912148 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:19.789583 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:19.789554 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:19.893271 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:19.893242 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:21.245029 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:21.244987 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:21.345124 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:21.345089 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:21.668283 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:21.668243 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:21.764375 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:21.764334 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:21.957155 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:21.957073 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:22.028074 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.028046 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-h45hc"] Jul 06 20:07:22.031136 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.031116 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h45hc" Jul 06 20:07:22.036528 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.036498 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lbch4\"" Jul 06 20:07:22.037485 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.037374 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 20:07:22.037485 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.037408 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 20:07:22.037622 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.037565 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 20:07:22.038747 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.038728 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 20:07:22.038853 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.038780 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 20:07:22.039246 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.039231 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 20:07:22.083508 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.083483 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:22.110744 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.110714 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/40f67c9c-cf00-4f97-a5b3-b4698737b55f-node-exporter-wtmp\") pod \"node-exporter-h45hc\" (UID: \"40f67c9c-cf00-4f97-a5b3-b4698737b55f\") " pod="openshift-monitoring/node-exporter-h45hc" Jul 06 20:07:22.110879 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.110750 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/40f67c9c-cf00-4f97-a5b3-b4698737b55f-node-exporter-textfile\") pod \"node-exporter-h45hc\" (UID: \"40f67c9c-cf00-4f97-a5b3-b4698737b55f\") " pod="openshift-monitoring/node-exporter-h45hc" Jul 06 20:07:22.110879 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.110771 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40f67c9c-cf00-4f97-a5b3-b4698737b55f-metrics-client-ca\") pod \"node-exporter-h45hc\" (UID: \"40f67c9c-cf00-4f97-a5b3-b4698737b55f\") " pod="openshift-monitoring/node-exporter-h45hc" Jul 06 20:07:22.110879 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.110813 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzrhq\" (UniqueName: \"kubernetes.io/projected/40f67c9c-cf00-4f97-a5b3-b4698737b55f-kube-api-access-jzrhq\") pod \"node-exporter-h45hc\" (UID: \"40f67c9c-cf00-4f97-a5b3-b4698737b55f\") " pod="openshift-monitoring/node-exporter-h45hc" Jul 06 20:07:22.111009 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.110882 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/40f67c9c-cf00-4f97-a5b3-b4698737b55f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h45hc\" (UID: \"40f67c9c-cf00-4f97-a5b3-b4698737b55f\") " pod="openshift-monitoring/node-exporter-h45hc" Jul 06 20:07:22.111009 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.110905 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/40f67c9c-cf00-4f97-a5b3-b4698737b55f-root\") pod \"node-exporter-h45hc\" (UID: \"40f67c9c-cf00-4f97-a5b3-b4698737b55f\") " pod="openshift-monitoring/node-exporter-h45hc" Jul 06 20:07:22.111009 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.110918 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40f67c9c-cf00-4f97-a5b3-b4698737b55f-sys\") pod \"node-exporter-h45hc\" (UID: \"40f67c9c-cf00-4f97-a5b3-b4698737b55f\") " pod="openshift-monitoring/node-exporter-h45hc" Jul 06 20:07:22.111009 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.110973 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/40f67c9c-cf00-4f97-a5b3-b4698737b55f-node-exporter-accelerators-collector-config\") pod \"node-exporter-h45hc\" (UID: \"40f67c9c-cf00-4f97-a5b3-b4698737b55f\") " pod="openshift-monitoring/node-exporter-h45hc" Jul 06 20:07:22.111009 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.111008 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/40f67c9c-cf00-4f97-a5b3-b4698737b55f-node-exporter-tls\") pod \"node-exporter-h45hc\" (UID: \"40f67c9c-cf00-4f97-a5b3-b4698737b55f\") " pod="openshift-monitoring/node-exporter-h45hc" Jul 06 20:07:22.172071 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.172037 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:22.212025 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.211951 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40f67c9c-cf00-4f97-a5b3-b4698737b55f-sys\") pod \"node-exporter-h45hc\" (UID: \"40f67c9c-cf00-4f97-a5b3-b4698737b55f\") " pod="openshift-monitoring/node-exporter-h45hc" Jul 06 20:07:22.212025 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.211987 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/40f67c9c-cf00-4f97-a5b3-b4698737b55f-node-exporter-accelerators-collector-config\") pod \"node-exporter-h45hc\" (UID: \"40f67c9c-cf00-4f97-a5b3-b4698737b55f\") " pod="openshift-monitoring/node-exporter-h45hc" Jul 06 20:07:22.212025 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.212013 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/40f67c9c-cf00-4f97-a5b3-b4698737b55f-node-exporter-tls\") pod \"node-exporter-h45hc\" (UID: \"40f67c9c-cf00-4f97-a5b3-b4698737b55f\") " pod="openshift-monitoring/node-exporter-h45hc" Jul 06 20:07:22.212253 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.212039 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/40f67c9c-cf00-4f97-a5b3-b4698737b55f-node-exporter-wtmp\") pod \"node-exporter-h45hc\" (UID: \"40f67c9c-cf00-4f97-a5b3-b4698737b55f\") " pod="openshift-monitoring/node-exporter-h45hc" Jul 06 20:07:22.212253 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.212065 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/40f67c9c-cf00-4f97-a5b3-b4698737b55f-node-exporter-textfile\") pod \"node-exporter-h45hc\" (UID: \"40f67c9c-cf00-4f97-a5b3-b4698737b55f\") " pod="openshift-monitoring/node-exporter-h45hc" Jul 06 20:07:22.212253 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.212069 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40f67c9c-cf00-4f97-a5b3-b4698737b55f-sys\") pod \"node-exporter-h45hc\" (UID: \"40f67c9c-cf00-4f97-a5b3-b4698737b55f\") " pod="openshift-monitoring/node-exporter-h45hc" Jul 06 20:07:22.212253 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.212087 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40f67c9c-cf00-4f97-a5b3-b4698737b55f-metrics-client-ca\") pod \"node-exporter-h45hc\" (UID: \"40f67c9c-cf00-4f97-a5b3-b4698737b55f\") " pod="openshift-monitoring/node-exporter-h45hc" Jul 06 20:07:22.212253 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.212129 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzrhq\" (UniqueName: \"kubernetes.io/projected/40f67c9c-cf00-4f97-a5b3-b4698737b55f-kube-api-access-jzrhq\") pod \"node-exporter-h45hc\" (UID: \"40f67c9c-cf00-4f97-a5b3-b4698737b55f\") " pod="openshift-monitoring/node-exporter-h45hc" Jul 06 20:07:22.212253 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.212169 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/40f67c9c-cf00-4f97-a5b3-b4698737b55f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h45hc\" (UID: \"40f67c9c-cf00-4f97-a5b3-b4698737b55f\") " pod="openshift-monitoring/node-exporter-h45hc" Jul 06 20:07:22.212253 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.212189 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/40f67c9c-cf00-4f97-a5b3-b4698737b55f-root\") pod \"node-exporter-h45hc\" (UID: \"40f67c9c-cf00-4f97-a5b3-b4698737b55f\") " pod="openshift-monitoring/node-exporter-h45hc" Jul 06 20:07:22.212253 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.212191 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/40f67c9c-cf00-4f97-a5b3-b4698737b55f-node-exporter-wtmp\") pod \"node-exporter-h45hc\" (UID: \"40f67c9c-cf00-4f97-a5b3-b4698737b55f\") " pod="openshift-monitoring/node-exporter-h45hc" Jul 06 20:07:22.212253 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.212255 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/40f67c9c-cf00-4f97-a5b3-b4698737b55f-root\") pod \"node-exporter-h45hc\" (UID: \"40f67c9c-cf00-4f97-a5b3-b4698737b55f\") " pod="openshift-monitoring/node-exporter-h45hc" Jul 06 20:07:22.212675 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.212506 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/40f67c9c-cf00-4f97-a5b3-b4698737b55f-node-exporter-textfile\") pod \"node-exporter-h45hc\" (UID: \"40f67c9c-cf00-4f97-a5b3-b4698737b55f\") " pod="openshift-monitoring/node-exporter-h45hc" Jul 06 20:07:22.212675 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.212655 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40f67c9c-cf00-4f97-a5b3-b4698737b55f-metrics-client-ca\") pod \"node-exporter-h45hc\" (UID: \"40f67c9c-cf00-4f97-a5b3-b4698737b55f\") " pod="openshift-monitoring/node-exporter-h45hc" Jul 06 20:07:22.212737 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.212678 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/40f67c9c-cf00-4f97-a5b3-b4698737b55f-node-exporter-accelerators-collector-config\") pod \"node-exporter-h45hc\" (UID: \"40f67c9c-cf00-4f97-a5b3-b4698737b55f\") " pod="openshift-monitoring/node-exporter-h45hc" Jul 06 20:07:22.214752 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.214732 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/40f67c9c-cf00-4f97-a5b3-b4698737b55f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h45hc\" (UID: \"40f67c9c-cf00-4f97-a5b3-b4698737b55f\") " pod="openshift-monitoring/node-exporter-h45hc" Jul 06 20:07:22.214838 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.214755 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/40f67c9c-cf00-4f97-a5b3-b4698737b55f-node-exporter-tls\") pod \"node-exporter-h45hc\" (UID: \"40f67c9c-cf00-4f97-a5b3-b4698737b55f\") " pod="openshift-monitoring/node-exporter-h45hc" Jul 06 20:07:22.221492 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.221439 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzrhq\" (UniqueName: \"kubernetes.io/projected/40f67c9c-cf00-4f97-a5b3-b4698737b55f-kube-api-access-jzrhq\") pod \"node-exporter-h45hc\" (UID: \"40f67c9c-cf00-4f97-a5b3-b4698737b55f\") " pod="openshift-monitoring/node-exporter-h45hc" Jul 06 20:07:22.269182 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.269132 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:22.339679 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.339648 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h45hc" Jul 06 20:07:22.349391 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:07:22.349363 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40f67c9c_cf00_4f97_a5b3_b4698737b55f.slice/crio-f22db51ced006d1580ccdfc189add31560f2487c0aa1923819a12d4d04f257ce WatchSource:0}: Error finding container f22db51ced006d1580ccdfc189add31560f2487c0aa1923819a12d4d04f257ce: Status 404 returned error can't find the container with id f22db51ced006d1580ccdfc189add31560f2487c0aa1923819a12d4d04f257ce Jul 06 20:07:22.364814 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.364788 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:22.499190 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.499097 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:22.563539 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.563502 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:22.690267 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.690233 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h45hc" event={"ID":"40f67c9c-cf00-4f97-a5b3-b4698737b55f","Type":"ContainerStarted","Data":"f22db51ced006d1580ccdfc189add31560f2487c0aa1923819a12d4d04f257ce"} Jul 06 20:07:22.886511 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:22.886418 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:23.083024 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:23.082991 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:23.227024 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:23.226993 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:23.494960 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:23.494883 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:24.699089 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:24.699048 2578 generic.go:356] "Generic (PLEG): container finished" podID="40f67c9c-cf00-4f97-a5b3-b4698737b55f" containerID="5b40773304b61f9d27afb5be6072a836fc866be89b7d24d2d18df023ec35989c" exitCode=0 Jul 06 20:07:24.699089 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:24.699093 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h45hc" event={"ID":"40f67c9c-cf00-4f97-a5b3-b4698737b55f","Type":"ContainerDied","Data":"5b40773304b61f9d27afb5be6072a836fc866be89b7d24d2d18df023ec35989c"} Jul 06 20:07:25.073686 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:25.073605 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:25.148998 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:25.148957 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:25.518047 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:25.518009 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:25.704402 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:25.704369 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h45hc" event={"ID":"40f67c9c-cf00-4f97-a5b3-b4698737b55f","Type":"ContainerStarted","Data":"fd77014c6397d852e06bac0927520c47147baaf859a7c31c73ca4365c964769e"} Jul 06 20:07:25.704792 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:25.704426 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h45hc" event={"ID":"40f67c9c-cf00-4f97-a5b3-b4698737b55f","Type":"ContainerStarted","Data":"48a9c55f0e7f4e7666688a3279c242dcb6241118465d198bf4d83ee387de2753"} Jul 06 20:07:25.728236 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:25.728191 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-h45hc" podStartSLOduration=2.130328378 podStartE2EDuration="3.728176917s" totalImagesPullingTime="1.482756204s" totalInitContainerRuntime="115.092335ms" isStatefulPod=true podCreationTimestamp="2026-07-06 20:07:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:07:25.726917039 +0000 UTC m=+96.938065279" watchObservedRunningTime="2026-07-06 20:07:25.728176917 +0000 UTC m=+96.939325135" Jul 06 20:07:26.501508 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:26.501474 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:26.614310 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:26.614276 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:26.719242 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:26.719204 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:26.729401 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:26.729381 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-5czzx"] Jul 06 20:07:26.731484 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:26.731445 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5czzx" Jul 06 20:07:26.734145 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:26.734123 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-jzk89\"" Jul 06 20:07:26.734688 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:26.734672 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 06 20:07:26.746632 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:26.746612 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-5czzx"] Jul 06 20:07:26.844800 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:26.844723 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:26.849726 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:26.849687 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmbnc\" (UniqueName: \"kubernetes.io/projected/57888923-ec2b-4511-8590-bdd4fd3ed9ce-kube-api-access-hmbnc\") pod \"monitoring-plugin-d5dcff7cf-5czzx\" (UID: \"57888923-ec2b-4511-8590-bdd4fd3ed9ce\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5czzx" Jul 06 20:07:26.849877 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:26.849747 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/57888923-ec2b-4511-8590-bdd4fd3ed9ce-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-5czzx\" (UID: \"57888923-ec2b-4511-8590-bdd4fd3ed9ce\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5czzx" Jul 06 20:07:26.906236 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:26.906194 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:26.951174 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:26.951141 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmbnc\" (UniqueName: \"kubernetes.io/projected/57888923-ec2b-4511-8590-bdd4fd3ed9ce-kube-api-access-hmbnc\") pod \"monitoring-plugin-d5dcff7cf-5czzx\" (UID: \"57888923-ec2b-4511-8590-bdd4fd3ed9ce\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5czzx" Jul 06 20:07:26.951321 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:26.951188 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/57888923-ec2b-4511-8590-bdd4fd3ed9ce-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-5czzx\" (UID: \"57888923-ec2b-4511-8590-bdd4fd3ed9ce\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5czzx" Jul 06 20:07:26.953608 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:26.953589 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/57888923-ec2b-4511-8590-bdd4fd3ed9ce-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-5czzx\" (UID: \"57888923-ec2b-4511-8590-bdd4fd3ed9ce\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5czzx" Jul 06 20:07:26.961096 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:26.961077 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmbnc\" (UniqueName: \"kubernetes.io/projected/57888923-ec2b-4511-8590-bdd4fd3ed9ce-kube-api-access-hmbnc\") pod \"monitoring-plugin-d5dcff7cf-5czzx\" (UID: \"57888923-ec2b-4511-8590-bdd4fd3ed9ce\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5czzx" Jul 06 20:07:27.039790 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:27.039753 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5czzx" Jul 06 20:07:27.171101 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:27.171067 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-5czzx"] Jul 06 20:07:27.173854 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:07:27.173821 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57888923_ec2b_4511_8590_bdd4fd3ed9ce.slice/crio-acc3a00ec15f3b9be06635baf9caf8ee6985175477b2e5c34c670393bfc17f70 WatchSource:0}: Error finding container acc3a00ec15f3b9be06635baf9caf8ee6985175477b2e5c34c670393bfc17f70: Status 404 returned error can't find the container with id acc3a00ec15f3b9be06635baf9caf8ee6985175477b2e5c34c670393bfc17f70 Jul 06 20:07:27.218096 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:27.218065 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:27.328470 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:27.328422 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:27.711600 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:27.711563 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5czzx" event={"ID":"57888923-ec2b-4511-8590-bdd4fd3ed9ce","Type":"ContainerStarted","Data":"acc3a00ec15f3b9be06635baf9caf8ee6985175477b2e5c34c670393bfc17f70"} Jul 06 20:07:28.281935 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.281901 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:28.351807 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.351769 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:28.353391 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.353367 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 20:07:28.356270 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.356245 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.359030 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.359007 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 20:07:28.359161 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.359007 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 20:07:28.359161 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.359009 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 20:07:28.360016 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.359995 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 20:07:28.360122 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.360048 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 20:07:28.360311 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.360294 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 20:07:28.360393 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.360328 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7f7k9jj6t260u\"" Jul 06 20:07:28.360747 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.360731 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-w7glg\"" Jul 06 20:07:28.360859 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.360748 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 20:07:28.360928 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.360751 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 06 20:07:28.362774 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.362750 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 20:07:28.363019 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.362998 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 20:07:28.363270 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.363254 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 20:07:28.363821 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.363801 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 20:07:28.364032 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.364014 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 20:07:28.364175 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.364156 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 20:07:28.364278 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.364265 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 20:07:28.392936 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.392900 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 20:07:28.465568 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.465535 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a5f1feda-1270-4213-b7c2-568b7f28c74c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.465725 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.465580 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a5f1feda-1270-4213-b7c2-568b7f28c74c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.465725 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.465614 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5f1feda-1270-4213-b7c2-568b7f28c74c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.465725 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.465640 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5f1feda-1270-4213-b7c2-568b7f28c74c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.465725 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.465664 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5f1feda-1270-4213-b7c2-568b7f28c74c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.465725 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.465683 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a5f1feda-1270-4213-b7c2-568b7f28c74c-config\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.465725 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.465701 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5f1feda-1270-4213-b7c2-568b7f28c74c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.466015 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.465744 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a5f1feda-1270-4213-b7c2-568b7f28c74c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.466015 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.465787 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a5f1feda-1270-4213-b7c2-568b7f28c74c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.466015 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.465814 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a5f1feda-1270-4213-b7c2-568b7f28c74c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.466015 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.465850 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a5f1feda-1270-4213-b7c2-568b7f28c74c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.466015 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.465882 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvng4\" (UniqueName: \"kubernetes.io/projected/a5f1feda-1270-4213-b7c2-568b7f28c74c-kube-api-access-nvng4\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.466015 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.465908 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a5f1feda-1270-4213-b7c2-568b7f28c74c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.466015 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.465943 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a5f1feda-1270-4213-b7c2-568b7f28c74c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.466015 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.465966 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a5f1feda-1270-4213-b7c2-568b7f28c74c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.466015 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.465994 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5f1feda-1270-4213-b7c2-568b7f28c74c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.466372 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.466030 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a5f1feda-1270-4213-b7c2-568b7f28c74c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.466372 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.466054 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a5f1feda-1270-4213-b7c2-568b7f28c74c-config-out\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.466372 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.466071 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5f1feda-1270-4213-b7c2-568b7f28c74c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.466372 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.466106 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a5f1feda-1270-4213-b7c2-568b7f28c74c-web-config\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.567367 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.567282 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a5f1feda-1270-4213-b7c2-568b7f28c74c-web-config\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.567367 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.567320 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a5f1feda-1270-4213-b7c2-568b7f28c74c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.567367 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.567348 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a5f1feda-1270-4213-b7c2-568b7f28c74c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.567367 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.567365 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5f1feda-1270-4213-b7c2-568b7f28c74c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.567755 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.567390 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5f1feda-1270-4213-b7c2-568b7f28c74c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.567755 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.567414 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5f1feda-1270-4213-b7c2-568b7f28c74c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.567755 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.567434 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a5f1feda-1270-4213-b7c2-568b7f28c74c-config\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.567755 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.567477 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5f1feda-1270-4213-b7c2-568b7f28c74c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.567755 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.567506 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a5f1feda-1270-4213-b7c2-568b7f28c74c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.567755 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.567537 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a5f1feda-1270-4213-b7c2-568b7f28c74c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.567755 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.567561 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a5f1feda-1270-4213-b7c2-568b7f28c74c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.567755 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.567586 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a5f1feda-1270-4213-b7c2-568b7f28c74c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.567755 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.567612 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvng4\" (UniqueName: \"kubernetes.io/projected/a5f1feda-1270-4213-b7c2-568b7f28c74c-kube-api-access-nvng4\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.567755 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.567636 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a5f1feda-1270-4213-b7c2-568b7f28c74c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.567755 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.567665 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a5f1feda-1270-4213-b7c2-568b7f28c74c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.567755 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.567689 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a5f1feda-1270-4213-b7c2-568b7f28c74c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.567755 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.567723 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5f1feda-1270-4213-b7c2-568b7f28c74c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.568298 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.567768 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a5f1feda-1270-4213-b7c2-568b7f28c74c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.568298 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.567798 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a5f1feda-1270-4213-b7c2-568b7f28c74c-config-out\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.568298 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.567823 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5f1feda-1270-4213-b7c2-568b7f28c74c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.568298 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.568041 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a5f1feda-1270-4213-b7c2-568b7f28c74c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.568298 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.568170 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5f1feda-1270-4213-b7c2-568b7f28c74c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.568551 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.568438 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a5f1feda-1270-4213-b7c2-568b7f28c74c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.568551 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.568510 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5f1feda-1270-4213-b7c2-568b7f28c74c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.569093 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.569069 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5f1feda-1270-4213-b7c2-568b7f28c74c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.570191 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.569548 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5f1feda-1270-4213-b7c2-568b7f28c74c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.571053 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.571031 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a5f1feda-1270-4213-b7c2-568b7f28c74c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.571759 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.571709 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a5f1feda-1270-4213-b7c2-568b7f28c74c-web-config\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.572087 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.571974 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a5f1feda-1270-4213-b7c2-568b7f28c74c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.573224 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.572424 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a5f1feda-1270-4213-b7c2-568b7f28c74c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.573224 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.572753 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5f1feda-1270-4213-b7c2-568b7f28c74c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.573224 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.572765 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5f1feda-1270-4213-b7c2-568b7f28c74c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.573224 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.573065 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a5f1feda-1270-4213-b7c2-568b7f28c74c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.573224 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.573193 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a5f1feda-1270-4213-b7c2-568b7f28c74c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.574106 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.573995 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a5f1feda-1270-4213-b7c2-568b7f28c74c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.574571 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.574538 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a5f1feda-1270-4213-b7c2-568b7f28c74c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.574670 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.574600 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a5f1feda-1270-4213-b7c2-568b7f28c74c-config\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.574816 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.574784 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a5f1feda-1270-4213-b7c2-568b7f28c74c-config-out\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.575908 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.575885 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a5f1feda-1270-4213-b7c2-568b7f28c74c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.592240 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.592218 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvng4\" (UniqueName: \"kubernetes.io/projected/a5f1feda-1270-4213-b7c2-568b7f28c74c-kube-api-access-nvng4\") pod \"prometheus-k8s-0\" (UID: \"a5f1feda-1270-4213-b7c2-568b7f28c74c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.669660 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.669623 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:28.718814 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.718718 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5czzx" event={"ID":"57888923-ec2b-4511-8590-bdd4fd3ed9ce","Type":"ContainerStarted","Data":"4233161b0cc512f68f1c1ab988ddfb01dd15d30b616f9b8e5b9c95d6267733aa"} Jul 06 20:07:28.719077 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.719055 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5czzx" Jul 06 20:07:28.723693 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.723672 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5czzx" Jul 06 20:07:28.749118 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.749059 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5czzx" podStartSLOduration=1.2924808749999999 podStartE2EDuration="2.749040643s" totalImagesPullingTime="1.456559768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:07:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:07:28.748386488 +0000 UTC m=+99.959534707" watchObservedRunningTime="2026-07-06 20:07:28.749040643 +0000 UTC m=+99.960188861" Jul 06 20:07:28.817550 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:28.817504 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 20:07:28.819432 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:07:28.819409 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f1feda_1270_4213_b7c2_568b7f28c74c.slice/crio-7e2e80505d25d4996059ed10cc72624921125bca62d853c142996cea75918389 WatchSource:0}: Error finding container 7e2e80505d25d4996059ed10cc72624921125bca62d853c142996cea75918389: Status 404 returned error can't find the container with id 7e2e80505d25d4996059ed10cc72624921125bca62d853c142996cea75918389 Jul 06 20:07:29.723914 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:29.723875 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5f1feda-1270-4213-b7c2-568b7f28c74c","Type":"ContainerStarted","Data":"7e2e80505d25d4996059ed10cc72624921125bca62d853c142996cea75918389"} Jul 06 20:07:30.206174 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:30.206140 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:30.302889 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:30.302850 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:30.728577 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:30.728536 2578 generic.go:356] "Generic (PLEG): container finished" podID="a5f1feda-1270-4213-b7c2-568b7f28c74c" containerID="ab48cb9a3dd7d0c6229fb35a57fce1faba09e1eabc2d9917b839d047f7c4e45d" exitCode=0 Jul 06 20:07:30.728958 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:30.728624 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5f1feda-1270-4213-b7c2-568b7f28c74c","Type":"ContainerDied","Data":"ab48cb9a3dd7d0c6229fb35a57fce1faba09e1eabc2d9917b839d047f7c4e45d"} Jul 06 20:07:35.638647 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:35.638616 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mbz65" Jul 06 20:07:35.748358 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:35.748325 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5f1feda-1270-4213-b7c2-568b7f28c74c","Type":"ContainerStarted","Data":"87dcc20e216304a905a4fd2b047f25d5048acf7c888062114308723295ac3c0c"} Jul 06 20:07:35.748557 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:35.748400 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5f1feda-1270-4213-b7c2-568b7f28c74c","Type":"ContainerStarted","Data":"47eb57b03aed94bf3b1db23aeac4a09a5221c1ec52a492fd4fae464ec6860313"} Jul 06 20:07:37.757385 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:37.757359 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5f1feda-1270-4213-b7c2-568b7f28c74c","Type":"ContainerStarted","Data":"53c15b5081aa45af9cdd9b76a9828b61a9503a83720734c29f1be82c919dc152"} Jul 06 20:07:38.764018 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:38.763983 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5f1feda-1270-4213-b7c2-568b7f28c74c","Type":"ContainerStarted","Data":"2d6d66031b835fdcdf89b72243fc3aa00d24ebe1f5e11058a6f902fd817bf57f"} Jul 06 20:07:38.764527 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:38.764044 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5f1feda-1270-4213-b7c2-568b7f28c74c","Type":"ContainerStarted","Data":"11f8888223d47b5bf211208722fd1885654c0e14c1ac99b579d6bf2fcf8ac0f4"} Jul 06 20:07:38.764527 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:38.764070 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5f1feda-1270-4213-b7c2-568b7f28c74c","Type":"ContainerStarted","Data":"8f42cadc070ed162d0eff6ad24d27e9c22f1fde8b6218bb9ba83868f6f7e4431"} Jul 06 20:07:38.806068 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:38.806003 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.036775745 podStartE2EDuration="10.805985337s" totalImagesPullingTime="7.745807741s" totalInitContainerRuntime="23.401851ms" isStatefulPod=false podCreationTimestamp="2026-07-06 20:07:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:07:38.803638874 +0000 UTC m=+110.014787105" watchObservedRunningTime="2026-07-06 20:07:38.805985337 +0000 UTC m=+110.017133554" Jul 06 20:07:43.670172 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:43.670131 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:07:45.887570 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:45.887536 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:45.986327 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:45.986292 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:46.048749 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:46.048717 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:53.688367 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:53.688331 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:53.790275 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:53.790240 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:54.358146 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:54.358114 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:54.447632 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:07:54.447595 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:01.795637 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:08:01.795594 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:01.949011 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:08:01.948975 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:02.046262 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:08:02.046170 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:28.670270 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:08:28.670219 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:08:28.689697 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:08:28.689673 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:08:28.950345 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:08:28.950273 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 20:08:38.234820 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:08:38.234751 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:38.306890 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:08:38.306853 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:49.066419 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:08:49.066339 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:49.163257 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:08:49.163218 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:49.225140 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:08:49.225109 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:49.770976 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:08:49.770936 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:49.828009 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:08:49.827975 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:50.955427 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:08:50.955391 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:51.016166 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:08:51.016135 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:54.347269 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:08:54.347232 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:54.451424 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:08:54.451388 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:54.557898 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:08:54.557867 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:59.709380 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:08:59.709345 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:59.811072 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:08:59.811034 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:59.873054 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:08:59.873013 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:14.545990 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:09:14.545942 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:14.650883 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:09:14.650848 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:27.616243 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:09:27.616205 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:27.675780 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:09:27.675744 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:34.428968 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:09:34.428932 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:34.498526 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:09:34.498500 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:40.945669 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:09:40.945628 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:42.320063 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:09:42.320031 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:44.570673 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:09:44.570632 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:44.670126 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:09:44.670086 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:46.016096 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:09:46.016055 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:46.491393 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:09:46.491357 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:46.620066 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:09:46.620035 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:46.748016 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:09:46.747926 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:48.661244 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:09:48.661201 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:49.072121 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:09:49.072036 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:51.565161 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:09:51.565124 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:51.631525 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:09:51.631484 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:54.413997 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:09:54.413955 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:57.304083 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:09:57.304048 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:00.446100 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:00.446064 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:02.771813 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:02.771784 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:02.872649 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:02.872616 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:07.570866 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:07.570827 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:07.652154 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:07.652117 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:09.963991 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:09.963914 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:22.237846 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:22.237799 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:22.313442 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:22.313409 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:26.574845 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:26.574810 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:26.668943 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:26.668903 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:27.426165 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:27.426130 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:27.679075 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:27.678970 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:27.794059 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:27.794024 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:27.839663 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:27.839635 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68cbf59454-4sslp"] Jul 06 20:10:27.841854 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:27.841838 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68cbf59454-4sslp" Jul 06 20:10:27.844420 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:27.844397 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-crnc6\"" Jul 06 20:10:27.844705 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:27.844684 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 20:10:27.844773 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:27.844734 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 20:10:27.844836 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:27.844818 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 20:10:27.844914 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:27.844902 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 20:10:27.869715 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:27.869686 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68cbf59454-4sslp"] Jul 06 20:10:27.892595 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:27.892562 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/630178c4-0b36-42bb-bbda-a53f32ec4fe0-webhook-cert\") pod \"opendatahub-operator-controller-manager-68cbf59454-4sslp\" (UID: \"630178c4-0b36-42bb-bbda-a53f32ec4fe0\") " pod="opendatahub/opendatahub-operator-controller-manager-68cbf59454-4sslp" Jul 06 20:10:27.892595 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:27.892599 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/630178c4-0b36-42bb-bbda-a53f32ec4fe0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68cbf59454-4sslp\" (UID: \"630178c4-0b36-42bb-bbda-a53f32ec4fe0\") " pod="opendatahub/opendatahub-operator-controller-manager-68cbf59454-4sslp" Jul 06 20:10:27.892778 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:27.892637 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/630178c4-0b36-42bb-bbda-a53f32ec4fe0-tmp\") pod \"opendatahub-operator-controller-manager-68cbf59454-4sslp\" (UID: \"630178c4-0b36-42bb-bbda-a53f32ec4fe0\") " pod="opendatahub/opendatahub-operator-controller-manager-68cbf59454-4sslp" Jul 06 20:10:27.892778 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:27.892655 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/630178c4-0b36-42bb-bbda-a53f32ec4fe0-manifests\") pod \"opendatahub-operator-controller-manager-68cbf59454-4sslp\" (UID: \"630178c4-0b36-42bb-bbda-a53f32ec4fe0\") " pod="opendatahub/opendatahub-operator-controller-manager-68cbf59454-4sslp" Jul 06 20:10:27.892778 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:27.892688 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8hfr\" (UniqueName: \"kubernetes.io/projected/630178c4-0b36-42bb-bbda-a53f32ec4fe0-kube-api-access-n8hfr\") pod \"opendatahub-operator-controller-manager-68cbf59454-4sslp\" (UID: \"630178c4-0b36-42bb-bbda-a53f32ec4fe0\") " pod="opendatahub/opendatahub-operator-controller-manager-68cbf59454-4sslp" Jul 06 20:10:27.993129 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:27.993035 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8hfr\" (UniqueName: \"kubernetes.io/projected/630178c4-0b36-42bb-bbda-a53f32ec4fe0-kube-api-access-n8hfr\") pod \"opendatahub-operator-controller-manager-68cbf59454-4sslp\" (UID: \"630178c4-0b36-42bb-bbda-a53f32ec4fe0\") " pod="opendatahub/opendatahub-operator-controller-manager-68cbf59454-4sslp" Jul 06 20:10:27.993129 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:27.993084 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/630178c4-0b36-42bb-bbda-a53f32ec4fe0-webhook-cert\") pod \"opendatahub-operator-controller-manager-68cbf59454-4sslp\" (UID: \"630178c4-0b36-42bb-bbda-a53f32ec4fe0\") " pod="opendatahub/opendatahub-operator-controller-manager-68cbf59454-4sslp" Jul 06 20:10:27.993129 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:27.993107 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/630178c4-0b36-42bb-bbda-a53f32ec4fe0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68cbf59454-4sslp\" (UID: \"630178c4-0b36-42bb-bbda-a53f32ec4fe0\") " pod="opendatahub/opendatahub-operator-controller-manager-68cbf59454-4sslp" Jul 06 20:10:27.993397 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:27.993140 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/630178c4-0b36-42bb-bbda-a53f32ec4fe0-tmp\") pod \"opendatahub-operator-controller-manager-68cbf59454-4sslp\" (UID: \"630178c4-0b36-42bb-bbda-a53f32ec4fe0\") " pod="opendatahub/opendatahub-operator-controller-manager-68cbf59454-4sslp" Jul 06 20:10:27.993397 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:27.993169 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/630178c4-0b36-42bb-bbda-a53f32ec4fe0-manifests\") pod \"opendatahub-operator-controller-manager-68cbf59454-4sslp\" (UID: \"630178c4-0b36-42bb-bbda-a53f32ec4fe0\") " pod="opendatahub/opendatahub-operator-controller-manager-68cbf59454-4sslp" Jul 06 20:10:27.993679 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:27.993659 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/630178c4-0b36-42bb-bbda-a53f32ec4fe0-manifests\") pod \"opendatahub-operator-controller-manager-68cbf59454-4sslp\" (UID: \"630178c4-0b36-42bb-bbda-a53f32ec4fe0\") " pod="opendatahub/opendatahub-operator-controller-manager-68cbf59454-4sslp" Jul 06 20:10:27.995645 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:27.995613 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/630178c4-0b36-42bb-bbda-a53f32ec4fe0-tmp\") pod \"opendatahub-operator-controller-manager-68cbf59454-4sslp\" (UID: \"630178c4-0b36-42bb-bbda-a53f32ec4fe0\") " pod="opendatahub/opendatahub-operator-controller-manager-68cbf59454-4sslp" Jul 06 20:10:27.995745 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:27.995669 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/630178c4-0b36-42bb-bbda-a53f32ec4fe0-webhook-cert\") pod \"opendatahub-operator-controller-manager-68cbf59454-4sslp\" (UID: \"630178c4-0b36-42bb-bbda-a53f32ec4fe0\") " pod="opendatahub/opendatahub-operator-controller-manager-68cbf59454-4sslp" Jul 06 20:10:27.995792 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:27.995770 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/630178c4-0b36-42bb-bbda-a53f32ec4fe0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68cbf59454-4sslp\" (UID: \"630178c4-0b36-42bb-bbda-a53f32ec4fe0\") " pod="opendatahub/opendatahub-operator-controller-manager-68cbf59454-4sslp" Jul 06 20:10:28.000500 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:28.000472 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:28.009971 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:28.009952 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8hfr\" (UniqueName: \"kubernetes.io/projected/630178c4-0b36-42bb-bbda-a53f32ec4fe0-kube-api-access-n8hfr\") pod \"opendatahub-operator-controller-manager-68cbf59454-4sslp\" (UID: \"630178c4-0b36-42bb-bbda-a53f32ec4fe0\") " pod="opendatahub/opendatahub-operator-controller-manager-68cbf59454-4sslp" Jul 06 20:10:28.096150 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:28.096112 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:28.151352 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:28.151321 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68cbf59454-4sslp" Jul 06 20:10:28.196381 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:28.196341 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:28.296742 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:28.296650 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:28.303947 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:28.303920 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68cbf59454-4sslp"] Jul 06 20:10:28.307597 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:10:28.307574 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod630178c4_0b36_42bb_bbda_a53f32ec4fe0.slice/crio-1cd5b7ec000feb4654568fabb2571593d598f505ce501ed4aecf7ee527bef9fe WatchSource:0}: Error finding container 1cd5b7ec000feb4654568fabb2571593d598f505ce501ed4aecf7ee527bef9fe: Status 404 returned error can't find the container with id 1cd5b7ec000feb4654568fabb2571593d598f505ce501ed4aecf7ee527bef9fe Jul 06 20:10:28.336372 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:28.336340 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68cbf59454-4sslp" event={"ID":"630178c4-0b36-42bb-bbda-a53f32ec4fe0","Type":"ContainerStarted","Data":"1cd5b7ec000feb4654568fabb2571593d598f505ce501ed4aecf7ee527bef9fe"} Jul 06 20:10:28.392692 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:28.392660 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:28.501927 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:28.501897 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:28.605100 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:28.605006 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:28.701446 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:28.701415 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:29.757656 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:29.757622 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:29.849051 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:29.848961 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:32.356394 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:32.356357 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68cbf59454-4sslp" event={"ID":"630178c4-0b36-42bb-bbda-a53f32ec4fe0","Type":"ContainerStarted","Data":"c411ea7d4293d7870f4ea4941aea0beedf8e48c22e031268e7afcc3e4a93c890"} Jul 06 20:10:33.360886 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:33.360849 2578 generic.go:356] "Generic (PLEG): container finished" podID="630178c4-0b36-42bb-bbda-a53f32ec4fe0" containerID="c411ea7d4293d7870f4ea4941aea0beedf8e48c22e031268e7afcc3e4a93c890" exitCode=0 Jul 06 20:10:33.361349 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:33.360928 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68cbf59454-4sslp" event={"ID":"630178c4-0b36-42bb-bbda-a53f32ec4fe0","Type":"ContainerDied","Data":"c411ea7d4293d7870f4ea4941aea0beedf8e48c22e031268e7afcc3e4a93c890"} Jul 06 20:10:34.366776 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:34.366744 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68cbf59454-4sslp" event={"ID":"630178c4-0b36-42bb-bbda-a53f32ec4fe0","Type":"ContainerStarted","Data":"dff2eb0665eded747fab21abdf698bb408a9366aabad308c266dad8f7fc5dd9e"} Jul 06 20:10:34.367156 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:34.366875 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-68cbf59454-4sslp" Jul 06 20:10:34.397515 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:34.397448 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-68cbf59454-4sslp" podStartSLOduration=2.690464332 podStartE2EDuration="7.397429637s" totalImagesPullingTime="3.62871612s" totalInitContainerRuntime="1.078249185s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:10:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:10:34.396282882 +0000 UTC m=+285.607431111" watchObservedRunningTime="2026-07-06 20:10:34.397429637 +0000 UTC m=+285.608577856" Jul 06 20:10:37.163930 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:37.163892 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:37.281517 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:37.281484 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:39.336099 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:39.336056 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:39.428200 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:39.428162 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:45.372155 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:45.372122 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-68cbf59454-4sslp" Jul 06 20:10:46.947175 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:46.947134 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:47.150095 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:47.150065 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:47.616964 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:47.616928 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:48.100807 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:48.100774 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:48.274099 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:48.274063 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:48.406998 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:48.406971 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:48.501507 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:48.501474 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:48.597329 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:48.597287 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:48.703591 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:48.703512 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:49.170357 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:49.170328 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dnfgp_0f4c3a8a-30c4-428e-88a6-dcf086ef8615/ovn-acl-logging/0.log" Jul 06 20:10:49.174310 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:49.174285 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dnfgp_0f4c3a8a-30c4-428e-88a6-dcf086ef8615/ovn-acl-logging/0.log" Jul 06 20:10:49.175859 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:49.175838 2578 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 20:10:49.206562 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:49.206527 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:49.291427 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:49.291275 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:49.464922 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:49.464837 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:51.019638 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:51.019595 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:54.466519 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:54.466484 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:54.528585 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:54.528546 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:55.379223 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:55.379190 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:55.631420 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:55.631328 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:55.729856 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:55.729824 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:57.593024 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:57.592993 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-79c95fdbc9-rhh4g"] Jul 06 20:10:57.595209 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:57.595194 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-79c95fdbc9-rhh4g" Jul 06 20:10:57.602350 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:57.602330 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 06 20:10:57.603933 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:57.603913 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 06 20:10:57.604040 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:57.604022 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 20:10:57.606607 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:57.606590 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 20:10:57.607875 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:57.607860 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-b4fsp\"" Jul 06 20:10:57.607875 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:57.607870 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 06 20:10:57.624852 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:57.624828 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-79c95fdbc9-rhh4g"] Jul 06 20:10:57.718337 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:57.718296 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0def7154-736a-4490-acf2-93ccb9ae1a35-manager-config\") pod \"lws-controller-manager-79c95fdbc9-rhh4g\" (UID: \"0def7154-736a-4490-acf2-93ccb9ae1a35\") " pod="openshift-lws-operator/lws-controller-manager-79c95fdbc9-rhh4g" Jul 06 20:10:57.718541 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:57.718352 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq7r6\" (UniqueName: \"kubernetes.io/projected/0def7154-736a-4490-acf2-93ccb9ae1a35-kube-api-access-mq7r6\") pod \"lws-controller-manager-79c95fdbc9-rhh4g\" (UID: \"0def7154-736a-4490-acf2-93ccb9ae1a35\") " pod="openshift-lws-operator/lws-controller-manager-79c95fdbc9-rhh4g" Jul 06 20:10:57.718541 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:57.718409 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0def7154-736a-4490-acf2-93ccb9ae1a35-cert\") pod \"lws-controller-manager-79c95fdbc9-rhh4g\" (UID: \"0def7154-736a-4490-acf2-93ccb9ae1a35\") " pod="openshift-lws-operator/lws-controller-manager-79c95fdbc9-rhh4g" Jul 06 20:10:57.718541 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:57.718477 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0def7154-736a-4490-acf2-93ccb9ae1a35-metrics-cert\") pod \"lws-controller-manager-79c95fdbc9-rhh4g\" (UID: \"0def7154-736a-4490-acf2-93ccb9ae1a35\") " pod="openshift-lws-operator/lws-controller-manager-79c95fdbc9-rhh4g" Jul 06 20:10:57.819088 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:57.819037 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0def7154-736a-4490-acf2-93ccb9ae1a35-cert\") pod \"lws-controller-manager-79c95fdbc9-rhh4g\" (UID: \"0def7154-736a-4490-acf2-93ccb9ae1a35\") " pod="openshift-lws-operator/lws-controller-manager-79c95fdbc9-rhh4g" Jul 06 20:10:57.819088 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:57.819099 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0def7154-736a-4490-acf2-93ccb9ae1a35-metrics-cert\") pod \"lws-controller-manager-79c95fdbc9-rhh4g\" (UID: \"0def7154-736a-4490-acf2-93ccb9ae1a35\") " pod="openshift-lws-operator/lws-controller-manager-79c95fdbc9-rhh4g" Jul 06 20:10:57.819302 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:57.819209 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0def7154-736a-4490-acf2-93ccb9ae1a35-manager-config\") pod \"lws-controller-manager-79c95fdbc9-rhh4g\" (UID: \"0def7154-736a-4490-acf2-93ccb9ae1a35\") " pod="openshift-lws-operator/lws-controller-manager-79c95fdbc9-rhh4g" Jul 06 20:10:57.819302 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:57.819271 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq7r6\" (UniqueName: \"kubernetes.io/projected/0def7154-736a-4490-acf2-93ccb9ae1a35-kube-api-access-mq7r6\") pod \"lws-controller-manager-79c95fdbc9-rhh4g\" (UID: \"0def7154-736a-4490-acf2-93ccb9ae1a35\") " pod="openshift-lws-operator/lws-controller-manager-79c95fdbc9-rhh4g" Jul 06 20:10:57.819844 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:57.819805 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0def7154-736a-4490-acf2-93ccb9ae1a35-manager-config\") pod \"lws-controller-manager-79c95fdbc9-rhh4g\" (UID: \"0def7154-736a-4490-acf2-93ccb9ae1a35\") " pod="openshift-lws-operator/lws-controller-manager-79c95fdbc9-rhh4g" Jul 06 20:10:57.821646 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:57.821623 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0def7154-736a-4490-acf2-93ccb9ae1a35-cert\") pod \"lws-controller-manager-79c95fdbc9-rhh4g\" (UID: \"0def7154-736a-4490-acf2-93ccb9ae1a35\") " pod="openshift-lws-operator/lws-controller-manager-79c95fdbc9-rhh4g" Jul 06 20:10:57.821747 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:57.821661 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0def7154-736a-4490-acf2-93ccb9ae1a35-metrics-cert\") pod \"lws-controller-manager-79c95fdbc9-rhh4g\" (UID: \"0def7154-736a-4490-acf2-93ccb9ae1a35\") " pod="openshift-lws-operator/lws-controller-manager-79c95fdbc9-rhh4g" Jul 06 20:10:57.834594 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:57.834562 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq7r6\" (UniqueName: \"kubernetes.io/projected/0def7154-736a-4490-acf2-93ccb9ae1a35-kube-api-access-mq7r6\") pod \"lws-controller-manager-79c95fdbc9-rhh4g\" (UID: \"0def7154-736a-4490-acf2-93ccb9ae1a35\") " pod="openshift-lws-operator/lws-controller-manager-79c95fdbc9-rhh4g" Jul 06 20:10:57.903571 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:57.903539 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-79c95fdbc9-rhh4g" Jul 06 20:10:57.979134 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:57.979098 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vblkn"] Jul 06 20:10:57.981741 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:57.981725 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vblkn" Jul 06 20:10:57.984736 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:57.984697 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 06 20:10:57.984736 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:57.984720 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 06 20:10:57.984927 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:57.984776 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 20:10:57.984927 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:57.984824 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 20:10:57.984927 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:57.984843 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 06 20:10:57.985092 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:57.985015 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 20:10:57.985249 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:57.985232 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-fbzlz\"" Jul 06 20:10:57.998594 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:57.998566 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vblkn"] Jul 06 20:10:58.020754 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:58.020723 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a3be73c6-e989-4ccc-aacf-a9ef3d8f1378-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-vblkn\" (UID: \"a3be73c6-e989-4ccc-aacf-a9ef3d8f1378\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vblkn" Jul 06 20:10:58.020907 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:58.020769 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3be73c6-e989-4ccc-aacf-a9ef3d8f1378-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-vblkn\" (UID: \"a3be73c6-e989-4ccc-aacf-a9ef3d8f1378\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vblkn" Jul 06 20:10:58.020907 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:58.020845 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2nfc\" (UniqueName: \"kubernetes.io/projected/a3be73c6-e989-4ccc-aacf-a9ef3d8f1378-kube-api-access-b2nfc\") pod \"istiod-openshift-gateway-94bb8fbfd-vblkn\" (UID: \"a3be73c6-e989-4ccc-aacf-a9ef3d8f1378\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vblkn" Jul 06 20:10:58.020907 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:58.020874 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a3be73c6-e989-4ccc-aacf-a9ef3d8f1378-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-vblkn\" (UID: \"a3be73c6-e989-4ccc-aacf-a9ef3d8f1378\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vblkn" Jul 06 20:10:58.021027 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:58.020912 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a3be73c6-e989-4ccc-aacf-a9ef3d8f1378-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-vblkn\" (UID: \"a3be73c6-e989-4ccc-aacf-a9ef3d8f1378\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vblkn" Jul 06 20:10:58.021027 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:58.020933 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3be73c6-e989-4ccc-aacf-a9ef3d8f1378-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-vblkn\" (UID: \"a3be73c6-e989-4ccc-aacf-a9ef3d8f1378\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vblkn" Jul 06 20:10:58.021027 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:58.020949 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a3be73c6-e989-4ccc-aacf-a9ef3d8f1378-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-vblkn\" (UID: \"a3be73c6-e989-4ccc-aacf-a9ef3d8f1378\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vblkn" Jul 06 20:10:58.043448 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:58.043423 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-79c95fdbc9-rhh4g"] Jul 06 20:10:58.045818 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:10:58.045792 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0def7154_736a_4490_acf2_93ccb9ae1a35.slice/crio-95b4bdf5d298b778e56204470635b892594b2c047bd4f5dc6a039acbea6110c8 WatchSource:0}: Error finding container 95b4bdf5d298b778e56204470635b892594b2c047bd4f5dc6a039acbea6110c8: Status 404 returned error can't find the container with id 95b4bdf5d298b778e56204470635b892594b2c047bd4f5dc6a039acbea6110c8 Jul 06 20:10:58.047679 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:58.047658 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 20:10:58.121522 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:58.121485 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a3be73c6-e989-4ccc-aacf-a9ef3d8f1378-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-vblkn\" (UID: \"a3be73c6-e989-4ccc-aacf-a9ef3d8f1378\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vblkn" Jul 06 20:10:58.121732 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:58.121540 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3be73c6-e989-4ccc-aacf-a9ef3d8f1378-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-vblkn\" (UID: \"a3be73c6-e989-4ccc-aacf-a9ef3d8f1378\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vblkn" Jul 06 20:10:58.121732 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:58.121577 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2nfc\" (UniqueName: \"kubernetes.io/projected/a3be73c6-e989-4ccc-aacf-a9ef3d8f1378-kube-api-access-b2nfc\") pod \"istiod-openshift-gateway-94bb8fbfd-vblkn\" (UID: \"a3be73c6-e989-4ccc-aacf-a9ef3d8f1378\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vblkn" Jul 06 20:10:58.121732 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:58.121601 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a3be73c6-e989-4ccc-aacf-a9ef3d8f1378-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-vblkn\" (UID: \"a3be73c6-e989-4ccc-aacf-a9ef3d8f1378\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vblkn" Jul 06 20:10:58.121732 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:58.121643 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a3be73c6-e989-4ccc-aacf-a9ef3d8f1378-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-vblkn\" (UID: \"a3be73c6-e989-4ccc-aacf-a9ef3d8f1378\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vblkn" Jul 06 20:10:58.121949 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:58.121776 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3be73c6-e989-4ccc-aacf-a9ef3d8f1378-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-vblkn\" (UID: \"a3be73c6-e989-4ccc-aacf-a9ef3d8f1378\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vblkn" Jul 06 20:10:58.121949 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:58.121820 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a3be73c6-e989-4ccc-aacf-a9ef3d8f1378-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-vblkn\" (UID: \"a3be73c6-e989-4ccc-aacf-a9ef3d8f1378\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vblkn" Jul 06 20:10:58.122415 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:58.122368 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a3be73c6-e989-4ccc-aacf-a9ef3d8f1378-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-vblkn\" (UID: \"a3be73c6-e989-4ccc-aacf-a9ef3d8f1378\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vblkn" Jul 06 20:10:58.124044 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:58.124018 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a3be73c6-e989-4ccc-aacf-a9ef3d8f1378-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-vblkn\" (UID: \"a3be73c6-e989-4ccc-aacf-a9ef3d8f1378\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vblkn" Jul 06 20:10:58.124137 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:58.124072 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a3be73c6-e989-4ccc-aacf-a9ef3d8f1378-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-vblkn\" (UID: \"a3be73c6-e989-4ccc-aacf-a9ef3d8f1378\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vblkn" Jul 06 20:10:58.124281 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:58.124265 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3be73c6-e989-4ccc-aacf-a9ef3d8f1378-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-vblkn\" (UID: \"a3be73c6-e989-4ccc-aacf-a9ef3d8f1378\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vblkn" Jul 06 20:10:58.124397 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:58.124381 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a3be73c6-e989-4ccc-aacf-a9ef3d8f1378-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-vblkn\" (UID: \"a3be73c6-e989-4ccc-aacf-a9ef3d8f1378\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vblkn" Jul 06 20:10:58.134227 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:58.134198 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3be73c6-e989-4ccc-aacf-a9ef3d8f1378-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-vblkn\" (UID: \"a3be73c6-e989-4ccc-aacf-a9ef3d8f1378\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vblkn" Jul 06 20:10:58.134396 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:58.134380 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2nfc\" (UniqueName: \"kubernetes.io/projected/a3be73c6-e989-4ccc-aacf-a9ef3d8f1378-kube-api-access-b2nfc\") pod \"istiod-openshift-gateway-94bb8fbfd-vblkn\" (UID: \"a3be73c6-e989-4ccc-aacf-a9ef3d8f1378\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vblkn" Jul 06 20:10:58.292363 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:58.292261 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vblkn" Jul 06 20:10:58.430727 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:58.430569 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vblkn"] Jul 06 20:10:58.433115 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:10:58.433088 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3be73c6_e989_4ccc_aacf_a9ef3d8f1378.slice/crio-118c2457b26e2ad611a50075b0e002a31b84c9bdf118ca671f9a3f4fa9bcb2a3 WatchSource:0}: Error finding container 118c2457b26e2ad611a50075b0e002a31b84c9bdf118ca671f9a3f4fa9bcb2a3: Status 404 returned error can't find the container with id 118c2457b26e2ad611a50075b0e002a31b84c9bdf118ca671f9a3f4fa9bcb2a3 Jul 06 20:10:58.445535 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:58.445507 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-79c95fdbc9-rhh4g" event={"ID":"0def7154-736a-4490-acf2-93ccb9ae1a35","Type":"ContainerStarted","Data":"95b4bdf5d298b778e56204470635b892594b2c047bd4f5dc6a039acbea6110c8"} Jul 06 20:10:58.446953 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:58.446931 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vblkn" event={"ID":"a3be73c6-e989-4ccc-aacf-a9ef3d8f1378","Type":"ContainerStarted","Data":"118c2457b26e2ad611a50075b0e002a31b84c9bdf118ca671f9a3f4fa9bcb2a3"} Jul 06 20:10:59.853363 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:59.853325 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:10:59.964317 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:10:59.964273 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:11:00.081894 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:11:00.081838 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:11:00.156567 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:11:00.156514 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:11:00.265063 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:11:00.265035 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:11:00.356859 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:11:00.356798 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:11:00.414649 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:11:00.414549 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:11:01.502801 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:11:01.502767 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 20:11:01.503072 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:11:01.502834 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 20:11:02.466840 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:11:02.466793 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-79c95fdbc9-rhh4g" event={"ID":"0def7154-736a-4490-acf2-93ccb9ae1a35","Type":"ContainerStarted","Data":"d3473025931804c9503d25c40d6cad1555040f3f4d630fdd520f1d2b41274b21"} Jul 06 20:11:02.467055 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:11:02.467031 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-79c95fdbc9-rhh4g" Jul 06 20:11:02.469098 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:11:02.469062 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vblkn" event={"ID":"a3be73c6-e989-4ccc-aacf-a9ef3d8f1378","Type":"ContainerStarted","Data":"4b247bb4bf3fa55edfe1726472ea0191821b7140b144d090f903a7b4ec45154e"} Jul 06 20:11:02.469299 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:11:02.469282 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vblkn" Jul 06 20:11:02.471508 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:11:02.471480 2578 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-vblkn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 20:11:02.471628 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:11:02.471546 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vblkn" podUID="a3be73c6-e989-4ccc-aacf-a9ef3d8f1378" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:11:02.505293 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:11:02.505223 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-79c95fdbc9-rhh4g" podStartSLOduration=2.05079821 podStartE2EDuration="5.505201081s" totalImagesPullingTime="3.454402871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:10:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:11:02.5040541 +0000 UTC m=+313.715202319" watchObservedRunningTime="2026-07-06 20:11:02.505201081 +0000 UTC m=+313.716349300" Jul 06 20:11:02.542394 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:11:02.542331 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vblkn" podStartSLOduration=2.474914969 podStartE2EDuration="5.542306866s" totalImagesPullingTime="3.067391897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:10:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:11:02.537962169 +0000 UTC m=+313.749110389" watchObservedRunningTime="2026-07-06 20:11:02.542306866 +0000 UTC m=+313.753455085" Jul 06 20:11:02.776347 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:11:02.776248 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:11:02.841015 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:11:02.840970 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:11:02.899961 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:11:02.899922 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:11:03.473294 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:11:03.473265 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vblkn" Jul 06 20:11:13.475382 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:11:13.475343 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-79c95fdbc9-rhh4g" Jul 06 20:11:17.978429 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:11:17.978387 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:11:48.524216 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:11:48.524126 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:11:48.623598 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:11:48.623560 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:11:48.751831 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:11:48.751801 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:18.168999 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:18.168956 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:21.748066 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:21.748033 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:21.848526 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:21.848485 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:25.016930 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:25.016897 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:25.076852 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:25.076818 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:26.812540 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:26.812504 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:27.016541 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:27.016510 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:29.162002 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:29.161960 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:29.425354 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:29.425256 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:29.563285 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:29.563237 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:31.787662 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:31.787627 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:31.887397 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:31.887358 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:33.770913 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:33.770877 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:33.875205 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:33.875170 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:36.149312 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:36.149262 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:36.671838 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:36.671804 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:37.013115 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:37.013024 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:37.124219 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:37.124182 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:37.213697 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:37.213666 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:37.374425 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:37.374341 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:37.859378 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:37.859345 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:37.962904 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:37.962873 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:43.265687 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:43.265651 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:43.366034 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:43.366001 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:46.366824 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:46.366787 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-54h95"] Jul 06 20:12:46.369100 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:46.369080 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-54h95" Jul 06 20:12:46.372900 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:46.372878 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 20:12:46.373021 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:46.372929 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 20:12:46.373226 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:46.373212 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-pn8tl\"" Jul 06 20:12:46.376886 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:46.376864 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/35a5306e-6521-4ee2-825a-95e29bfa109a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-54h95\" (UID: \"35a5306e-6521-4ee2-825a-95e29bfa109a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-54h95" Jul 06 20:12:46.377028 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:46.376911 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbbl5\" (UniqueName: \"kubernetes.io/projected/35a5306e-6521-4ee2-825a-95e29bfa109a-kube-api-access-tbbl5\") pod \"kuadrant-operator-controller-manager-55467d9f7-54h95\" (UID: \"35a5306e-6521-4ee2-825a-95e29bfa109a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-54h95" Jul 06 20:12:46.382818 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:46.382782 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-54h95"] Jul 06 20:12:46.449329 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:46.449296 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-54h95"] Jul 06 20:12:46.449650 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:12:46.449623 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-tbbl5], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-54h95" podUID="35a5306e-6521-4ee2-825a-95e29bfa109a" Jul 06 20:12:46.477795 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:46.477756 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbbl5\" (UniqueName: \"kubernetes.io/projected/35a5306e-6521-4ee2-825a-95e29bfa109a-kube-api-access-tbbl5\") pod \"kuadrant-operator-controller-manager-55467d9f7-54h95\" (UID: \"35a5306e-6521-4ee2-825a-95e29bfa109a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-54h95" Jul 06 20:12:46.477973 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:46.477836 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/35a5306e-6521-4ee2-825a-95e29bfa109a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-54h95\" (UID: \"35a5306e-6521-4ee2-825a-95e29bfa109a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-54h95" Jul 06 20:12:46.478250 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:46.478233 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/35a5306e-6521-4ee2-825a-95e29bfa109a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-54h95\" (UID: \"35a5306e-6521-4ee2-825a-95e29bfa109a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-54h95" Jul 06 20:12:46.500970 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:46.500938 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbbl5\" (UniqueName: \"kubernetes.io/projected/35a5306e-6521-4ee2-825a-95e29bfa109a-kube-api-access-tbbl5\") pod \"kuadrant-operator-controller-manager-55467d9f7-54h95\" (UID: \"35a5306e-6521-4ee2-825a-95e29bfa109a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-54h95" Jul 06 20:12:46.841154 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:46.841117 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-54h95" Jul 06 20:12:46.847370 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:46.847342 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-54h95" Jul 06 20:12:46.880035 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:46.879997 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbbl5\" (UniqueName: \"kubernetes.io/projected/35a5306e-6521-4ee2-825a-95e29bfa109a-kube-api-access-tbbl5\") pod \"35a5306e-6521-4ee2-825a-95e29bfa109a\" (UID: \"35a5306e-6521-4ee2-825a-95e29bfa109a\") " Jul 06 20:12:46.880237 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:46.880056 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/35a5306e-6521-4ee2-825a-95e29bfa109a-extensions-socket-volume\") pod \"35a5306e-6521-4ee2-825a-95e29bfa109a\" (UID: \"35a5306e-6521-4ee2-825a-95e29bfa109a\") " Jul 06 20:12:46.880406 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:46.880376 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a5306e-6521-4ee2-825a-95e29bfa109a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "35a5306e-6521-4ee2-825a-95e29bfa109a" (UID: "35a5306e-6521-4ee2-825a-95e29bfa109a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:12:46.882382 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:46.882356 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a5306e-6521-4ee2-825a-95e29bfa109a-kube-api-access-tbbl5" (OuterVolumeSpecName: "kube-api-access-tbbl5") pod "35a5306e-6521-4ee2-825a-95e29bfa109a" (UID: "35a5306e-6521-4ee2-825a-95e29bfa109a"). InnerVolumeSpecName "kube-api-access-tbbl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:12:46.981324 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:46.981285 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbbl5\" (UniqueName: \"kubernetes.io/projected/35a5306e-6521-4ee2-825a-95e29bfa109a-kube-api-access-tbbl5\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:12:46.981324 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:46.981319 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/35a5306e-6521-4ee2-825a-95e29bfa109a-extensions-socket-volume\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:12:47.845108 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:47.845071 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-54h95" Jul 06 20:12:47.901710 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:47.901678 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-54h95"] Jul 06 20:12:47.910019 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:47.909988 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-54h95"] Jul 06 20:12:48.095648 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:48.095552 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:48.209989 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:48.209948 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:48.302512 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:48.302483 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:49.297675 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:49.297647 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35a5306e-6521-4ee2-825a-95e29bfa109a" path="/var/lib/kubelet/pods/35a5306e-6521-4ee2-825a-95e29bfa109a/volumes" Jul 06 20:12:49.883095 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:49.883058 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:49.990826 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:49.990787 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:52.446191 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:52.446140 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:52.543645 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:52.543612 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:55.996167 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:55.996128 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:12:56.069849 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:12:56.069815 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:03.853361 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:03.853320 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:03.919252 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:03.919209 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:10.428800 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:10.428708 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:10.522420 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:10.522383 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:15.489437 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:15.489398 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:15.595050 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:15.595015 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:20.274929 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:20.274886 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:20.285902 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:20.285870 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-qhlvw"] Jul 06 20:13:20.288350 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:20.288333 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-qhlvw" Jul 06 20:13:20.291022 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:20.290999 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 20:13:20.291022 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:20.291002 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 06 20:13:20.291284 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:20.291128 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 20:13:20.291917 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:20.291899 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fgg7s\"" Jul 06 20:13:20.311731 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:20.311699 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-qhlvw"] Jul 06 20:13:20.357385 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:20.357344 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95z6g\" (UniqueName: \"kubernetes.io/projected/6cc25409-a07a-42ed-bf0d-d3b65e0981ee-kube-api-access-95z6g\") pod \"limitador-limitador-64c94fd64f-qhlvw\" (UID: \"6cc25409-a07a-42ed-bf0d-d3b65e0981ee\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-qhlvw" Jul 06 20:13:20.357385 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:20.357403 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6cc25409-a07a-42ed-bf0d-d3b65e0981ee-config-file\") pod \"limitador-limitador-64c94fd64f-qhlvw\" (UID: \"6cc25409-a07a-42ed-bf0d-d3b65e0981ee\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-qhlvw" Jul 06 20:13:20.386610 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:20.386576 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-qhlvw"] Jul 06 20:13:20.388345 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:20.388317 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:20.458486 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:20.458422 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6cc25409-a07a-42ed-bf0d-d3b65e0981ee-config-file\") pod \"limitador-limitador-64c94fd64f-qhlvw\" (UID: \"6cc25409-a07a-42ed-bf0d-d3b65e0981ee\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-qhlvw" Jul 06 20:13:20.458737 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:20.458547 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95z6g\" (UniqueName: \"kubernetes.io/projected/6cc25409-a07a-42ed-bf0d-d3b65e0981ee-kube-api-access-95z6g\") pod \"limitador-limitador-64c94fd64f-qhlvw\" (UID: \"6cc25409-a07a-42ed-bf0d-d3b65e0981ee\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-qhlvw" Jul 06 20:13:20.459100 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:20.459080 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6cc25409-a07a-42ed-bf0d-d3b65e0981ee-config-file\") pod \"limitador-limitador-64c94fd64f-qhlvw\" (UID: \"6cc25409-a07a-42ed-bf0d-d3b65e0981ee\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-qhlvw" Jul 06 20:13:20.471216 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:20.471178 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95z6g\" (UniqueName: \"kubernetes.io/projected/6cc25409-a07a-42ed-bf0d-d3b65e0981ee-kube-api-access-95z6g\") pod \"limitador-limitador-64c94fd64f-qhlvw\" (UID: \"6cc25409-a07a-42ed-bf0d-d3b65e0981ee\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-qhlvw" Jul 06 20:13:20.504284 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:20.504242 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:20.598311 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:20.598229 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-qhlvw" Jul 06 20:13:20.743537 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:20.743503 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-qhlvw"] Jul 06 20:13:20.746408 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:13:20.746375 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc25409_a07a_42ed_bf0d_d3b65e0981ee.slice/crio-40706517aed0e8516d0248c2c5c64cda793cf370651238a2698d6048bcc4a5da WatchSource:0}: Error finding container 40706517aed0e8516d0248c2c5c64cda793cf370651238a2698d6048bcc4a5da: Status 404 returned error can't find the container with id 40706517aed0e8516d0248c2c5c64cda793cf370651238a2698d6048bcc4a5da Jul 06 20:13:20.965397 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:20.965365 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-qhlvw" event={"ID":"6cc25409-a07a-42ed-bf0d-d3b65e0981ee","Type":"ContainerStarted","Data":"40706517aed0e8516d0248c2c5c64cda793cf370651238a2698d6048bcc4a5da"} Jul 06 20:13:21.406447 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:21.406356 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:21.470822 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:21.470783 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:23.977924 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:23.977881 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-qhlvw" event={"ID":"6cc25409-a07a-42ed-bf0d-d3b65e0981ee","Type":"ContainerStarted","Data":"a285af44da379f243ef03677600ccc607f446552ad973c3266fc67f679aa0367"} Jul 06 20:13:23.978411 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:23.978009 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-qhlvw" Jul 06 20:13:23.999226 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:23.999173 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-qhlvw" podStartSLOduration=1.411401034 podStartE2EDuration="3.99915356s" totalImagesPullingTime="2.587752526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:13:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:13:23.998445851 +0000 UTC m=+455.209594070" watchObservedRunningTime="2026-07-06 20:13:23.99915356 +0000 UTC m=+455.210301783" Jul 06 20:13:34.983026 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:34.982998 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-qhlvw" Jul 06 20:13:36.114536 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:36.114501 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-qhlvw"] Jul 06 20:13:36.114945 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:36.114684 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-qhlvw" podUID="6cc25409-a07a-42ed-bf0d-d3b65e0981ee" containerName="limitador" containerID="cri-o://a285af44da379f243ef03677600ccc607f446552ad973c3266fc67f679aa0367" gracePeriod=30 Jul 06 20:13:36.661135 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:36.661108 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-qhlvw" Jul 06 20:13:36.801519 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:36.801408 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95z6g\" (UniqueName: \"kubernetes.io/projected/6cc25409-a07a-42ed-bf0d-d3b65e0981ee-kube-api-access-95z6g\") pod \"6cc25409-a07a-42ed-bf0d-d3b65e0981ee\" (UID: \"6cc25409-a07a-42ed-bf0d-d3b65e0981ee\") " Jul 06 20:13:36.801519 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:36.801480 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6cc25409-a07a-42ed-bf0d-d3b65e0981ee-config-file\") pod \"6cc25409-a07a-42ed-bf0d-d3b65e0981ee\" (UID: \"6cc25409-a07a-42ed-bf0d-d3b65e0981ee\") " Jul 06 20:13:36.801791 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:36.801767 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cc25409-a07a-42ed-bf0d-d3b65e0981ee-config-file" (OuterVolumeSpecName: "config-file") pod "6cc25409-a07a-42ed-bf0d-d3b65e0981ee" (UID: "6cc25409-a07a-42ed-bf0d-d3b65e0981ee"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:13:36.803784 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:36.803748 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cc25409-a07a-42ed-bf0d-d3b65e0981ee-kube-api-access-95z6g" (OuterVolumeSpecName: "kube-api-access-95z6g") pod "6cc25409-a07a-42ed-bf0d-d3b65e0981ee" (UID: "6cc25409-a07a-42ed-bf0d-d3b65e0981ee"). InnerVolumeSpecName "kube-api-access-95z6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:13:36.902428 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:36.902387 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95z6g\" (UniqueName: \"kubernetes.io/projected/6cc25409-a07a-42ed-bf0d-d3b65e0981ee-kube-api-access-95z6g\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:13:36.902428 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:36.902421 2578 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6cc25409-a07a-42ed-bf0d-d3b65e0981ee-config-file\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:13:37.031354 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:37.031311 2578 generic.go:356] "Generic (PLEG): container finished" podID="6cc25409-a07a-42ed-bf0d-d3b65e0981ee" containerID="a285af44da379f243ef03677600ccc607f446552ad973c3266fc67f679aa0367" exitCode=0 Jul 06 20:13:37.031354 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:37.031357 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-qhlvw" event={"ID":"6cc25409-a07a-42ed-bf0d-d3b65e0981ee","Type":"ContainerDied","Data":"a285af44da379f243ef03677600ccc607f446552ad973c3266fc67f679aa0367"} Jul 06 20:13:37.031615 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:37.031364 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-qhlvw" Jul 06 20:13:37.031615 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:37.031414 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-qhlvw" event={"ID":"6cc25409-a07a-42ed-bf0d-d3b65e0981ee","Type":"ContainerDied","Data":"40706517aed0e8516d0248c2c5c64cda793cf370651238a2698d6048bcc4a5da"} Jul 06 20:13:37.031615 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:37.031491 2578 scope.go:117] "RemoveContainer" containerID="a285af44da379f243ef03677600ccc607f446552ad973c3266fc67f679aa0367" Jul 06 20:13:37.041088 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:37.041063 2578 scope.go:117] "RemoveContainer" containerID="a285af44da379f243ef03677600ccc607f446552ad973c3266fc67f679aa0367" Jul 06 20:13:37.041396 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:13:37.041375 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a285af44da379f243ef03677600ccc607f446552ad973c3266fc67f679aa0367\": container with ID starting with a285af44da379f243ef03677600ccc607f446552ad973c3266fc67f679aa0367 not found: ID does not exist" containerID="a285af44da379f243ef03677600ccc607f446552ad973c3266fc67f679aa0367" Jul 06 20:13:37.041491 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:37.041410 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a285af44da379f243ef03677600ccc607f446552ad973c3266fc67f679aa0367"} err="failed to get container status \"a285af44da379f243ef03677600ccc607f446552ad973c3266fc67f679aa0367\": rpc error: code = NotFound desc = could not find container \"a285af44da379f243ef03677600ccc607f446552ad973c3266fc67f679aa0367\": container with ID starting with a285af44da379f243ef03677600ccc607f446552ad973c3266fc67f679aa0367 not found: ID does not exist" Jul 06 20:13:37.057481 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:37.057383 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-qhlvw"] Jul 06 20:13:37.062146 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:37.062116 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-qhlvw"] Jul 06 20:13:37.298180 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:37.298146 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cc25409-a07a-42ed-bf0d-d3b65e0981ee" path="/var/lib/kubelet/pods/6cc25409-a07a-42ed-bf0d-d3b65e0981ee/volumes" Jul 06 20:13:41.345654 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:41.345611 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:41.429555 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:41.429517 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:41.784043 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:41.784010 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-bws4b"] Jul 06 20:13:41.784280 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:41.784268 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cc25409-a07a-42ed-bf0d-d3b65e0981ee" containerName="limitador" Jul 06 20:13:41.784280 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:41.784281 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cc25409-a07a-42ed-bf0d-d3b65e0981ee" containerName="limitador" Jul 06 20:13:41.784399 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:41.784340 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cc25409-a07a-42ed-bf0d-d3b65e0981ee" containerName="limitador" Jul 06 20:13:41.787820 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:41.787798 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bws4b" Jul 06 20:13:41.790548 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:41.790450 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 06 20:13:41.790548 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:41.790527 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 06 20:13:41.790548 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:41.790527 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-bxpcq\"" Jul 06 20:13:41.791306 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:41.791283 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 06 20:13:41.797815 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:41.797791 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-bws4b"] Jul 06 20:13:41.826402 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:41.826370 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:41.840846 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:41.840800 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3542392d-c2b3-4fb5-be9a-87efd976b054-data\") pod \"postgres-69fb4594cf-bws4b\" (UID: \"3542392d-c2b3-4fb5-be9a-87efd976b054\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bws4b" Jul 06 20:13:41.841003 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:41.840862 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nb94\" (UniqueName: \"kubernetes.io/projected/3542392d-c2b3-4fb5-be9a-87efd976b054-kube-api-access-2nb94\") pod \"postgres-69fb4594cf-bws4b\" (UID: \"3542392d-c2b3-4fb5-be9a-87efd976b054\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bws4b" Jul 06 20:13:41.941407 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:41.941375 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3542392d-c2b3-4fb5-be9a-87efd976b054-data\") pod \"postgres-69fb4594cf-bws4b\" (UID: \"3542392d-c2b3-4fb5-be9a-87efd976b054\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bws4b" Jul 06 20:13:41.941407 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:41.941412 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nb94\" (UniqueName: \"kubernetes.io/projected/3542392d-c2b3-4fb5-be9a-87efd976b054-kube-api-access-2nb94\") pod \"postgres-69fb4594cf-bws4b\" (UID: \"3542392d-c2b3-4fb5-be9a-87efd976b054\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bws4b" Jul 06 20:13:41.941822 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:41.941804 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3542392d-c2b3-4fb5-be9a-87efd976b054-data\") pod \"postgres-69fb4594cf-bws4b\" (UID: \"3542392d-c2b3-4fb5-be9a-87efd976b054\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bws4b" Jul 06 20:13:41.952669 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:41.952641 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nb94\" (UniqueName: \"kubernetes.io/projected/3542392d-c2b3-4fb5-be9a-87efd976b054-kube-api-access-2nb94\") pod \"postgres-69fb4594cf-bws4b\" (UID: \"3542392d-c2b3-4fb5-be9a-87efd976b054\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bws4b" Jul 06 20:13:42.101147 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:42.101043 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bws4b" Jul 06 20:13:42.235135 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:42.235100 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-bws4b"] Jul 06 20:13:42.238748 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:13:42.238722 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3542392d_c2b3_4fb5_be9a_87efd976b054.slice/crio-c48e90899157e8fac6132480fbf39e63c0c9f0634c636b526157957c50dda1a0 WatchSource:0}: Error finding container c48e90899157e8fac6132480fbf39e63c0c9f0634c636b526157957c50dda1a0: Status 404 returned error can't find the container with id c48e90899157e8fac6132480fbf39e63c0c9f0634c636b526157957c50dda1a0 Jul 06 20:13:43.054778 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:43.054748 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bws4b" event={"ID":"3542392d-c2b3-4fb5-be9a-87efd976b054","Type":"ContainerStarted","Data":"c48e90899157e8fac6132480fbf39e63c0c9f0634c636b526157957c50dda1a0"} Jul 06 20:13:48.078985 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:48.078945 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bws4b" event={"ID":"3542392d-c2b3-4fb5-be9a-87efd976b054","Type":"ContainerStarted","Data":"a9537e1f251236e9bf0dff2f354133eaf8ff11667892d31b021c5810e1e91dbd"} Jul 06 20:13:48.079357 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:48.079072 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bws4b" Jul 06 20:13:49.173593 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:49.173533 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bws4b" podStartSLOduration=3.29223553 podStartE2EDuration="8.173513096s" totalImagesPullingTime="4.881277566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:13:49.169181361 +0000 UTC m=+480.380329580" watchObservedRunningTime="2026-07-06 20:13:49.173513096 +0000 UTC m=+480.384661315" Jul 06 20:13:54.113352 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:54.113320 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bws4b" Jul 06 20:13:54.470498 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:54.470442 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:13:54.711703 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:13:54.711661 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:14:04.689144 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:04.689107 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5df956d7f8-mxqtm"] Jul 06 20:14:04.691887 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:04.691865 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5df956d7f8-mxqtm" Jul 06 20:14:04.694604 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:04.694573 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 20:14:04.694604 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:04.694596 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 20:14:04.694772 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:04.694642 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7clct\"" Jul 06 20:14:04.705820 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:04.705789 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5df956d7f8-mxqtm"] Jul 06 20:14:04.722545 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:04.722495 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:14:04.725341 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:04.725309 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc27c793-61db-4260-b8f0-f60d5cdd35cb-webhook-cert\") pod \"maas-controller-5df956d7f8-mxqtm\" (UID: \"dc27c793-61db-4260-b8f0-f60d5cdd35cb\") " pod="opendatahub/maas-controller-5df956d7f8-mxqtm" Jul 06 20:14:04.725445 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:04.725421 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzww6\" (UniqueName: \"kubernetes.io/projected/dc27c793-61db-4260-b8f0-f60d5cdd35cb-kube-api-access-xzww6\") pod \"maas-controller-5df956d7f8-mxqtm\" (UID: \"dc27c793-61db-4260-b8f0-f60d5cdd35cb\") " pod="opendatahub/maas-controller-5df956d7f8-mxqtm" Jul 06 20:14:04.820629 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:04.820587 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:14:04.826472 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:04.826439 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc27c793-61db-4260-b8f0-f60d5cdd35cb-webhook-cert\") pod \"maas-controller-5df956d7f8-mxqtm\" (UID: \"dc27c793-61db-4260-b8f0-f60d5cdd35cb\") " pod="opendatahub/maas-controller-5df956d7f8-mxqtm" Jul 06 20:14:04.826581 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:04.826525 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzww6\" (UniqueName: \"kubernetes.io/projected/dc27c793-61db-4260-b8f0-f60d5cdd35cb-kube-api-access-xzww6\") pod \"maas-controller-5df956d7f8-mxqtm\" (UID: \"dc27c793-61db-4260-b8f0-f60d5cdd35cb\") " pod="opendatahub/maas-controller-5df956d7f8-mxqtm" Jul 06 20:14:04.826581 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:14:04.826544 2578 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 06 20:14:04.826653 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:14:04.826605 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc27c793-61db-4260-b8f0-f60d5cdd35cb-webhook-cert podName:dc27c793-61db-4260-b8f0-f60d5cdd35cb nodeName:}" failed. No retries permitted until 2026-07-06 20:14:05.326588577 +0000 UTC m=+496.537736774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/dc27c793-61db-4260-b8f0-f60d5cdd35cb-webhook-cert") pod "maas-controller-5df956d7f8-mxqtm" (UID: "dc27c793-61db-4260-b8f0-f60d5cdd35cb") : secret "maas-controller-webhook-cert" not found Jul 06 20:14:04.839862 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:04.839835 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzww6\" (UniqueName: \"kubernetes.io/projected/dc27c793-61db-4260-b8f0-f60d5cdd35cb-kube-api-access-xzww6\") pod \"maas-controller-5df956d7f8-mxqtm\" (UID: \"dc27c793-61db-4260-b8f0-f60d5cdd35cb\") " pod="opendatahub/maas-controller-5df956d7f8-mxqtm" Jul 06 20:14:04.888906 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:04.888871 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65954644c5-jk49n"] Jul 06 20:14:04.891312 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:04.891290 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65954644c5-jk49n" Jul 06 20:14:04.902943 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:04.902912 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65954644c5-jk49n"] Jul 06 20:14:04.921877 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:04.921850 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:14:04.926973 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:04.926952 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d48c5414-9ad7-4052-be71-f44ca1b20a9c-webhook-cert\") pod \"maas-controller-65954644c5-jk49n\" (UID: \"d48c5414-9ad7-4052-be71-f44ca1b20a9c\") " pod="opendatahub/maas-controller-65954644c5-jk49n" Jul 06 20:14:04.927079 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:04.926989 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv2hz\" (UniqueName: \"kubernetes.io/projected/d48c5414-9ad7-4052-be71-f44ca1b20a9c-kube-api-access-sv2hz\") pod \"maas-controller-65954644c5-jk49n\" (UID: \"d48c5414-9ad7-4052-be71-f44ca1b20a9c\") " pod="opendatahub/maas-controller-65954644c5-jk49n" Jul 06 20:14:05.027980 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:05.027892 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv2hz\" (UniqueName: \"kubernetes.io/projected/d48c5414-9ad7-4052-be71-f44ca1b20a9c-kube-api-access-sv2hz\") pod \"maas-controller-65954644c5-jk49n\" (UID: \"d48c5414-9ad7-4052-be71-f44ca1b20a9c\") " pod="opendatahub/maas-controller-65954644c5-jk49n" Jul 06 20:14:05.028143 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:05.028002 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d48c5414-9ad7-4052-be71-f44ca1b20a9c-webhook-cert\") pod \"maas-controller-65954644c5-jk49n\" (UID: \"d48c5414-9ad7-4052-be71-f44ca1b20a9c\") " pod="opendatahub/maas-controller-65954644c5-jk49n" Jul 06 20:14:05.030540 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:05.030521 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d48c5414-9ad7-4052-be71-f44ca1b20a9c-webhook-cert\") pod \"maas-controller-65954644c5-jk49n\" (UID: \"d48c5414-9ad7-4052-be71-f44ca1b20a9c\") " pod="opendatahub/maas-controller-65954644c5-jk49n" Jul 06 20:14:05.038249 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:05.038228 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv2hz\" (UniqueName: \"kubernetes.io/projected/d48c5414-9ad7-4052-be71-f44ca1b20a9c-kube-api-access-sv2hz\") pod \"maas-controller-65954644c5-jk49n\" (UID: \"d48c5414-9ad7-4052-be71-f44ca1b20a9c\") " pod="opendatahub/maas-controller-65954644c5-jk49n" Jul 06 20:14:05.201467 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:05.201432 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65954644c5-jk49n" Jul 06 20:14:05.330174 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:05.330144 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc27c793-61db-4260-b8f0-f60d5cdd35cb-webhook-cert\") pod \"maas-controller-5df956d7f8-mxqtm\" (UID: \"dc27c793-61db-4260-b8f0-f60d5cdd35cb\") " pod="opendatahub/maas-controller-5df956d7f8-mxqtm" Jul 06 20:14:05.333086 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:05.333060 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc27c793-61db-4260-b8f0-f60d5cdd35cb-webhook-cert\") pod \"maas-controller-5df956d7f8-mxqtm\" (UID: \"dc27c793-61db-4260-b8f0-f60d5cdd35cb\") " pod="opendatahub/maas-controller-5df956d7f8-mxqtm" Jul 06 20:14:05.333796 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:05.333776 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65954644c5-jk49n"] Jul 06 20:14:05.335975 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:14:05.335954 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd48c5414_9ad7_4052_be71_f44ca1b20a9c.slice/crio-3021784ad68d837dde62030ea2069f83c42e7cdb4166f7bc43b3d9f19702b8bf WatchSource:0}: Error finding container 3021784ad68d837dde62030ea2069f83c42e7cdb4166f7bc43b3d9f19702b8bf: Status 404 returned error can't find the container with id 3021784ad68d837dde62030ea2069f83c42e7cdb4166f7bc43b3d9f19702b8bf Jul 06 20:14:05.601487 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:05.601349 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5df956d7f8-mxqtm" Jul 06 20:14:05.736183 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:05.736138 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5df956d7f8-mxqtm"] Jul 06 20:14:05.738879 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:14:05.738845 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc27c793_61db_4260_b8f0_f60d5cdd35cb.slice/crio-f4b13cfbee88bd258c95c9a634dfcca00220f3f3242d011070b4a9ef8c0acd62 WatchSource:0}: Error finding container f4b13cfbee88bd258c95c9a634dfcca00220f3f3242d011070b4a9ef8c0acd62: Status 404 returned error can't find the container with id f4b13cfbee88bd258c95c9a634dfcca00220f3f3242d011070b4a9ef8c0acd62 Jul 06 20:14:06.148567 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:06.148523 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65954644c5-jk49n" event={"ID":"d48c5414-9ad7-4052-be71-f44ca1b20a9c","Type":"ContainerStarted","Data":"3021784ad68d837dde62030ea2069f83c42e7cdb4166f7bc43b3d9f19702b8bf"} Jul 06 20:14:06.150751 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:06.150723 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5df956d7f8-mxqtm" event={"ID":"dc27c793-61db-4260-b8f0-f60d5cdd35cb","Type":"ContainerStarted","Data":"f4b13cfbee88bd258c95c9a634dfcca00220f3f3242d011070b4a9ef8c0acd62"} Jul 06 20:14:07.701501 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:07.701434 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:14:08.165649 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:08.165616 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65954644c5-jk49n" event={"ID":"d48c5414-9ad7-4052-be71-f44ca1b20a9c","Type":"ContainerStarted","Data":"693fc4db6338a960a6252347e5c666b4dce4f6551d27df7ea44d31dc97db9c8d"} Jul 06 20:14:08.165837 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:08.165788 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65954644c5-jk49n" Jul 06 20:14:08.167093 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:08.167067 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5df956d7f8-mxqtm" event={"ID":"dc27c793-61db-4260-b8f0-f60d5cdd35cb","Type":"ContainerStarted","Data":"80ffe977287a4dbb84093df436556231a627eabba3ff462b4955e75cf9af21d5"} Jul 06 20:14:08.167212 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:08.167197 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5df956d7f8-mxqtm" Jul 06 20:14:08.191756 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:08.191688 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-65954644c5-jk49n" podStartSLOduration=1.478184235 podStartE2EDuration="4.191668698s" totalImagesPullingTime="2.713484463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:14:08.189378258 +0000 UTC m=+499.400526476" watchObservedRunningTime="2026-07-06 20:14:08.191668698 +0000 UTC m=+499.402816917" Jul 06 20:14:08.213409 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:08.213343 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5df956d7f8-mxqtm" podStartSLOduration=1.894008457 podStartE2EDuration="4.213319535s" totalImagesPullingTime="2.319311078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:14:08.210450102 +0000 UTC m=+499.421598325" watchObservedRunningTime="2026-07-06 20:14:08.213319535 +0000 UTC m=+499.424467754" Jul 06 20:14:08.287400 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:08.287359 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:14:08.382067 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:08.381969 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:14:19.176428 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:19.176400 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5df956d7f8-mxqtm" Jul 06 20:14:19.176869 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:19.176515 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65954644c5-jk49n" Jul 06 20:14:19.260200 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:19.260167 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5df956d7f8-mxqtm"] Jul 06 20:14:19.260392 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:19.260367 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-5df956d7f8-mxqtm" podUID="dc27c793-61db-4260-b8f0-f60d5cdd35cb" containerName="manager" containerID="cri-o://80ffe977287a4dbb84093df436556231a627eabba3ff462b4955e75cf9af21d5" gracePeriod=30 Jul 06 20:14:20.020422 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:20.020397 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5df956d7f8-mxqtm" Jul 06 20:14:20.155761 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:20.155683 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc27c793-61db-4260-b8f0-f60d5cdd35cb-webhook-cert\") pod \"dc27c793-61db-4260-b8f0-f60d5cdd35cb\" (UID: \"dc27c793-61db-4260-b8f0-f60d5cdd35cb\") " Jul 06 20:14:20.155761 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:20.155749 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzww6\" (UniqueName: \"kubernetes.io/projected/dc27c793-61db-4260-b8f0-f60d5cdd35cb-kube-api-access-xzww6\") pod \"dc27c793-61db-4260-b8f0-f60d5cdd35cb\" (UID: \"dc27c793-61db-4260-b8f0-f60d5cdd35cb\") " Jul 06 20:14:20.158028 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:20.158000 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc27c793-61db-4260-b8f0-f60d5cdd35cb-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "dc27c793-61db-4260-b8f0-f60d5cdd35cb" (UID: "dc27c793-61db-4260-b8f0-f60d5cdd35cb"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:14:20.158144 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:20.158037 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc27c793-61db-4260-b8f0-f60d5cdd35cb-kube-api-access-xzww6" (OuterVolumeSpecName: "kube-api-access-xzww6") pod "dc27c793-61db-4260-b8f0-f60d5cdd35cb" (UID: "dc27c793-61db-4260-b8f0-f60d5cdd35cb"). InnerVolumeSpecName "kube-api-access-xzww6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:14:20.212695 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:20.212658 2578 generic.go:356] "Generic (PLEG): container finished" podID="dc27c793-61db-4260-b8f0-f60d5cdd35cb" containerID="80ffe977287a4dbb84093df436556231a627eabba3ff462b4955e75cf9af21d5" exitCode=0 Jul 06 20:14:20.213073 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:20.212705 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5df956d7f8-mxqtm" Jul 06 20:14:20.213073 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:20.212748 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5df956d7f8-mxqtm" event={"ID":"dc27c793-61db-4260-b8f0-f60d5cdd35cb","Type":"ContainerDied","Data":"80ffe977287a4dbb84093df436556231a627eabba3ff462b4955e75cf9af21d5"} Jul 06 20:14:20.213073 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:20.212803 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5df956d7f8-mxqtm" event={"ID":"dc27c793-61db-4260-b8f0-f60d5cdd35cb","Type":"ContainerDied","Data":"f4b13cfbee88bd258c95c9a634dfcca00220f3f3242d011070b4a9ef8c0acd62"} Jul 06 20:14:20.213073 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:20.212830 2578 scope.go:117] "RemoveContainer" containerID="80ffe977287a4dbb84093df436556231a627eabba3ff462b4955e75cf9af21d5" Jul 06 20:14:20.221821 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:20.221803 2578 scope.go:117] "RemoveContainer" containerID="80ffe977287a4dbb84093df436556231a627eabba3ff462b4955e75cf9af21d5" Jul 06 20:14:20.222056 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:14:20.222040 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80ffe977287a4dbb84093df436556231a627eabba3ff462b4955e75cf9af21d5\": container with ID starting with 80ffe977287a4dbb84093df436556231a627eabba3ff462b4955e75cf9af21d5 not found: ID does not exist" containerID="80ffe977287a4dbb84093df436556231a627eabba3ff462b4955e75cf9af21d5" Jul 06 20:14:20.222112 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:20.222062 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80ffe977287a4dbb84093df436556231a627eabba3ff462b4955e75cf9af21d5"} err="failed to get container status \"80ffe977287a4dbb84093df436556231a627eabba3ff462b4955e75cf9af21d5\": rpc error: code = NotFound desc = could not find container \"80ffe977287a4dbb84093df436556231a627eabba3ff462b4955e75cf9af21d5\": container with ID starting with 80ffe977287a4dbb84093df436556231a627eabba3ff462b4955e75cf9af21d5 not found: ID does not exist" Jul 06 20:14:20.240827 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:20.240801 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5df956d7f8-mxqtm"] Jul 06 20:14:20.247096 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:20.247072 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5df956d7f8-mxqtm"] Jul 06 20:14:20.256846 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:20.256828 2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc27c793-61db-4260-b8f0-f60d5cdd35cb-webhook-cert\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:14:20.256912 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:20.256850 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzww6\" (UniqueName: \"kubernetes.io/projected/dc27c793-61db-4260-b8f0-f60d5cdd35cb-kube-api-access-xzww6\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:14:21.297212 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:21.297181 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc27c793-61db-4260-b8f0-f60d5cdd35cb" path="/var/lib/kubelet/pods/dc27c793-61db-4260-b8f0-f60d5cdd35cb/volumes" Jul 06 20:14:39.031037 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:39.030998 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:14:39.088273 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:39.088237 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:14:40.738126 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:40.738040 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:14:41.205154 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.205118 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:14:41.298264 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.298236 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7cbc9b8f4d-2v2gx"] Jul 06 20:14:41.298528 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.298515 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc27c793-61db-4260-b8f0-f60d5cdd35cb" containerName="manager" Jul 06 20:14:41.298528 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.298529 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc27c793-61db-4260-b8f0-f60d5cdd35cb" containerName="manager" Jul 06 20:14:41.298616 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.298582 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc27c793-61db-4260-b8f0-f60d5cdd35cb" containerName="manager" Jul 06 20:14:41.328185 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.328154 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7cbc9b8f4d-2v2gx"] Jul 06 20:14:41.328323 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.328271 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7cbc9b8f4d-2v2gx" Jul 06 20:14:41.330956 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.330939 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 06 20:14:41.331071 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.330965 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hsq27\"" Jul 06 20:14:41.333239 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.333218 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6196e12-44ff-40dc-a70c-8ce07fadb03e-maas-api-tls\") pod \"maas-api-7cbc9b8f4d-2v2gx\" (UID: \"b6196e12-44ff-40dc-a70c-8ce07fadb03e\") " pod="odh-ai-gateway-infra/maas-api-7cbc9b8f4d-2v2gx" Jul 06 20:14:41.333325 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.333263 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6196e12-44ff-40dc-a70c-8ce07fadb03e-openshift-service-ca\") pod \"maas-api-7cbc9b8f4d-2v2gx\" (UID: \"b6196e12-44ff-40dc-a70c-8ce07fadb03e\") " pod="odh-ai-gateway-infra/maas-api-7cbc9b8f4d-2v2gx" Jul 06 20:14:41.333325 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.333295 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6jgb\" (UniqueName: \"kubernetes.io/projected/b6196e12-44ff-40dc-a70c-8ce07fadb03e-kube-api-access-b6jgb\") pod \"maas-api-7cbc9b8f4d-2v2gx\" (UID: \"b6196e12-44ff-40dc-a70c-8ce07fadb03e\") " pod="odh-ai-gateway-infra/maas-api-7cbc9b8f4d-2v2gx" Jul 06 20:14:41.434758 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.434728 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6196e12-44ff-40dc-a70c-8ce07fadb03e-maas-api-tls\") pod \"maas-api-7cbc9b8f4d-2v2gx\" (UID: \"b6196e12-44ff-40dc-a70c-8ce07fadb03e\") " pod="odh-ai-gateway-infra/maas-api-7cbc9b8f4d-2v2gx" Jul 06 20:14:41.434758 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.434765 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6196e12-44ff-40dc-a70c-8ce07fadb03e-openshift-service-ca\") pod \"maas-api-7cbc9b8f4d-2v2gx\" (UID: \"b6196e12-44ff-40dc-a70c-8ce07fadb03e\") " pod="odh-ai-gateway-infra/maas-api-7cbc9b8f4d-2v2gx" Jul 06 20:14:41.435025 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.434783 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6jgb\" (UniqueName: \"kubernetes.io/projected/b6196e12-44ff-40dc-a70c-8ce07fadb03e-kube-api-access-b6jgb\") pod \"maas-api-7cbc9b8f4d-2v2gx\" (UID: \"b6196e12-44ff-40dc-a70c-8ce07fadb03e\") " pod="odh-ai-gateway-infra/maas-api-7cbc9b8f4d-2v2gx" Jul 06 20:14:41.435025 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:14:41.434884 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 06 20:14:41.435025 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:14:41.434984 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6196e12-44ff-40dc-a70c-8ce07fadb03e-maas-api-tls podName:b6196e12-44ff-40dc-a70c-8ce07fadb03e nodeName:}" failed. No retries permitted until 2026-07-06 20:14:41.934963567 +0000 UTC m=+533.146111780 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b6196e12-44ff-40dc-a70c-8ce07fadb03e-maas-api-tls") pod "maas-api-7cbc9b8f4d-2v2gx" (UID: "b6196e12-44ff-40dc-a70c-8ce07fadb03e") : secret "maas-api-serving-cert" not found Jul 06 20:14:41.435327 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.435308 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6196e12-44ff-40dc-a70c-8ce07fadb03e-openshift-service-ca\") pod \"maas-api-7cbc9b8f4d-2v2gx\" (UID: \"b6196e12-44ff-40dc-a70c-8ce07fadb03e\") " pod="odh-ai-gateway-infra/maas-api-7cbc9b8f4d-2v2gx" Jul 06 20:14:41.446064 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.446041 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6jgb\" (UniqueName: \"kubernetes.io/projected/b6196e12-44ff-40dc-a70c-8ce07fadb03e-kube-api-access-b6jgb\") pod \"maas-api-7cbc9b8f4d-2v2gx\" (UID: \"b6196e12-44ff-40dc-a70c-8ce07fadb03e\") " pod="odh-ai-gateway-infra/maas-api-7cbc9b8f4d-2v2gx" Jul 06 20:14:41.549489 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.549385 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:14:41.701324 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.701292 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:14:41.802134 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.802037 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:14:41.816538 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.816509 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-ptpvt"] Jul 06 20:14:41.818738 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.818723 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-ptpvt" Jul 06 20:14:41.821091 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.821068 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 06 20:14:41.830635 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.830613 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-ptpvt"] Jul 06 20:14:41.838426 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.838406 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mw4p\" (UniqueName: \"kubernetes.io/projected/f2d06033-e757-4936-bce6-3b23f1a033d7-kube-api-access-5mw4p\") pod \"payload-pre-processing-7964457696-ptpvt\" (UID: \"f2d06033-e757-4936-bce6-3b23f1a033d7\") " pod="openshift-ingress/payload-pre-processing-7964457696-ptpvt" Jul 06 20:14:41.838564 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.838483 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2d06033-e757-4936-bce6-3b23f1a033d7-plugins-config-volume\") pod \"payload-pre-processing-7964457696-ptpvt\" (UID: \"f2d06033-e757-4936-bce6-3b23f1a033d7\") " pod="openshift-ingress/payload-pre-processing-7964457696-ptpvt" Jul 06 20:14:41.848582 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.848558 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-h7qvz"] Jul 06 20:14:41.850795 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.850771 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-h7qvz" Jul 06 20:14:41.854006 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.853989 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-rz5s4\"" Jul 06 20:14:41.863090 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.863069 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-h7qvz"] Jul 06 20:14:41.939719 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.939673 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2d06033-e757-4936-bce6-3b23f1a033d7-plugins-config-volume\") pod \"payload-pre-processing-7964457696-ptpvt\" (UID: \"f2d06033-e757-4936-bce6-3b23f1a033d7\") " pod="openshift-ingress/payload-pre-processing-7964457696-ptpvt" Jul 06 20:14:41.939916 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.939761 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw45q\" (UniqueName: \"kubernetes.io/projected/d03334da-8cb5-45dd-a1d2-9076cbaba240-kube-api-access-jw45q\") pod \"payload-processing-58f86c7798-h7qvz\" (UID: \"d03334da-8cb5-45dd-a1d2-9076cbaba240\") " pod="openshift-ingress/payload-processing-58f86c7798-h7qvz" Jul 06 20:14:41.939916 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.939803 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d03334da-8cb5-45dd-a1d2-9076cbaba240-plugins-config-volume\") pod \"payload-processing-58f86c7798-h7qvz\" (UID: \"d03334da-8cb5-45dd-a1d2-9076cbaba240\") " pod="openshift-ingress/payload-processing-58f86c7798-h7qvz" Jul 06 20:14:41.939916 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.939853 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6196e12-44ff-40dc-a70c-8ce07fadb03e-maas-api-tls\") pod \"maas-api-7cbc9b8f4d-2v2gx\" (UID: \"b6196e12-44ff-40dc-a70c-8ce07fadb03e\") " pod="odh-ai-gateway-infra/maas-api-7cbc9b8f4d-2v2gx" Jul 06 20:14:41.939916 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.939880 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mw4p\" (UniqueName: \"kubernetes.io/projected/f2d06033-e757-4936-bce6-3b23f1a033d7-kube-api-access-5mw4p\") pod \"payload-pre-processing-7964457696-ptpvt\" (UID: \"f2d06033-e757-4936-bce6-3b23f1a033d7\") " pod="openshift-ingress/payload-pre-processing-7964457696-ptpvt" Jul 06 20:14:41.940569 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.940549 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2d06033-e757-4936-bce6-3b23f1a033d7-plugins-config-volume\") pod \"payload-pre-processing-7964457696-ptpvt\" (UID: \"f2d06033-e757-4936-bce6-3b23f1a033d7\") " pod="openshift-ingress/payload-pre-processing-7964457696-ptpvt" Jul 06 20:14:41.942609 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.942579 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6196e12-44ff-40dc-a70c-8ce07fadb03e-maas-api-tls\") pod \"maas-api-7cbc9b8f4d-2v2gx\" (UID: \"b6196e12-44ff-40dc-a70c-8ce07fadb03e\") " pod="odh-ai-gateway-infra/maas-api-7cbc9b8f4d-2v2gx" Jul 06 20:14:41.950161 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:41.950134 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mw4p\" (UniqueName: \"kubernetes.io/projected/f2d06033-e757-4936-bce6-3b23f1a033d7-kube-api-access-5mw4p\") pod \"payload-pre-processing-7964457696-ptpvt\" (UID: \"f2d06033-e757-4936-bce6-3b23f1a033d7\") " pod="openshift-ingress/payload-pre-processing-7964457696-ptpvt" Jul 06 20:14:42.040517 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:42.040479 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw45q\" (UniqueName: \"kubernetes.io/projected/d03334da-8cb5-45dd-a1d2-9076cbaba240-kube-api-access-jw45q\") pod \"payload-processing-58f86c7798-h7qvz\" (UID: \"d03334da-8cb5-45dd-a1d2-9076cbaba240\") " pod="openshift-ingress/payload-processing-58f86c7798-h7qvz" Jul 06 20:14:42.040517 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:42.040520 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d03334da-8cb5-45dd-a1d2-9076cbaba240-plugins-config-volume\") pod \"payload-processing-58f86c7798-h7qvz\" (UID: \"d03334da-8cb5-45dd-a1d2-9076cbaba240\") " pod="openshift-ingress/payload-processing-58f86c7798-h7qvz" Jul 06 20:14:42.041370 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:42.041345 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d03334da-8cb5-45dd-a1d2-9076cbaba240-plugins-config-volume\") pod \"payload-processing-58f86c7798-h7qvz\" (UID: \"d03334da-8cb5-45dd-a1d2-9076cbaba240\") " pod="openshift-ingress/payload-processing-58f86c7798-h7qvz" Jul 06 20:14:42.053187 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:42.053125 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw45q\" (UniqueName: \"kubernetes.io/projected/d03334da-8cb5-45dd-a1d2-9076cbaba240-kube-api-access-jw45q\") pod \"payload-processing-58f86c7798-h7qvz\" (UID: \"d03334da-8cb5-45dd-a1d2-9076cbaba240\") " pod="openshift-ingress/payload-processing-58f86c7798-h7qvz" Jul 06 20:14:42.129271 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:42.129235 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-ptpvt" Jul 06 20:14:42.160519 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:42.160486 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-h7qvz" Jul 06 20:14:42.239245 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:42.239055 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7cbc9b8f4d-2v2gx" Jul 06 20:14:42.275429 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:42.275201 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-ptpvt"] Jul 06 20:14:42.278006 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:14:42.277973 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d06033_e757_4936_bce6_3b23f1a033d7.slice/crio-d5b80ffd0d634e03c9a939f71493ddbc54b2b920b35d470e61da925dd01cde42 WatchSource:0}: Error finding container d5b80ffd0d634e03c9a939f71493ddbc54b2b920b35d470e61da925dd01cde42: Status 404 returned error can't find the container with id d5b80ffd0d634e03c9a939f71493ddbc54b2b920b35d470e61da925dd01cde42 Jul 06 20:14:42.300067 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:42.300038 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-ptpvt" event={"ID":"f2d06033-e757-4936-bce6-3b23f1a033d7","Type":"ContainerStarted","Data":"d5b80ffd0d634e03c9a939f71493ddbc54b2b920b35d470e61da925dd01cde42"} Jul 06 20:14:42.305441 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:42.305371 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-h7qvz"] Jul 06 20:14:42.308848 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:14:42.308818 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd03334da_8cb5_45dd_a1d2_9076cbaba240.slice/crio-5c4637fb2ef9e2c0da3688b1cce24dd3744bad95b304c67fd7a8456b883c52c4 WatchSource:0}: Error finding container 5c4637fb2ef9e2c0da3688b1cce24dd3744bad95b304c67fd7a8456b883c52c4: Status 404 returned error can't find the container with id 5c4637fb2ef9e2c0da3688b1cce24dd3744bad95b304c67fd7a8456b883c52c4 Jul 06 20:14:42.378982 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:42.378950 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7cbc9b8f4d-2v2gx"] Jul 06 20:14:42.381810 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:14:42.381786 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6196e12_44ff_40dc_a70c_8ce07fadb03e.slice/crio-5cd0a2695e845dc1a66dadb154549f467bdf578c54c71fd978fc2f4718d0cd77 WatchSource:0}: Error finding container 5cd0a2695e845dc1a66dadb154549f467bdf578c54c71fd978fc2f4718d0cd77: Status 404 returned error can't find the container with id 5cd0a2695e845dc1a66dadb154549f467bdf578c54c71fd978fc2f4718d0cd77 Jul 06 20:14:43.304940 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:43.304908 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-h7qvz" event={"ID":"d03334da-8cb5-45dd-a1d2-9076cbaba240","Type":"ContainerStarted","Data":"5c4637fb2ef9e2c0da3688b1cce24dd3744bad95b304c67fd7a8456b883c52c4"} Jul 06 20:14:43.306044 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:43.306019 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7cbc9b8f4d-2v2gx" event={"ID":"b6196e12-44ff-40dc-a70c-8ce07fadb03e","Type":"ContainerStarted","Data":"5cd0a2695e845dc1a66dadb154549f467bdf578c54c71fd978fc2f4718d0cd77"} Jul 06 20:14:44.313586 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:44.313366 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-ptpvt" event={"ID":"f2d06033-e757-4936-bce6-3b23f1a033d7","Type":"ContainerStarted","Data":"0546a401bda66edc07c6a3ab1a4d6ed0547719f1fd969012fee31a577984d1c1"} Jul 06 20:14:44.313586 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:44.313496 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-ptpvt" Jul 06 20:14:44.315219 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:44.315192 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-h7qvz" event={"ID":"d03334da-8cb5-45dd-a1d2-9076cbaba240","Type":"ContainerStarted","Data":"40a9c0c6ddae572a7c99ab2d1950361463a9590b526c4188030cfb54da75fe8b"} Jul 06 20:14:44.315379 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:44.315277 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-h7qvz" Jul 06 20:14:44.332577 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:44.332521 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-ptpvt" podStartSLOduration=1.87927425 podStartE2EDuration="3.332467193s" totalImagesPullingTime="1.453192943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:14:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:14:44.330375878 +0000 UTC m=+535.541524099" watchObservedRunningTime="2026-07-06 20:14:44.332467193 +0000 UTC m=+535.543615405" Jul 06 20:14:44.356138 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:44.356075 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-h7qvz" podStartSLOduration=1.930950205 podStartE2EDuration="3.3560527s" totalImagesPullingTime="1.425102495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:14:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:14:44.355201221 +0000 UTC m=+535.566349430" watchObservedRunningTime="2026-07-06 20:14:44.3560527 +0000 UTC m=+535.567200919" Jul 06 20:14:45.320475 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:45.320424 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7cbc9b8f4d-2v2gx" event={"ID":"b6196e12-44ff-40dc-a70c-8ce07fadb03e","Type":"ContainerStarted","Data":"a4bce7a9a6786d3e52dc5887743d4906c9bcea0a73320368d546ba27865a2a0a"} Jul 06 20:14:45.321040 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:45.320663 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7cbc9b8f4d-2v2gx" Jul 06 20:14:45.340304 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:45.340259 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7cbc9b8f4d-2v2gx" podStartSLOduration=2.042016201 podStartE2EDuration="4.340241376s" totalImagesPullingTime="2.298225175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:14:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:14:45.338310527 +0000 UTC m=+536.549458747" watchObservedRunningTime="2026-07-06 20:14:45.340241376 +0000 UTC m=+536.551389650" Jul 06 20:14:51.330236 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:51.330207 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7cbc9b8f4d-2v2gx" Jul 06 20:14:52.333416 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:52.333379 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:14:52.430283 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:52.430245 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:14:52.813177 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:52.813136 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:14:52.921078 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:52.921044 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:14:55.322629 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:55.322600 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-h7qvz" Jul 06 20:14:55.323012 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:14:55.322875 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-ptpvt" Jul 06 20:15:07.911614 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:07.911568 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76"] Jul 06 20:15:07.916051 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:07.916032 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76" Jul 06 20:15:07.919932 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:07.919896 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 20:15:07.919932 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:07.919912 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 06 20:15:07.919932 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:07.919896 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-66n6v\"" Jul 06 20:15:07.920193 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:07.919944 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 20:15:07.926879 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:07.926856 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76"] Jul 06 20:15:07.934707 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:07.934662 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:08.066161 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:08.066123 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/917e277b-543c-4010-8e59-a2fc6461ec76-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76\" (UID: \"917e277b-543c-4010-8e59-a2fc6461ec76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76" Jul 06 20:15:08.066349 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:08.066179 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/917e277b-543c-4010-8e59-a2fc6461ec76-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76\" (UID: \"917e277b-543c-4010-8e59-a2fc6461ec76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76" Jul 06 20:15:08.066349 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:08.066210 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/917e277b-543c-4010-8e59-a2fc6461ec76-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76\" (UID: \"917e277b-543c-4010-8e59-a2fc6461ec76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76" Jul 06 20:15:08.066349 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:08.066248 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/917e277b-543c-4010-8e59-a2fc6461ec76-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76\" (UID: \"917e277b-543c-4010-8e59-a2fc6461ec76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76" Jul 06 20:15:08.066349 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:08.066290 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/917e277b-543c-4010-8e59-a2fc6461ec76-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76\" (UID: \"917e277b-543c-4010-8e59-a2fc6461ec76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76" Jul 06 20:15:08.066349 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:08.066318 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzlt9\" (UniqueName: \"kubernetes.io/projected/917e277b-543c-4010-8e59-a2fc6461ec76-kube-api-access-rzlt9\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76\" (UID: \"917e277b-543c-4010-8e59-a2fc6461ec76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76" Jul 06 20:15:08.167240 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:08.167147 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/917e277b-543c-4010-8e59-a2fc6461ec76-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76\" (UID: \"917e277b-543c-4010-8e59-a2fc6461ec76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76" Jul 06 20:15:08.167240 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:08.167190 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/917e277b-543c-4010-8e59-a2fc6461ec76-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76\" (UID: \"917e277b-543c-4010-8e59-a2fc6461ec76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76" Jul 06 20:15:08.167240 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:08.167210 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/917e277b-543c-4010-8e59-a2fc6461ec76-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76\" (UID: \"917e277b-543c-4010-8e59-a2fc6461ec76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76" Jul 06 20:15:08.167240 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:08.167237 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/917e277b-543c-4010-8e59-a2fc6461ec76-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76\" (UID: \"917e277b-543c-4010-8e59-a2fc6461ec76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76" Jul 06 20:15:08.167512 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:08.167271 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/917e277b-543c-4010-8e59-a2fc6461ec76-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76\" (UID: \"917e277b-543c-4010-8e59-a2fc6461ec76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76" Jul 06 20:15:08.167512 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:08.167290 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzlt9\" (UniqueName: \"kubernetes.io/projected/917e277b-543c-4010-8e59-a2fc6461ec76-kube-api-access-rzlt9\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76\" (UID: \"917e277b-543c-4010-8e59-a2fc6461ec76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76" Jul 06 20:15:08.168028 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:08.167999 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/917e277b-543c-4010-8e59-a2fc6461ec76-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76\" (UID: \"917e277b-543c-4010-8e59-a2fc6461ec76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76" Jul 06 20:15:08.168182 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:08.168027 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/917e277b-543c-4010-8e59-a2fc6461ec76-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76\" (UID: \"917e277b-543c-4010-8e59-a2fc6461ec76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76" Jul 06 20:15:08.168269 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:08.168114 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/917e277b-543c-4010-8e59-a2fc6461ec76-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76\" (UID: \"917e277b-543c-4010-8e59-a2fc6461ec76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76" Jul 06 20:15:08.174045 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:08.174018 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/917e277b-543c-4010-8e59-a2fc6461ec76-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76\" (UID: \"917e277b-543c-4010-8e59-a2fc6461ec76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76" Jul 06 20:15:08.174361 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:08.174342 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/917e277b-543c-4010-8e59-a2fc6461ec76-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76\" (UID: \"917e277b-543c-4010-8e59-a2fc6461ec76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76" Jul 06 20:15:08.178663 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:08.178635 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzlt9\" (UniqueName: \"kubernetes.io/projected/917e277b-543c-4010-8e59-a2fc6461ec76-kube-api-access-rzlt9\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76\" (UID: \"917e277b-543c-4010-8e59-a2fc6461ec76\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76" Jul 06 20:15:08.227483 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:08.227419 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76" Jul 06 20:15:08.376931 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:08.376898 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76"] Jul 06 20:15:08.379904 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:15:08.379880 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod917e277b_543c_4010_8e59_a2fc6461ec76.slice/crio-610b6b40cfd8287278b46bef2871e8a0467a4078d15a96cf5bf12ffef1fc8b31 WatchSource:0}: Error finding container 610b6b40cfd8287278b46bef2871e8a0467a4078d15a96cf5bf12ffef1fc8b31: Status 404 returned error can't find the container with id 610b6b40cfd8287278b46bef2871e8a0467a4078d15a96cf5bf12ffef1fc8b31 Jul 06 20:15:08.413282 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:08.413252 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76" event={"ID":"917e277b-543c-4010-8e59-a2fc6461ec76","Type":"ContainerStarted","Data":"610b6b40cfd8287278b46bef2871e8a0467a4078d15a96cf5bf12ffef1fc8b31"} Jul 06 20:15:10.424450 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:10.424411 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76" event={"ID":"917e277b-543c-4010-8e59-a2fc6461ec76","Type":"ContainerStarted","Data":"c16563accc06c29728cad8bdcd0d4503608ae6f1d9e1910a67da4bfa2d27a409"} Jul 06 20:15:10.461567 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:10.461516 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76" podStartSLOduration=2.177559865 podStartE2EDuration="3.461496934s" totalImagesPullingTime="1.283937069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:15:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:15:10.460797932 +0000 UTC m=+561.671946151" watchObservedRunningTime="2026-07-06 20:15:10.461496934 +0000 UTC m=+561.672645151" Jul 06 20:15:16.325178 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:16.325125 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:18.228211 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:18.228175 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76" Jul 06 20:15:18.228633 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:18.228238 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76" Jul 06 20:15:18.240984 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:18.240956 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76" Jul 06 20:15:18.470302 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:18.470273 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76" Jul 06 20:15:26.239358 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:26.239318 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:34.210522 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:34.210494 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn"] Jul 06 20:15:34.213784 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:34.213765 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn" Jul 06 20:15:34.218685 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:34.218659 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 06 20:15:34.227366 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:34.227343 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn"] Jul 06 20:15:34.246071 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:34.246041 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:34.298642 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:34.298597 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4c33e1b-3286-4f90-92be-b2bcd94e3385-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn\" (UID: \"c4c33e1b-3286-4f90-92be-b2bcd94e3385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn" Jul 06 20:15:34.298642 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:34.298654 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4c33e1b-3286-4f90-92be-b2bcd94e3385-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn\" (UID: \"c4c33e1b-3286-4f90-92be-b2bcd94e3385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn" Jul 06 20:15:34.298918 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:34.298743 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4c33e1b-3286-4f90-92be-b2bcd94e3385-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn\" (UID: \"c4c33e1b-3286-4f90-92be-b2bcd94e3385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn" Jul 06 20:15:34.298918 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:34.298781 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4c33e1b-3286-4f90-92be-b2bcd94e3385-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn\" (UID: \"c4c33e1b-3286-4f90-92be-b2bcd94e3385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn" Jul 06 20:15:34.298918 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:34.298810 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4c33e1b-3286-4f90-92be-b2bcd94e3385-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn\" (UID: \"c4c33e1b-3286-4f90-92be-b2bcd94e3385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn" Jul 06 20:15:34.298918 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:34.298887 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k88h7\" (UniqueName: \"kubernetes.io/projected/c4c33e1b-3286-4f90-92be-b2bcd94e3385-kube-api-access-k88h7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn\" (UID: \"c4c33e1b-3286-4f90-92be-b2bcd94e3385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn" Jul 06 20:15:34.399635 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:34.399587 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k88h7\" (UniqueName: \"kubernetes.io/projected/c4c33e1b-3286-4f90-92be-b2bcd94e3385-kube-api-access-k88h7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn\" (UID: \"c4c33e1b-3286-4f90-92be-b2bcd94e3385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn" Jul 06 20:15:34.399635 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:34.399638 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4c33e1b-3286-4f90-92be-b2bcd94e3385-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn\" (UID: \"c4c33e1b-3286-4f90-92be-b2bcd94e3385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn" Jul 06 20:15:34.399887 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:34.399668 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4c33e1b-3286-4f90-92be-b2bcd94e3385-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn\" (UID: \"c4c33e1b-3286-4f90-92be-b2bcd94e3385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn" Jul 06 20:15:34.399887 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:34.399701 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4c33e1b-3286-4f90-92be-b2bcd94e3385-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn\" (UID: \"c4c33e1b-3286-4f90-92be-b2bcd94e3385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn" Jul 06 20:15:34.399887 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:34.399730 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4c33e1b-3286-4f90-92be-b2bcd94e3385-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn\" (UID: \"c4c33e1b-3286-4f90-92be-b2bcd94e3385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn" Jul 06 20:15:34.399887 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:34.399756 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4c33e1b-3286-4f90-92be-b2bcd94e3385-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn\" (UID: \"c4c33e1b-3286-4f90-92be-b2bcd94e3385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn" Jul 06 20:15:34.400085 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:34.400057 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4c33e1b-3286-4f90-92be-b2bcd94e3385-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn\" (UID: \"c4c33e1b-3286-4f90-92be-b2bcd94e3385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn" Jul 06 20:15:34.400221 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:34.400202 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4c33e1b-3286-4f90-92be-b2bcd94e3385-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn\" (UID: \"c4c33e1b-3286-4f90-92be-b2bcd94e3385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn" Jul 06 20:15:34.400221 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:34.400210 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4c33e1b-3286-4f90-92be-b2bcd94e3385-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn\" (UID: \"c4c33e1b-3286-4f90-92be-b2bcd94e3385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn" Jul 06 20:15:34.402176 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:34.402154 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4c33e1b-3286-4f90-92be-b2bcd94e3385-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn\" (UID: \"c4c33e1b-3286-4f90-92be-b2bcd94e3385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn" Jul 06 20:15:34.402346 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:34.402328 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4c33e1b-3286-4f90-92be-b2bcd94e3385-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn\" (UID: \"c4c33e1b-3286-4f90-92be-b2bcd94e3385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn" Jul 06 20:15:34.414143 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:34.414104 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k88h7\" (UniqueName: \"kubernetes.io/projected/c4c33e1b-3286-4f90-92be-b2bcd94e3385-kube-api-access-k88h7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn\" (UID: \"c4c33e1b-3286-4f90-92be-b2bcd94e3385\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn" Jul 06 20:15:34.524850 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:34.524761 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn" Jul 06 20:15:34.670046 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:34.670014 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn"] Jul 06 20:15:34.672964 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:15:34.672934 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4c33e1b_3286_4f90_92be_b2bcd94e3385.slice/crio-ac69e38e8dd706c498c5dc06dfd9a851ebf467d09637282c2d21766e6bea6192 WatchSource:0}: Error finding container ac69e38e8dd706c498c5dc06dfd9a851ebf467d09637282c2d21766e6bea6192: Status 404 returned error can't find the container with id ac69e38e8dd706c498c5dc06dfd9a851ebf467d09637282c2d21766e6bea6192 Jul 06 20:15:35.528213 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:35.528166 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn" event={"ID":"c4c33e1b-3286-4f90-92be-b2bcd94e3385","Type":"ContainerStarted","Data":"9ced7a4067ca45505ee4d87752ea02db0d961c6a6bb09977d4f9cc0f6f6b02d3"} Jul 06 20:15:35.528622 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:35.528228 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn" event={"ID":"c4c33e1b-3286-4f90-92be-b2bcd94e3385","Type":"ContainerStarted","Data":"ac69e38e8dd706c498c5dc06dfd9a851ebf467d09637282c2d21766e6bea6192"} Jul 06 20:15:35.550781 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:35.550719 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn" podStartSLOduration=1.329159086 podStartE2EDuration="1.550698229s" totalImagesPullingTime="221.539143ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:15:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:15:35.550187836 +0000 UTC m=+586.761336053" watchObservedRunningTime="2026-07-06 20:15:35.550698229 +0000 UTC m=+586.761846448" Jul 06 20:15:44.525642 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:44.525606 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn" Jul 06 20:15:44.526106 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:44.525674 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn" Jul 06 20:15:44.538368 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:44.538342 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn" Jul 06 20:15:44.577427 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:44.577401 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn" Jul 06 20:15:46.480725 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:46.480687 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:49.197184 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:49.197157 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dnfgp_0f4c3a8a-30c4-428e-88a6-dcf086ef8615/ovn-acl-logging/0.log" Jul 06 20:15:49.198578 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:15:49.198553 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dnfgp_0f4c3a8a-30c4-428e-88a6-dcf086ef8615/ovn-acl-logging/0.log" Jul 06 20:16:05.678684 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:16:05.678640 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:16:06.864584 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:16:06.864552 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:16:07.353101 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:16:07.353061 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:16:22.113825 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:16:22.113784 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:17:00.171899 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:17:00.171857 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-65954644c5-jk49n" podUID="d48c5414-9ad7-4052-be71-f44ca1b20a9c" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.21:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 20:17:08.811391 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:17:08.811350 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:17:08.908776 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:17:08.908744 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:17:27.320606 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:17:27.320556 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:17:27.413205 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:17:27.413156 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:17:27.950257 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:17:27.950222 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:17:28.527347 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:17:28.527318 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:17:28.628978 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:17:28.628941 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:18:41.905347 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:41.905312 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:18:42.046603 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:42.046569 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65954644c5-jk49n"] Jul 06 20:18:42.046856 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:42.046831 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-65954644c5-jk49n" podUID="d48c5414-9ad7-4052-be71-f44ca1b20a9c" containerName="manager" containerID="cri-o://693fc4db6338a960a6252347e5c666b4dce4f6551d27df7ea44d31dc97db9c8d" gracePeriod=30 Jul 06 20:18:42.295259 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:42.295230 2578 generic.go:356] "Generic (PLEG): container finished" podID="d48c5414-9ad7-4052-be71-f44ca1b20a9c" containerID="693fc4db6338a960a6252347e5c666b4dce4f6551d27df7ea44d31dc97db9c8d" exitCode=0 Jul 06 20:18:42.295402 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:42.295297 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65954644c5-jk49n" event={"ID":"d48c5414-9ad7-4052-be71-f44ca1b20a9c","Type":"ContainerDied","Data":"693fc4db6338a960a6252347e5c666b4dce4f6551d27df7ea44d31dc97db9c8d"} Jul 06 20:18:42.295402 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:42.295347 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65954644c5-jk49n" event={"ID":"d48c5414-9ad7-4052-be71-f44ca1b20a9c","Type":"ContainerDied","Data":"3021784ad68d837dde62030ea2069f83c42e7cdb4166f7bc43b3d9f19702b8bf"} Jul 06 20:18:42.295402 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:42.295364 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3021784ad68d837dde62030ea2069f83c42e7cdb4166f7bc43b3d9f19702b8bf" Jul 06 20:18:42.295595 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:42.295580 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65954644c5-jk49n" Jul 06 20:18:42.429549 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:42.429450 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d48c5414-9ad7-4052-be71-f44ca1b20a9c-webhook-cert\") pod \"d48c5414-9ad7-4052-be71-f44ca1b20a9c\" (UID: \"d48c5414-9ad7-4052-be71-f44ca1b20a9c\") " Jul 06 20:18:42.429713 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:42.429573 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sv2hz\" (UniqueName: \"kubernetes.io/projected/d48c5414-9ad7-4052-be71-f44ca1b20a9c-kube-api-access-sv2hz\") pod \"d48c5414-9ad7-4052-be71-f44ca1b20a9c\" (UID: \"d48c5414-9ad7-4052-be71-f44ca1b20a9c\") " Jul 06 20:18:42.431732 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:42.431710 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d48c5414-9ad7-4052-be71-f44ca1b20a9c-kube-api-access-sv2hz" (OuterVolumeSpecName: "kube-api-access-sv2hz") pod "d48c5414-9ad7-4052-be71-f44ca1b20a9c" (UID: "d48c5414-9ad7-4052-be71-f44ca1b20a9c"). InnerVolumeSpecName "kube-api-access-sv2hz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:18:42.431824 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:42.431749 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d48c5414-9ad7-4052-be71-f44ca1b20a9c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d48c5414-9ad7-4052-be71-f44ca1b20a9c" (UID: "d48c5414-9ad7-4052-be71-f44ca1b20a9c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:18:42.530932 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:42.530890 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv2hz\" (UniqueName: \"kubernetes.io/projected/d48c5414-9ad7-4052-be71-f44ca1b20a9c-kube-api-access-sv2hz\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:18:42.530932 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:42.530923 2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d48c5414-9ad7-4052-be71-f44ca1b20a9c-webhook-cert\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:18:43.299245 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:43.299209 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65954644c5-jk49n" Jul 06 20:18:43.338467 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:43.338418 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65954644c5-jk49n"] Jul 06 20:18:43.349664 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:43.349632 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-65954644c5-jk49n"] Jul 06 20:18:45.298234 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:45.298198 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d48c5414-9ad7-4052-be71-f44ca1b20a9c" path="/var/lib/kubelet/pods/d48c5414-9ad7-4052-be71-f44ca1b20a9c/volumes" Jul 06 20:18:51.411666 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:51.411626 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:18:57.092932 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:57.092898 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65954644c5-g5sgh"] Jul 06 20:18:57.093498 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:57.093413 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d48c5414-9ad7-4052-be71-f44ca1b20a9c" containerName="manager" Jul 06 20:18:57.093498 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:57.093432 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d48c5414-9ad7-4052-be71-f44ca1b20a9c" containerName="manager" Jul 06 20:18:57.093695 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:57.093533 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d48c5414-9ad7-4052-be71-f44ca1b20a9c" containerName="manager" Jul 06 20:18:57.096395 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:57.096372 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65954644c5-g5sgh" Jul 06 20:18:57.101080 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:57.101058 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 20:18:57.101219 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:57.101059 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7clct\"" Jul 06 20:18:57.101734 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:57.101718 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 20:18:57.154244 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:57.154211 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hll7z\" (UniqueName: \"kubernetes.io/projected/a886baf9-c25d-4a4e-8eb5-2a2b48234466-kube-api-access-hll7z\") pod \"maas-controller-65954644c5-g5sgh\" (UID: \"a886baf9-c25d-4a4e-8eb5-2a2b48234466\") " pod="opendatahub/maas-controller-65954644c5-g5sgh" Jul 06 20:18:57.154408 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:57.154257 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a886baf9-c25d-4a4e-8eb5-2a2b48234466-webhook-cert\") pod \"maas-controller-65954644c5-g5sgh\" (UID: \"a886baf9-c25d-4a4e-8eb5-2a2b48234466\") " pod="opendatahub/maas-controller-65954644c5-g5sgh" Jul 06 20:18:57.254946 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:57.254909 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hll7z\" (UniqueName: \"kubernetes.io/projected/a886baf9-c25d-4a4e-8eb5-2a2b48234466-kube-api-access-hll7z\") pod \"maas-controller-65954644c5-g5sgh\" (UID: \"a886baf9-c25d-4a4e-8eb5-2a2b48234466\") " pod="opendatahub/maas-controller-65954644c5-g5sgh" Jul 06 20:18:57.255143 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:57.254991 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a886baf9-c25d-4a4e-8eb5-2a2b48234466-webhook-cert\") pod \"maas-controller-65954644c5-g5sgh\" (UID: \"a886baf9-c25d-4a4e-8eb5-2a2b48234466\") " pod="opendatahub/maas-controller-65954644c5-g5sgh" Jul 06 20:18:57.255771 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:57.255695 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65954644c5-g5sgh"] Jul 06 20:18:57.258028 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:57.257996 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:18:57.258389 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:57.258370 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a886baf9-c25d-4a4e-8eb5-2a2b48234466-webhook-cert\") pod \"maas-controller-65954644c5-g5sgh\" (UID: \"a886baf9-c25d-4a4e-8eb5-2a2b48234466\") " pod="opendatahub/maas-controller-65954644c5-g5sgh" Jul 06 20:18:57.273319 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:57.273282 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hll7z\" (UniqueName: \"kubernetes.io/projected/a886baf9-c25d-4a4e-8eb5-2a2b48234466-kube-api-access-hll7z\") pod \"maas-controller-65954644c5-g5sgh\" (UID: \"a886baf9-c25d-4a4e-8eb5-2a2b48234466\") " pod="opendatahub/maas-controller-65954644c5-g5sgh" Jul 06 20:18:57.407201 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:57.407169 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65954644c5-g5sgh" Jul 06 20:18:57.565209 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:57.565183 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65954644c5-g5sgh"] Jul 06 20:18:57.567955 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:18:57.567914 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda886baf9_c25d_4a4e_8eb5_2a2b48234466.slice/crio-11d56777450b2228a1f938aba65af2c6d7deefd108491e1134e9c70c3f4254fe WatchSource:0}: Error finding container 11d56777450b2228a1f938aba65af2c6d7deefd108491e1134e9c70c3f4254fe: Status 404 returned error can't find the container with id 11d56777450b2228a1f938aba65af2c6d7deefd108491e1134e9c70c3f4254fe Jul 06 20:18:57.569421 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:57.569397 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 20:18:58.362527 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:58.362403 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65954644c5-g5sgh" event={"ID":"a886baf9-c25d-4a4e-8eb5-2a2b48234466","Type":"ContainerStarted","Data":"96ec7aa10bb51ad36992e4fd673247a9b3ea75103417a188651f27eb9c61965c"} Jul 06 20:18:58.362527 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:58.362483 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65954644c5-g5sgh" event={"ID":"a886baf9-c25d-4a4e-8eb5-2a2b48234466","Type":"ContainerStarted","Data":"11d56777450b2228a1f938aba65af2c6d7deefd108491e1134e9c70c3f4254fe"} Jul 06 20:18:58.362527 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:58.362508 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65954644c5-g5sgh" Jul 06 20:18:58.388804 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:58.388753 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-65954644c5-g5sgh" podStartSLOduration=6.883764918 podStartE2EDuration="7.388736357s" totalImagesPullingTime="504.971439ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:18:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:18:58.386623257 +0000 UTC m=+789.597771489" watchObservedRunningTime="2026-07-06 20:18:58.388736357 +0000 UTC m=+789.599884586" Jul 06 20:18:58.420589 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:18:58.420558 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:09.374940 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:19:09.374908 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65954644c5-g5sgh" Jul 06 20:19:13.253400 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:19:13.253358 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:13.321167 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:19:13.321133 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:21.911143 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:19:21.911106 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:22.021236 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:19:22.021200 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:22.119972 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:19:22.119936 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:29.396277 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:19:29.396213 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:29.524652 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:19:29.524617 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:29.653144 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:19:29.653042 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:29.968887 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:19:29.968801 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:37.219826 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:19:37.219792 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:37.317140 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:19:37.317100 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:20:49.223572 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:20:49.223491 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dnfgp_0f4c3a8a-30c4-428e-88a6-dcf086ef8615/ovn-acl-logging/0.log" Jul 06 20:20:49.224123 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:20:49.223778 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dnfgp_0f4c3a8a-30c4-428e-88a6-dcf086ef8615/ovn-acl-logging/0.log" Jul 06 20:20:49.243157 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:20:49.243135 2578 scope.go:117] "RemoveContainer" containerID="693fc4db6338a960a6252347e5c666b4dce4f6551d27df7ea44d31dc97db9c8d" Jul 06 20:22:51.038659 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:22:51.038624 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:24:01.958234 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:24:01.958184 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:24:02.018687 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:24:02.018652 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:24:35.403307 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:24:35.403272 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:25:49.248252 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:25:49.248224 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dnfgp_0f4c3a8a-30c4-428e-88a6-dcf086ef8615/ovn-acl-logging/0.log" Jul 06 20:25:49.249055 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:25:49.249037 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dnfgp_0f4c3a8a-30c4-428e-88a6-dcf086ef8615/ovn-acl-logging/0.log" Jul 06 20:26:00.321887 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:26:00.321852 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:28:50.411099 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:28:50.411050 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:30:23.202435 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:30:23.202403 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:30:23.272865 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:30:23.272836 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:30:29.945954 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:30:29.945915 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:30:40.368150 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:30:40.368113 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-65954644c5-g5sgh" podUID="a886baf9-c25d-4a4e-8eb5-2a2b48234466" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.27:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 20:30:45.316475 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:30:45.316424 2578 generic.go:356] "Generic (PLEG): container finished" podID="a886baf9-c25d-4a4e-8eb5-2a2b48234466" containerID="96ec7aa10bb51ad36992e4fd673247a9b3ea75103417a188651f27eb9c61965c" exitCode=1 Jul 06 20:30:45.316867 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:30:45.316502 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65954644c5-g5sgh" event={"ID":"a886baf9-c25d-4a4e-8eb5-2a2b48234466","Type":"ContainerDied","Data":"96ec7aa10bb51ad36992e4fd673247a9b3ea75103417a188651f27eb9c61965c"} Jul 06 20:30:45.316867 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:30:45.316825 2578 scope.go:117] "RemoveContainer" containerID="96ec7aa10bb51ad36992e4fd673247a9b3ea75103417a188651f27eb9c61965c" Jul 06 20:30:45.317159 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:30:45.317145 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 20:30:45.318115 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:30:45.318097 2578 generic.go:356] "Generic (PLEG): container finished" podID="0def7154-736a-4490-acf2-93ccb9ae1a35" containerID="d3473025931804c9503d25c40d6cad1555040f3f4d630fdd520f1d2b41274b21" exitCode=1 Jul 06 20:30:45.318180 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:30:45.318156 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-79c95fdbc9-rhh4g" event={"ID":"0def7154-736a-4490-acf2-93ccb9ae1a35","Type":"ContainerDied","Data":"d3473025931804c9503d25c40d6cad1555040f3f4d630fdd520f1d2b41274b21"} Jul 06 20:30:45.318443 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:30:45.318431 2578 scope.go:117] "RemoveContainer" containerID="d3473025931804c9503d25c40d6cad1555040f3f4d630fdd520f1d2b41274b21" Jul 06 20:30:45.872073 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:30:45.872050 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 20:30:46.023579 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:30:46.023532 2578 controller.go:195] "Failed to update lease" err="Put \"https://ab9f98892658242508f171bd05a809f2-b2773de990c4921d.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-131-218.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Jul 06 20:30:46.327423 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:30:46.327384 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-79c95fdbc9-rhh4g" event={"ID":"0def7154-736a-4490-acf2-93ccb9ae1a35","Type":"ContainerStarted","Data":"21b72c90a70eda69873f56cf748703db38c64a7f38ffaa1b60d32d298258dab4"} Jul 06 20:30:46.327894 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:30:46.327637 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-79c95fdbc9-rhh4g" Jul 06 20:30:46.329266 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:30:46.329243 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65954644c5-g5sgh" event={"ID":"a886baf9-c25d-4a4e-8eb5-2a2b48234466","Type":"ContainerStarted","Data":"084ab72114d7ff29561eb38a0d8a3b866d6c94f61ffdfb10152e306ab0a7fb98"} Jul 06 20:30:46.329428 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:30:46.329414 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65954644c5-g5sgh" Jul 06 20:30:47.518442 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:30:47.518409 2578 controller.go:195] "Failed to update lease" err="Operation cannot be fulfilled on leases.coordination.k8s.io \"ip-10-0-131-218.ec2.internal\": the object has been modified; please apply your changes to the latest version and try again" Jul 06 20:30:47.645900 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:30:47.645863 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:30:47.774043 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:30:47.773973 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:30:47.890176 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:30:47.890143 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:30:49.273234 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:30:49.273206 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dnfgp_0f4c3a8a-30c4-428e-88a6-dcf086ef8615/ovn-acl-logging/0.log" Jul 06 20:30:49.274374 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:30:49.274354 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dnfgp_0f4c3a8a-30c4-428e-88a6-dcf086ef8615/ovn-acl-logging/0.log" Jul 06 20:30:57.335071 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:30:57.335039 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-79c95fdbc9-rhh4g" Jul 06 20:30:57.348303 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:30:57.348272 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65954644c5-g5sgh" Jul 06 20:31:08.645601 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:08.645561 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:11.682010 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:11.681933 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:11.767305 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:11.767270 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:12.138024 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:12.137939 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:19.411372 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:19.411341 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:19.473377 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:19.473335 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:19.685299 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:19.685222 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:19.786159 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:19.786125 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:20.882342 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:20.882309 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:21.005732 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:21.005704 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v"] Jul 06 20:31:21.008969 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:21.008951 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v" Jul 06 20:31:21.011567 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:21.011544 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8a8b3310-serving-cert\"" Jul 06 20:31:21.012021 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:21.011991 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:21.033856 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:21.033823 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v"] Jul 06 20:31:21.104127 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:21.104090 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:21.183603 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:21.183568 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlx22\" (UniqueName: \"kubernetes.io/projected/4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2-kube-api-access-tlx22\") pod \"maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v\" (UID: \"4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v" Jul 06 20:31:21.183781 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:21.183638 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2-maas-api-tls\") pod \"maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v\" (UID: \"4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v" Jul 06 20:31:21.183781 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:21.183660 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2-openshift-service-ca\") pod \"maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v\" (UID: \"4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v" Jul 06 20:31:21.220648 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:21.220620 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:21.284823 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:21.284788 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2-maas-api-tls\") pod \"maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v\" (UID: \"4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v" Jul 06 20:31:21.284823 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:21.284825 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2-openshift-service-ca\") pod \"maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v\" (UID: \"4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v" Jul 06 20:31:21.285028 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:21.284873 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlx22\" (UniqueName: \"kubernetes.io/projected/4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2-kube-api-access-tlx22\") pod \"maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v\" (UID: \"4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v" Jul 06 20:31:21.285581 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:21.285561 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2-openshift-service-ca\") pod \"maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v\" (UID: \"4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v" Jul 06 20:31:21.287417 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:21.287395 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2-maas-api-tls\") pod \"maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v\" (UID: \"4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v" Jul 06 20:31:21.296131 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:21.296107 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlx22\" (UniqueName: \"kubernetes.io/projected/4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2-kube-api-access-tlx22\") pod \"maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v\" (UID: \"4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v" Jul 06 20:31:21.319549 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:21.319522 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v" Jul 06 20:31:21.461821 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:21.461798 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v"] Jul 06 20:31:21.463811 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:31:21.463784 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a8f8b40_7d30_40a2_87fb_9406cb8d5aa2.slice/crio-cf09f81de5b2ee996c8ad724cb19ccd7457f201e6a41d5ff84e856727576692e WatchSource:0}: Error finding container cf09f81de5b2ee996c8ad724cb19ccd7457f201e6a41d5ff84e856727576692e: Status 404 returned error can't find the container with id cf09f81de5b2ee996c8ad724cb19ccd7457f201e6a41d5ff84e856727576692e Jul 06 20:31:21.486086 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:21.486057 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v" event={"ID":"4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2","Type":"ContainerStarted","Data":"cf09f81de5b2ee996c8ad724cb19ccd7457f201e6a41d5ff84e856727576692e"} Jul 06 20:31:21.769050 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:21.768955 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:21.873326 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:21.873294 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:22.493331 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:22.493298 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v" event={"ID":"4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2","Type":"ContainerStarted","Data":"c4fd5506c2b61a62d8bbd3004d3abeeda366ae449ee9bdbb9944b802dbe3469c"} Jul 06 20:31:22.493740 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:22.493409 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v" Jul 06 20:31:22.519151 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:22.519083 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v" podStartSLOduration=2.046049125 podStartE2EDuration="2.519064025s" totalImagesPullingTime="473.0149ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:31:22.517160104 +0000 UTC m=+1533.728308344" watchObservedRunningTime="2026-07-06 20:31:22.519064025 +0000 UTC m=+1533.730212244" Jul 06 20:31:23.631146 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:23.631114 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:25.955369 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:25.955321 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:26.136758 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:26.136710 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:26.204279 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:26.204248 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:26.495919 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:26.495891 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v"] Jul 06 20:31:26.496119 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:26.496089 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v" podUID="4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2" containerName="maas-api" containerID="cri-o://c4fd5506c2b61a62d8bbd3004d3abeeda366ae449ee9bdbb9944b802dbe3469c" gracePeriod=30 Jul 06 20:31:26.501122 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:26.501104 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v" Jul 06 20:31:26.527280 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:26.527257 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:26.744372 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:26.744346 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v" Jul 06 20:31:26.832146 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:26.832062 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2-openshift-service-ca\") pod \"4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2\" (UID: \"4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2\") " Jul 06 20:31:26.832146 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:26.832108 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlx22\" (UniqueName: \"kubernetes.io/projected/4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2-kube-api-access-tlx22\") pod \"4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2\" (UID: \"4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2\") " Jul 06 20:31:26.832342 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:26.832161 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2-maas-api-tls\") pod \"4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2\" (UID: \"4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2\") " Jul 06 20:31:26.832402 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:26.832385 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2" (UID: "4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:31:26.834410 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:26.834383 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2-kube-api-access-tlx22" (OuterVolumeSpecName: "kube-api-access-tlx22") pod "4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2" (UID: "4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2"). InnerVolumeSpecName "kube-api-access-tlx22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:31:26.834532 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:26.834403 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2" (UID: "4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:31:26.933312 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:26.933274 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlx22\" (UniqueName: \"kubernetes.io/projected/4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2-kube-api-access-tlx22\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:31:26.933312 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:26.933306 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2-maas-api-tls\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:31:26.933312 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:26.933315 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2-openshift-service-ca\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:31:27.520889 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:27.520855 2578 generic.go:356] "Generic (PLEG): container finished" podID="4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2" containerID="c4fd5506c2b61a62d8bbd3004d3abeeda366ae449ee9bdbb9944b802dbe3469c" exitCode=0 Jul 06 20:31:27.521324 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:27.520902 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v" Jul 06 20:31:27.521324 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:27.520942 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v" event={"ID":"4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2","Type":"ContainerDied","Data":"c4fd5506c2b61a62d8bbd3004d3abeeda366ae449ee9bdbb9944b802dbe3469c"} Jul 06 20:31:27.521324 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:27.520998 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v" event={"ID":"4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2","Type":"ContainerDied","Data":"cf09f81de5b2ee996c8ad724cb19ccd7457f201e6a41d5ff84e856727576692e"} Jul 06 20:31:27.521324 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:27.521019 2578 scope.go:117] "RemoveContainer" containerID="c4fd5506c2b61a62d8bbd3004d3abeeda366ae449ee9bdbb9944b802dbe3469c" Jul 06 20:31:27.529193 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:27.529175 2578 scope.go:117] "RemoveContainer" containerID="c4fd5506c2b61a62d8bbd3004d3abeeda366ae449ee9bdbb9944b802dbe3469c" Jul 06 20:31:27.529490 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:31:27.529470 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4fd5506c2b61a62d8bbd3004d3abeeda366ae449ee9bdbb9944b802dbe3469c\": container with ID starting with c4fd5506c2b61a62d8bbd3004d3abeeda366ae449ee9bdbb9944b802dbe3469c not found: ID does not exist" containerID="c4fd5506c2b61a62d8bbd3004d3abeeda366ae449ee9bdbb9944b802dbe3469c" Jul 06 20:31:27.529535 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:27.529501 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4fd5506c2b61a62d8bbd3004d3abeeda366ae449ee9bdbb9944b802dbe3469c"} err="failed to get container status \"c4fd5506c2b61a62d8bbd3004d3abeeda366ae449ee9bdbb9944b802dbe3469c\": rpc error: code = NotFound desc = could not find container \"c4fd5506c2b61a62d8bbd3004d3abeeda366ae449ee9bdbb9944b802dbe3469c\": container with ID starting with c4fd5506c2b61a62d8bbd3004d3abeeda366ae449ee9bdbb9944b802dbe3469c not found: ID does not exist" Jul 06 20:31:27.543229 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:27.543202 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v"] Jul 06 20:31:27.554096 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:27.554073 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8a8b3310-6c4d555bc-k6j9v"] Jul 06 20:31:27.607090 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:27.607064 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:29.297597 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:29.297564 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2" path="/var/lib/kubelet/pods/4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2/volumes" Jul 06 20:31:33.410481 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:33.410418 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:33.472413 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:33.472359 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:33.685624 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:33.685596 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:33.751941 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:33.751913 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:34.860149 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:34.860119 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:34.960032 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:34.960003 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:34.960868 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:34.960845 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-915683a6-5877487f76-pzzw6"] Jul 06 20:31:34.961226 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:34.961211 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2" containerName="maas-api" Jul 06 20:31:34.961263 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:34.961230 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2" containerName="maas-api" Jul 06 20:31:34.961295 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:34.961285 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a8f8b40-7d30-40a2-87fb-9406cb8d5aa2" containerName="maas-api" Jul 06 20:31:34.965394 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:34.965374 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-915683a6-5877487f76-pzzw6" Jul 06 20:31:34.968153 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:34.968132 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-915683a6-serving-cert\"" Jul 06 20:31:34.974962 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:34.974935 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-915683a6-5877487f76-pzzw6"] Jul 06 20:31:35.003856 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:35.003828 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st5tv\" (UniqueName: \"kubernetes.io/projected/ec296d0d-fbd3-4b90-93dc-931a92675f24-kube-api-access-st5tv\") pod \"maas-api-e2e-ait-915683a6-5877487f76-pzzw6\" (UID: \"ec296d0d-fbd3-4b90-93dc-931a92675f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-915683a6-5877487f76-pzzw6" Jul 06 20:31:35.004040 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:35.003885 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec296d0d-fbd3-4b90-93dc-931a92675f24-openshift-service-ca\") pod \"maas-api-e2e-ait-915683a6-5877487f76-pzzw6\" (UID: \"ec296d0d-fbd3-4b90-93dc-931a92675f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-915683a6-5877487f76-pzzw6" Jul 06 20:31:35.004040 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:35.003935 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec296d0d-fbd3-4b90-93dc-931a92675f24-maas-api-tls\") pod \"maas-api-e2e-ait-915683a6-5877487f76-pzzw6\" (UID: \"ec296d0d-fbd3-4b90-93dc-931a92675f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-915683a6-5877487f76-pzzw6" Jul 06 20:31:35.069299 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:35.069265 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:35.104766 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:35.104736 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st5tv\" (UniqueName: \"kubernetes.io/projected/ec296d0d-fbd3-4b90-93dc-931a92675f24-kube-api-access-st5tv\") pod \"maas-api-e2e-ait-915683a6-5877487f76-pzzw6\" (UID: \"ec296d0d-fbd3-4b90-93dc-931a92675f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-915683a6-5877487f76-pzzw6" Jul 06 20:31:35.104952 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:35.104785 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec296d0d-fbd3-4b90-93dc-931a92675f24-openshift-service-ca\") pod \"maas-api-e2e-ait-915683a6-5877487f76-pzzw6\" (UID: \"ec296d0d-fbd3-4b90-93dc-931a92675f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-915683a6-5877487f76-pzzw6" Jul 06 20:31:35.104952 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:35.104814 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec296d0d-fbd3-4b90-93dc-931a92675f24-maas-api-tls\") pod \"maas-api-e2e-ait-915683a6-5877487f76-pzzw6\" (UID: \"ec296d0d-fbd3-4b90-93dc-931a92675f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-915683a6-5877487f76-pzzw6" Jul 06 20:31:35.104952 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:31:35.104928 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-915683a6-serving-cert: secret "maas-api-e2e-ait-915683a6-serving-cert" not found Jul 06 20:31:35.105127 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:31:35.104992 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec296d0d-fbd3-4b90-93dc-931a92675f24-maas-api-tls podName:ec296d0d-fbd3-4b90-93dc-931a92675f24 nodeName:}" failed. No retries permitted until 2026-07-06 20:31:35.604975202 +0000 UTC m=+1546.816123398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ec296d0d-fbd3-4b90-93dc-931a92675f24-maas-api-tls") pod "maas-api-e2e-ait-915683a6-5877487f76-pzzw6" (UID: "ec296d0d-fbd3-4b90-93dc-931a92675f24") : secret "maas-api-e2e-ait-915683a6-serving-cert" not found Jul 06 20:31:35.105568 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:35.105546 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec296d0d-fbd3-4b90-93dc-931a92675f24-openshift-service-ca\") pod \"maas-api-e2e-ait-915683a6-5877487f76-pzzw6\" (UID: \"ec296d0d-fbd3-4b90-93dc-931a92675f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-915683a6-5877487f76-pzzw6" Jul 06 20:31:35.123653 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:35.123594 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st5tv\" (UniqueName: \"kubernetes.io/projected/ec296d0d-fbd3-4b90-93dc-931a92675f24-kube-api-access-st5tv\") pod \"maas-api-e2e-ait-915683a6-5877487f76-pzzw6\" (UID: \"ec296d0d-fbd3-4b90-93dc-931a92675f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-915683a6-5877487f76-pzzw6" Jul 06 20:31:35.349336 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:35.349299 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:35.607761 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:35.607726 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec296d0d-fbd3-4b90-93dc-931a92675f24-maas-api-tls\") pod \"maas-api-e2e-ait-915683a6-5877487f76-pzzw6\" (UID: \"ec296d0d-fbd3-4b90-93dc-931a92675f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-915683a6-5877487f76-pzzw6" Jul 06 20:31:35.610443 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:35.610410 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec296d0d-fbd3-4b90-93dc-931a92675f24-maas-api-tls\") pod \"maas-api-e2e-ait-915683a6-5877487f76-pzzw6\" (UID: \"ec296d0d-fbd3-4b90-93dc-931a92675f24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-915683a6-5877487f76-pzzw6" Jul 06 20:31:35.795483 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:35.795423 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:35.876583 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:35.876503 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-915683a6-5877487f76-pzzw6" Jul 06 20:31:35.892144 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:35.892090 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:36.017556 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:36.017528 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-915683a6-5877487f76-pzzw6"] Jul 06 20:31:36.019632 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:31:36.019603 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec296d0d_fbd3_4b90_93dc_931a92675f24.slice/crio-c838c7d108774f19b8527378d828865f82c4513562d83f52608e654ff2cda7c1 WatchSource:0}: Error finding container c838c7d108774f19b8527378d828865f82c4513562d83f52608e654ff2cda7c1: Status 404 returned error can't find the container with id c838c7d108774f19b8527378d828865f82c4513562d83f52608e654ff2cda7c1 Jul 06 20:31:36.564548 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:36.564508 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-915683a6-5877487f76-pzzw6" event={"ID":"ec296d0d-fbd3-4b90-93dc-931a92675f24","Type":"ContainerStarted","Data":"3b5947f17d753c932c0418621e76614024713cfe6eaccd5a4fe7fa9e8631b768"} Jul 06 20:31:36.564747 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:36.564575 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-915683a6-5877487f76-pzzw6" event={"ID":"ec296d0d-fbd3-4b90-93dc-931a92675f24","Type":"ContainerStarted","Data":"c838c7d108774f19b8527378d828865f82c4513562d83f52608e654ff2cda7c1"} Jul 06 20:31:36.564747 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:36.564649 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-915683a6-5877487f76-pzzw6" Jul 06 20:31:36.609643 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:36.609593 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-915683a6-5877487f76-pzzw6" podStartSLOduration=2.189353628 podStartE2EDuration="2.60957486s" totalImagesPullingTime="420.221232ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:31:36.608915141 +0000 UTC m=+1547.820063359" watchObservedRunningTime="2026-07-06 20:31:36.60957486 +0000 UTC m=+1547.820723077" Jul 06 20:31:37.048040 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:37.048007 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:39.916165 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:39.916124 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:40.046561 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:40.046523 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:40.308026 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:40.307950 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-915683a6-5877487f76-pzzw6"] Jul 06 20:31:40.308274 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:40.308241 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-915683a6-5877487f76-pzzw6" podUID="ec296d0d-fbd3-4b90-93dc-931a92675f24" containerName="maas-api" containerID="cri-o://3b5947f17d753c932c0418621e76614024713cfe6eaccd5a4fe7fa9e8631b768" gracePeriod=30 Jul 06 20:31:40.329257 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:40.329229 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:40.557812 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:40.557791 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-915683a6-5877487f76-pzzw6" Jul 06 20:31:40.583255 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:40.583182 2578 generic.go:356] "Generic (PLEG): container finished" podID="ec296d0d-fbd3-4b90-93dc-931a92675f24" containerID="3b5947f17d753c932c0418621e76614024713cfe6eaccd5a4fe7fa9e8631b768" exitCode=0 Jul 06 20:31:40.583255 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:40.583227 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-915683a6-5877487f76-pzzw6" Jul 06 20:31:40.583405 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:40.583257 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-915683a6-5877487f76-pzzw6" event={"ID":"ec296d0d-fbd3-4b90-93dc-931a92675f24","Type":"ContainerDied","Data":"3b5947f17d753c932c0418621e76614024713cfe6eaccd5a4fe7fa9e8631b768"} Jul 06 20:31:40.583405 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:40.583308 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-915683a6-5877487f76-pzzw6" event={"ID":"ec296d0d-fbd3-4b90-93dc-931a92675f24","Type":"ContainerDied","Data":"c838c7d108774f19b8527378d828865f82c4513562d83f52608e654ff2cda7c1"} Jul 06 20:31:40.583405 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:40.583331 2578 scope.go:117] "RemoveContainer" containerID="3b5947f17d753c932c0418621e76614024713cfe6eaccd5a4fe7fa9e8631b768" Jul 06 20:31:40.592628 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:40.592601 2578 scope.go:117] "RemoveContainer" containerID="3b5947f17d753c932c0418621e76614024713cfe6eaccd5a4fe7fa9e8631b768" Jul 06 20:31:40.592877 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:31:40.592857 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b5947f17d753c932c0418621e76614024713cfe6eaccd5a4fe7fa9e8631b768\": container with ID starting with 3b5947f17d753c932c0418621e76614024713cfe6eaccd5a4fe7fa9e8631b768 not found: ID does not exist" containerID="3b5947f17d753c932c0418621e76614024713cfe6eaccd5a4fe7fa9e8631b768" Jul 06 20:31:40.592937 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:40.592888 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b5947f17d753c932c0418621e76614024713cfe6eaccd5a4fe7fa9e8631b768"} err="failed to get container status \"3b5947f17d753c932c0418621e76614024713cfe6eaccd5a4fe7fa9e8631b768\": rpc error: code = NotFound desc = could not find container \"3b5947f17d753c932c0418621e76614024713cfe6eaccd5a4fe7fa9e8631b768\": container with ID starting with 3b5947f17d753c932c0418621e76614024713cfe6eaccd5a4fe7fa9e8631b768 not found: ID does not exist" Jul 06 20:31:40.650542 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:40.650518 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-st5tv\" (UniqueName: \"kubernetes.io/projected/ec296d0d-fbd3-4b90-93dc-931a92675f24-kube-api-access-st5tv\") pod \"ec296d0d-fbd3-4b90-93dc-931a92675f24\" (UID: \"ec296d0d-fbd3-4b90-93dc-931a92675f24\") " Jul 06 20:31:40.650649 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:40.650566 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec296d0d-fbd3-4b90-93dc-931a92675f24-maas-api-tls\") pod \"ec296d0d-fbd3-4b90-93dc-931a92675f24\" (UID: \"ec296d0d-fbd3-4b90-93dc-931a92675f24\") " Jul 06 20:31:40.650649 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:40.650627 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec296d0d-fbd3-4b90-93dc-931a92675f24-openshift-service-ca\") pod \"ec296d0d-fbd3-4b90-93dc-931a92675f24\" (UID: \"ec296d0d-fbd3-4b90-93dc-931a92675f24\") " Jul 06 20:31:40.651018 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:40.650990 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec296d0d-fbd3-4b90-93dc-931a92675f24-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ec296d0d-fbd3-4b90-93dc-931a92675f24" (UID: "ec296d0d-fbd3-4b90-93dc-931a92675f24"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:31:40.652692 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:40.652671 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec296d0d-fbd3-4b90-93dc-931a92675f24-kube-api-access-st5tv" (OuterVolumeSpecName: "kube-api-access-st5tv") pod "ec296d0d-fbd3-4b90-93dc-931a92675f24" (UID: "ec296d0d-fbd3-4b90-93dc-931a92675f24"). InnerVolumeSpecName "kube-api-access-st5tv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:31:40.652785 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:40.652715 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec296d0d-fbd3-4b90-93dc-931a92675f24-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ec296d0d-fbd3-4b90-93dc-931a92675f24" (UID: "ec296d0d-fbd3-4b90-93dc-931a92675f24"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:31:40.751792 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:40.751759 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec296d0d-fbd3-4b90-93dc-931a92675f24-openshift-service-ca\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:31:40.751792 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:40.751787 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st5tv\" (UniqueName: \"kubernetes.io/projected/ec296d0d-fbd3-4b90-93dc-931a92675f24-kube-api-access-st5tv\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:31:40.751792 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:40.751799 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec296d0d-fbd3-4b90-93dc-931a92675f24-maas-api-tls\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:31:40.913208 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:40.913173 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-915683a6-5877487f76-pzzw6"] Jul 06 20:31:40.918631 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:40.918604 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-915683a6-5877487f76-pzzw6"] Jul 06 20:31:41.298478 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:41.298422 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec296d0d-fbd3-4b90-93dc-931a92675f24" path="/var/lib/kubelet/pods/ec296d0d-fbd3-4b90-93dc-931a92675f24/volumes" Jul 06 20:31:46.018204 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:46.018170 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:46.121002 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:46.120975 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:48.288219 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:48.288191 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:53.836876 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:53.836834 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:53.926067 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:53.926032 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:54.160914 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:54.160883 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:54.257054 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:54.257011 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:55.312244 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:55.312214 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:55.423605 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:55.423565 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:55.427610 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:55.427584 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd"] Jul 06 20:31:55.427950 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:55.427932 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec296d0d-fbd3-4b90-93dc-931a92675f24" containerName="maas-api" Jul 06 20:31:55.427950 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:55.427950 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec296d0d-fbd3-4b90-93dc-931a92675f24" containerName="maas-api" Jul 06 20:31:55.428088 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:55.428003 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec296d0d-fbd3-4b90-93dc-931a92675f24" containerName="maas-api" Jul 06 20:31:55.432027 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:55.432011 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd" Jul 06 20:31:55.437760 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:55.437738 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-2c6e75a8-serving-cert\"" Jul 06 20:31:55.453323 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:55.453294 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd"] Jul 06 20:31:55.529153 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:55.529121 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:55.580207 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:55.580123 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7fa2628-3a17-4013-8876-b15e71ec6131-openshift-service-ca\") pod \"maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd\" (UID: \"f7fa2628-3a17-4013-8876-b15e71ec6131\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd" Jul 06 20:31:55.580207 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:55.580181 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdw9p\" (UniqueName: \"kubernetes.io/projected/f7fa2628-3a17-4013-8876-b15e71ec6131-kube-api-access-rdw9p\") pod \"maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd\" (UID: \"f7fa2628-3a17-4013-8876-b15e71ec6131\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd" Jul 06 20:31:55.580426 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:55.580254 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7fa2628-3a17-4013-8876-b15e71ec6131-maas-api-tls\") pod \"maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd\" (UID: \"f7fa2628-3a17-4013-8876-b15e71ec6131\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd" Jul 06 20:31:55.660210 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:55.660178 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:55.681406 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:55.681376 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7fa2628-3a17-4013-8876-b15e71ec6131-openshift-service-ca\") pod \"maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd\" (UID: \"f7fa2628-3a17-4013-8876-b15e71ec6131\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd" Jul 06 20:31:55.681564 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:55.681442 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdw9p\" (UniqueName: \"kubernetes.io/projected/f7fa2628-3a17-4013-8876-b15e71ec6131-kube-api-access-rdw9p\") pod \"maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd\" (UID: \"f7fa2628-3a17-4013-8876-b15e71ec6131\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd" Jul 06 20:31:55.681564 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:55.681510 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7fa2628-3a17-4013-8876-b15e71ec6131-maas-api-tls\") pod \"maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd\" (UID: \"f7fa2628-3a17-4013-8876-b15e71ec6131\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd" Jul 06 20:31:55.682095 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:55.682073 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7fa2628-3a17-4013-8876-b15e71ec6131-openshift-service-ca\") pod \"maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd\" (UID: \"f7fa2628-3a17-4013-8876-b15e71ec6131\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd" Jul 06 20:31:55.683979 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:55.683958 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7fa2628-3a17-4013-8876-b15e71ec6131-maas-api-tls\") pod \"maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd\" (UID: \"f7fa2628-3a17-4013-8876-b15e71ec6131\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd" Jul 06 20:31:55.709859 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:55.709832 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdw9p\" (UniqueName: \"kubernetes.io/projected/f7fa2628-3a17-4013-8876-b15e71ec6131-kube-api-access-rdw9p\") pod \"maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd\" (UID: \"f7fa2628-3a17-4013-8876-b15e71ec6131\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd" Jul 06 20:31:55.741725 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:55.741694 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd" Jul 06 20:31:55.891260 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:31:55.891223 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7fa2628_3a17_4013_8876_b15e71ec6131.slice/crio-1b0c2f32519edba084a284464d59f39fa1dbe043217a43dc9d1491a15d838243 WatchSource:0}: Error finding container 1b0c2f32519edba084a284464d59f39fa1dbe043217a43dc9d1491a15d838243: Status 404 returned error can't find the container with id 1b0c2f32519edba084a284464d59f39fa1dbe043217a43dc9d1491a15d838243 Jul 06 20:31:55.894384 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:55.894363 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd"] Jul 06 20:31:56.357881 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:56.357845 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:56.492930 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:56.492894 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:56.651231 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:56.651188 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd" event={"ID":"f7fa2628-3a17-4013-8876-b15e71ec6131","Type":"ContainerStarted","Data":"303bd30e876a2c7131ef599581c0ecf6997a32e194356fb6b32a05815dba377c"} Jul 06 20:31:56.651410 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:56.651249 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd" event={"ID":"f7fa2628-3a17-4013-8876-b15e71ec6131","Type":"ContainerStarted","Data":"1b0c2f32519edba084a284464d59f39fa1dbe043217a43dc9d1491a15d838243"} Jul 06 20:31:56.651410 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:56.651319 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd" Jul 06 20:31:56.689368 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:56.689313 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd" podStartSLOduration=1.2834396080000001 podStartE2EDuration="1.689295125s" totalImagesPullingTime="405.855517ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:31:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:31:56.687868561 +0000 UTC m=+1567.899016780" watchObservedRunningTime="2026-07-06 20:31:56.689295125 +0000 UTC m=+1567.900443343" Jul 06 20:31:57.546777 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:57.546750 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:59.777106 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:59.777056 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:59.957964 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:31:59.957921 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:32:00.087445 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:00.087342 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:32:00.300184 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:00.300147 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd"] Jul 06 20:32:00.300418 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:00.300392 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd" podUID="f7fa2628-3a17-4013-8876-b15e71ec6131" containerName="maas-api" containerID="cri-o://303bd30e876a2c7131ef599581c0ecf6997a32e194356fb6b32a05815dba377c" gracePeriod=30 Jul 06 20:32:00.331389 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:00.331353 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:32:00.548984 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:00.548961 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd" Jul 06 20:32:00.626024 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:00.625984 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7fa2628-3a17-4013-8876-b15e71ec6131-maas-api-tls\") pod \"f7fa2628-3a17-4013-8876-b15e71ec6131\" (UID: \"f7fa2628-3a17-4013-8876-b15e71ec6131\") " Jul 06 20:32:00.626207 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:00.626056 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7fa2628-3a17-4013-8876-b15e71ec6131-openshift-service-ca\") pod \"f7fa2628-3a17-4013-8876-b15e71ec6131\" (UID: \"f7fa2628-3a17-4013-8876-b15e71ec6131\") " Jul 06 20:32:00.626207 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:00.626105 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdw9p\" (UniqueName: \"kubernetes.io/projected/f7fa2628-3a17-4013-8876-b15e71ec6131-kube-api-access-rdw9p\") pod \"f7fa2628-3a17-4013-8876-b15e71ec6131\" (UID: \"f7fa2628-3a17-4013-8876-b15e71ec6131\") " Jul 06 20:32:00.626416 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:00.626393 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7fa2628-3a17-4013-8876-b15e71ec6131-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f7fa2628-3a17-4013-8876-b15e71ec6131" (UID: "f7fa2628-3a17-4013-8876-b15e71ec6131"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:32:00.628327 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:00.628298 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7fa2628-3a17-4013-8876-b15e71ec6131-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f7fa2628-3a17-4013-8876-b15e71ec6131" (UID: "f7fa2628-3a17-4013-8876-b15e71ec6131"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:32:00.628413 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:00.628344 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7fa2628-3a17-4013-8876-b15e71ec6131-kube-api-access-rdw9p" (OuterVolumeSpecName: "kube-api-access-rdw9p") pod "f7fa2628-3a17-4013-8876-b15e71ec6131" (UID: "f7fa2628-3a17-4013-8876-b15e71ec6131"). InnerVolumeSpecName "kube-api-access-rdw9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:32:00.668010 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:00.667928 2578 generic.go:356] "Generic (PLEG): container finished" podID="f7fa2628-3a17-4013-8876-b15e71ec6131" containerID="303bd30e876a2c7131ef599581c0ecf6997a32e194356fb6b32a05815dba377c" exitCode=0 Jul 06 20:32:00.668010 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:00.667967 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd" Jul 06 20:32:00.668223 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:00.667970 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd" event={"ID":"f7fa2628-3a17-4013-8876-b15e71ec6131","Type":"ContainerDied","Data":"303bd30e876a2c7131ef599581c0ecf6997a32e194356fb6b32a05815dba377c"} Jul 06 20:32:00.668223 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:00.668120 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd" event={"ID":"f7fa2628-3a17-4013-8876-b15e71ec6131","Type":"ContainerDied","Data":"1b0c2f32519edba084a284464d59f39fa1dbe043217a43dc9d1491a15d838243"} Jul 06 20:32:00.668223 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:00.668156 2578 scope.go:117] "RemoveContainer" containerID="303bd30e876a2c7131ef599581c0ecf6997a32e194356fb6b32a05815dba377c" Jul 06 20:32:00.677424 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:00.677409 2578 scope.go:117] "RemoveContainer" containerID="303bd30e876a2c7131ef599581c0ecf6997a32e194356fb6b32a05815dba377c" Jul 06 20:32:00.677718 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:32:00.677697 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"303bd30e876a2c7131ef599581c0ecf6997a32e194356fb6b32a05815dba377c\": container with ID starting with 303bd30e876a2c7131ef599581c0ecf6997a32e194356fb6b32a05815dba377c not found: ID does not exist" containerID="303bd30e876a2c7131ef599581c0ecf6997a32e194356fb6b32a05815dba377c" Jul 06 20:32:00.677768 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:00.677726 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"303bd30e876a2c7131ef599581c0ecf6997a32e194356fb6b32a05815dba377c"} err="failed to get container status \"303bd30e876a2c7131ef599581c0ecf6997a32e194356fb6b32a05815dba377c\": rpc error: code = NotFound desc = could not find container \"303bd30e876a2c7131ef599581c0ecf6997a32e194356fb6b32a05815dba377c\": container with ID starting with 303bd30e876a2c7131ef599581c0ecf6997a32e194356fb6b32a05815dba377c not found: ID does not exist" Jul 06 20:32:00.693831 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:00.693808 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd"] Jul 06 20:32:00.697116 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:00.697094 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2c6e75a8-6d767c9c85-pz2hd"] Jul 06 20:32:00.727008 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:00.726980 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7fa2628-3a17-4013-8876-b15e71ec6131-maas-api-tls\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:32:00.727008 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:00.727007 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7fa2628-3a17-4013-8876-b15e71ec6131-openshift-service-ca\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:32:00.727167 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:00.727017 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdw9p\" (UniqueName: \"kubernetes.io/projected/f7fa2628-3a17-4013-8876-b15e71ec6131-kube-api-access-rdw9p\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:32:01.298080 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:01.298047 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7fa2628-3a17-4013-8876-b15e71ec6131" path="/var/lib/kubelet/pods/f7fa2628-3a17-4013-8876-b15e71ec6131/volumes" Jul 06 20:32:01.457548 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:01.457508 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:32:06.609166 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:06.609130 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:32:06.691954 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:06.691910 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:32:07.108358 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:07.108299 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:32:07.216073 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:07.216042 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:32:07.310676 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:07.310638 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:32:08.855364 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:08.855305 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:32:15.296707 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:15.296677 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:32:15.359155 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:15.359127 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:32:15.431908 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:15.431867 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:32:15.532333 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:15.532295 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:32:15.640348 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:15.640266 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:32:16.075896 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:16.075860 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:32:42.102148 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:42.102048 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:32:49.391041 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:49.391003 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:32:49.450257 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:49.450224 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:32:49.542820 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:49.542793 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:32:49.634498 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:32:49.634395 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:33:05.568364 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:05.568333 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:33:12.253691 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:12.253657 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:33:12.316660 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:12.316613 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:33:12.419693 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:12.419662 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:33:12.534096 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:12.534011 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:33:23.042903 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:23.042862 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:33:23.873278 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:23.873244 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:33:30.067713 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:30.067673 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:33:30.174003 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:30.173969 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:33:30.274079 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:30.274044 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:33:30.368375 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:30.368294 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:33:31.194705 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:31.194673 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:33:31.293428 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:31.293393 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:33:31.391336 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:31.391299 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:33:33.031186 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:33.031157 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:33:39.496214 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:39.496184 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:33:39.593027 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:39.592995 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:33:40.152087 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:40.152056 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:33:47.722174 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:47.722141 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:33:47.786998 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:47.786967 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:33:47.960027 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:47.959963 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:33:48.306102 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:48.306068 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:33:48.418040 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:48.418008 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:33:51.217904 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:51.217860 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:33:51.287110 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:51.287071 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf"] Jul 06 20:33:51.287293 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:51.287067 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:33:51.287470 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:51.287437 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7fa2628-3a17-4013-8876-b15e71ec6131" containerName="maas-api" Jul 06 20:33:51.287470 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:51.287468 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7fa2628-3a17-4013-8876-b15e71ec6131" containerName="maas-api" Jul 06 20:33:51.287595 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:51.287535 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7fa2628-3a17-4013-8876-b15e71ec6131" containerName="maas-api" Jul 06 20:33:51.290425 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:51.290405 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf" Jul 06 20:33:51.292793 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:51.292771 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a8dc09e7-serving-cert\"" Jul 06 20:33:51.295777 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:51.295750 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a57d3db-4264-46dd-a694-a202b4916ee3-maas-api-tls\") pod \"maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf\" (UID: \"3a57d3db-4264-46dd-a694-a202b4916ee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf" Jul 06 20:33:51.295903 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:51.295796 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxs5j\" (UniqueName: \"kubernetes.io/projected/3a57d3db-4264-46dd-a694-a202b4916ee3-kube-api-access-wxs5j\") pod \"maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf\" (UID: \"3a57d3db-4264-46dd-a694-a202b4916ee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf" Jul 06 20:33:51.295976 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:51.295928 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a57d3db-4264-46dd-a694-a202b4916ee3-openshift-service-ca\") pod \"maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf\" (UID: \"3a57d3db-4264-46dd-a694-a202b4916ee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf" Jul 06 20:33:51.301509 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:51.301485 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf"] Jul 06 20:33:51.348362 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:51.348330 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:33:51.396397 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:51.396358 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a57d3db-4264-46dd-a694-a202b4916ee3-openshift-service-ca\") pod \"maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf\" (UID: \"3a57d3db-4264-46dd-a694-a202b4916ee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf" Jul 06 20:33:51.396572 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:51.396414 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a57d3db-4264-46dd-a694-a202b4916ee3-maas-api-tls\") pod \"maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf\" (UID: \"3a57d3db-4264-46dd-a694-a202b4916ee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf" Jul 06 20:33:51.396572 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:51.396440 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxs5j\" (UniqueName: \"kubernetes.io/projected/3a57d3db-4264-46dd-a694-a202b4916ee3-kube-api-access-wxs5j\") pod \"maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf\" (UID: \"3a57d3db-4264-46dd-a694-a202b4916ee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf" Jul 06 20:33:51.396572 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:33:51.396534 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-a8dc09e7-serving-cert: secret "maas-api-e2e-mt-a8dc09e7-serving-cert" not found Jul 06 20:33:51.396681 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:33:51.396624 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a57d3db-4264-46dd-a694-a202b4916ee3-maas-api-tls podName:3a57d3db-4264-46dd-a694-a202b4916ee3 nodeName:}" failed. No retries permitted until 2026-07-06 20:33:51.896600718 +0000 UTC m=+1683.107748914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3a57d3db-4264-46dd-a694-a202b4916ee3-maas-api-tls") pod "maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf" (UID: "3a57d3db-4264-46dd-a694-a202b4916ee3") : secret "maas-api-e2e-mt-a8dc09e7-serving-cert" not found Jul 06 20:33:51.397070 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:51.397052 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a57d3db-4264-46dd-a694-a202b4916ee3-openshift-service-ca\") pod \"maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf\" (UID: \"3a57d3db-4264-46dd-a694-a202b4916ee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf" Jul 06 20:33:51.409287 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:51.409260 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxs5j\" (UniqueName: \"kubernetes.io/projected/3a57d3db-4264-46dd-a694-a202b4916ee3-kube-api-access-wxs5j\") pod \"maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf\" (UID: \"3a57d3db-4264-46dd-a694-a202b4916ee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf" Jul 06 20:33:51.595081 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:51.595003 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:33:51.816629 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:51.816601 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:33:51.900431 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:51.900401 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a57d3db-4264-46dd-a694-a202b4916ee3-maas-api-tls\") pod \"maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf\" (UID: \"3a57d3db-4264-46dd-a694-a202b4916ee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf" Jul 06 20:33:51.903017 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:51.902992 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a57d3db-4264-46dd-a694-a202b4916ee3-maas-api-tls\") pod \"maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf\" (UID: \"3a57d3db-4264-46dd-a694-a202b4916ee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf" Jul 06 20:33:52.201975 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:52.201866 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf" Jul 06 20:33:52.336508 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:52.336478 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf"] Jul 06 20:33:52.338638 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:33:52.338611 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a57d3db_4264_46dd_a694_a202b4916ee3.slice/crio-0603120eb05661b8e4ec9c1f48ed9079a5d4d0ac2572b08a7f19eacd9b0eaa10 WatchSource:0}: Error finding container 0603120eb05661b8e4ec9c1f48ed9079a5d4d0ac2572b08a7f19eacd9b0eaa10: Status 404 returned error can't find the container with id 0603120eb05661b8e4ec9c1f48ed9079a5d4d0ac2572b08a7f19eacd9b0eaa10 Jul 06 20:33:52.968440 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:52.968354 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:33:53.148149 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:53.148117 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf" event={"ID":"3a57d3db-4264-46dd-a694-a202b4916ee3","Type":"ContainerStarted","Data":"8a0426a0f6dc9f46de54ee3915a213a3107fbf979cab29c63cdd54b34dceee35"} Jul 06 20:33:53.148313 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:53.148172 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf" event={"ID":"3a57d3db-4264-46dd-a694-a202b4916ee3","Type":"ContainerStarted","Data":"0603120eb05661b8e4ec9c1f48ed9079a5d4d0ac2572b08a7f19eacd9b0eaa10"} Jul 06 20:33:53.148313 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:53.148245 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf" Jul 06 20:33:53.170249 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:53.170201 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf" podStartSLOduration=1.805766814 podStartE2EDuration="2.1701855s" totalImagesPullingTime="364.418686ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:33:53.168364391 +0000 UTC m=+1684.379512608" watchObservedRunningTime="2026-07-06 20:33:53.1701855 +0000 UTC m=+1684.381333760" Jul 06 20:33:55.670792 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:55.670743 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:33:56.322710 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:56.322676 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:33:56.423519 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:56.423481 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:33:56.754192 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:56.754158 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:33:59.158526 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:33:59.158492 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf" Jul 06 20:34:03.244447 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:03.244414 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:34:03.312607 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:03.312578 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:34:03.728139 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:03.728105 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:34:04.130041 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:04.129964 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:34:04.332513 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:04.332482 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf"] Jul 06 20:34:04.332876 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:04.332694 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf" podUID="3a57d3db-4264-46dd-a694-a202b4916ee3" containerName="maas-api" containerID="cri-o://8a0426a0f6dc9f46de54ee3915a213a3107fbf979cab29c63cdd54b34dceee35" gracePeriod=30 Jul 06 20:34:04.378722 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:04.378696 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:34:04.580903 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:04.580880 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf" Jul 06 20:34:04.717970 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:04.717871 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a57d3db-4264-46dd-a694-a202b4916ee3-maas-api-tls\") pod \"3a57d3db-4264-46dd-a694-a202b4916ee3\" (UID: \"3a57d3db-4264-46dd-a694-a202b4916ee3\") " Jul 06 20:34:04.717970 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:04.717948 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxs5j\" (UniqueName: \"kubernetes.io/projected/3a57d3db-4264-46dd-a694-a202b4916ee3-kube-api-access-wxs5j\") pod \"3a57d3db-4264-46dd-a694-a202b4916ee3\" (UID: \"3a57d3db-4264-46dd-a694-a202b4916ee3\") " Jul 06 20:34:04.718150 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:04.718025 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a57d3db-4264-46dd-a694-a202b4916ee3-openshift-service-ca\") pod \"3a57d3db-4264-46dd-a694-a202b4916ee3\" (UID: \"3a57d3db-4264-46dd-a694-a202b4916ee3\") " Jul 06 20:34:04.718390 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:04.718359 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a57d3db-4264-46dd-a694-a202b4916ee3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3a57d3db-4264-46dd-a694-a202b4916ee3" (UID: "3a57d3db-4264-46dd-a694-a202b4916ee3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:34:04.720102 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:04.720074 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a57d3db-4264-46dd-a694-a202b4916ee3-kube-api-access-wxs5j" (OuterVolumeSpecName: "kube-api-access-wxs5j") pod "3a57d3db-4264-46dd-a694-a202b4916ee3" (UID: "3a57d3db-4264-46dd-a694-a202b4916ee3"). InnerVolumeSpecName "kube-api-access-wxs5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:34:04.720191 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:04.720122 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a57d3db-4264-46dd-a694-a202b4916ee3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3a57d3db-4264-46dd-a694-a202b4916ee3" (UID: "3a57d3db-4264-46dd-a694-a202b4916ee3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:34:04.819449 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:04.819409 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxs5j\" (UniqueName: \"kubernetes.io/projected/3a57d3db-4264-46dd-a694-a202b4916ee3-kube-api-access-wxs5j\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:34:04.819449 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:04.819440 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a57d3db-4264-46dd-a694-a202b4916ee3-openshift-service-ca\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:34:04.819449 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:04.819451 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a57d3db-4264-46dd-a694-a202b4916ee3-maas-api-tls\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:34:05.200210 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:05.200178 2578 generic.go:356] "Generic (PLEG): container finished" podID="3a57d3db-4264-46dd-a694-a202b4916ee3" containerID="8a0426a0f6dc9f46de54ee3915a213a3107fbf979cab29c63cdd54b34dceee35" exitCode=0 Jul 06 20:34:05.200383 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:05.200225 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf" Jul 06 20:34:05.200383 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:05.200239 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf" event={"ID":"3a57d3db-4264-46dd-a694-a202b4916ee3","Type":"ContainerDied","Data":"8a0426a0f6dc9f46de54ee3915a213a3107fbf979cab29c63cdd54b34dceee35"} Jul 06 20:34:05.200383 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:05.200281 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf" event={"ID":"3a57d3db-4264-46dd-a694-a202b4916ee3","Type":"ContainerDied","Data":"0603120eb05661b8e4ec9c1f48ed9079a5d4d0ac2572b08a7f19eacd9b0eaa10"} Jul 06 20:34:05.200383 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:05.200304 2578 scope.go:117] "RemoveContainer" containerID="8a0426a0f6dc9f46de54ee3915a213a3107fbf979cab29c63cdd54b34dceee35" Jul 06 20:34:05.209034 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:05.209016 2578 scope.go:117] "RemoveContainer" containerID="8a0426a0f6dc9f46de54ee3915a213a3107fbf979cab29c63cdd54b34dceee35" Jul 06 20:34:05.209279 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:34:05.209259 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a0426a0f6dc9f46de54ee3915a213a3107fbf979cab29c63cdd54b34dceee35\": container with ID starting with 8a0426a0f6dc9f46de54ee3915a213a3107fbf979cab29c63cdd54b34dceee35 not found: ID does not exist" containerID="8a0426a0f6dc9f46de54ee3915a213a3107fbf979cab29c63cdd54b34dceee35" Jul 06 20:34:05.209330 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:05.209289 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a0426a0f6dc9f46de54ee3915a213a3107fbf979cab29c63cdd54b34dceee35"} err="failed to get container status \"8a0426a0f6dc9f46de54ee3915a213a3107fbf979cab29c63cdd54b34dceee35\": rpc error: code = NotFound desc = could not find container \"8a0426a0f6dc9f46de54ee3915a213a3107fbf979cab29c63cdd54b34dceee35\": container with ID starting with 8a0426a0f6dc9f46de54ee3915a213a3107fbf979cab29c63cdd54b34dceee35 not found: ID does not exist" Jul 06 20:34:05.228943 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:05.228915 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf"] Jul 06 20:34:05.236040 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:05.236018 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a8dc09e7-7dd5f97cc9-5swnf"] Jul 06 20:34:05.298936 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:05.298905 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a57d3db-4264-46dd-a694-a202b4916ee3" path="/var/lib/kubelet/pods/3a57d3db-4264-46dd-a694-a202b4916ee3/volumes" Jul 06 20:34:10.847413 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:10.847330 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:34:10.915672 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:10.915634 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:34:11.111856 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:11.111754 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:34:11.463663 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:11.463628 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:34:11.563060 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:11.563026 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:34:11.706135 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:11.706098 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:34:18.327778 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:18.327743 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:34:18.401329 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:18.401293 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:34:18.483566 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:18.483530 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:34:18.579994 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:18.579903 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:34:18.684150 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:18.684114 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:34:25.014301 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:25.014261 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:34:25.113597 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:25.113564 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:34:31.820165 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:31.820127 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:34:32.128308 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:32.128224 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:34:32.347171 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:32.347140 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:34:35.437065 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:35.437030 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:34:35.538155 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:35.538118 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:34:35.652343 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:35.652303 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h"] Jul 06 20:34:35.652682 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:35.652668 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a57d3db-4264-46dd-a694-a202b4916ee3" containerName="maas-api" Jul 06 20:34:35.652682 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:35.652683 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a57d3db-4264-46dd-a694-a202b4916ee3" containerName="maas-api" Jul 06 20:34:35.652799 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:35.652743 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a57d3db-4264-46dd-a694-a202b4916ee3" containerName="maas-api" Jul 06 20:34:35.656006 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:35.655986 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h" Jul 06 20:34:35.658956 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:35.658933 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-080e3c37-serving-cert\"" Jul 06 20:34:35.665764 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:35.665741 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h"] Jul 06 20:34:35.679374 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:35.679343 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:34:35.694404 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:35.694331 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aaefe21-1f90-476c-987e-c9c6645be576-openshift-service-ca\") pod \"maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h\" (UID: \"4aaefe21-1f90-476c-987e-c9c6645be576\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h" Jul 06 20:34:35.694404 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:35.694367 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aaefe21-1f90-476c-987e-c9c6645be576-maas-api-tls\") pod \"maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h\" (UID: \"4aaefe21-1f90-476c-987e-c9c6645be576\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h" Jul 06 20:34:35.694404 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:35.694392 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79b2t\" (UniqueName: \"kubernetes.io/projected/4aaefe21-1f90-476c-987e-c9c6645be576-kube-api-access-79b2t\") pod \"maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h\" (UID: \"4aaefe21-1f90-476c-987e-c9c6645be576\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h" Jul 06 20:34:35.795013 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:35.794980 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aaefe21-1f90-476c-987e-c9c6645be576-openshift-service-ca\") pod \"maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h\" (UID: \"4aaefe21-1f90-476c-987e-c9c6645be576\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h" Jul 06 20:34:35.795216 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:35.795023 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aaefe21-1f90-476c-987e-c9c6645be576-maas-api-tls\") pod \"maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h\" (UID: \"4aaefe21-1f90-476c-987e-c9c6645be576\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h" Jul 06 20:34:35.795216 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:35.795059 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79b2t\" (UniqueName: \"kubernetes.io/projected/4aaefe21-1f90-476c-987e-c9c6645be576-kube-api-access-79b2t\") pod \"maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h\" (UID: \"4aaefe21-1f90-476c-987e-c9c6645be576\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h" Jul 06 20:34:35.795216 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:34:35.795196 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-080e3c37-serving-cert: secret "maas-api-e2e-mt-080e3c37-serving-cert" not found Jul 06 20:34:35.795380 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:34:35.795269 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4aaefe21-1f90-476c-987e-c9c6645be576-maas-api-tls podName:4aaefe21-1f90-476c-987e-c9c6645be576 nodeName:}" failed. No retries permitted until 2026-07-06 20:34:36.295248081 +0000 UTC m=+1727.506396279 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4aaefe21-1f90-476c-987e-c9c6645be576-maas-api-tls") pod "maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h" (UID: "4aaefe21-1f90-476c-987e-c9c6645be576") : secret "maas-api-e2e-mt-080e3c37-serving-cert" not found Jul 06 20:34:35.795646 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:35.795624 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aaefe21-1f90-476c-987e-c9c6645be576-openshift-service-ca\") pod \"maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h\" (UID: \"4aaefe21-1f90-476c-987e-c9c6645be576\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h" Jul 06 20:34:35.805507 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:35.805483 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79b2t\" (UniqueName: \"kubernetes.io/projected/4aaefe21-1f90-476c-987e-c9c6645be576-kube-api-access-79b2t\") pod \"maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h\" (UID: \"4aaefe21-1f90-476c-987e-c9c6645be576\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h" Jul 06 20:34:35.945642 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:35.945554 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:34:36.183199 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:36.183169 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:34:36.283418 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:36.283340 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:34:36.299173 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:36.299139 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aaefe21-1f90-476c-987e-c9c6645be576-maas-api-tls\") pod \"maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h\" (UID: \"4aaefe21-1f90-476c-987e-c9c6645be576\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h" Jul 06 20:34:36.301861 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:36.301828 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aaefe21-1f90-476c-987e-c9c6645be576-maas-api-tls\") pod \"maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h\" (UID: \"4aaefe21-1f90-476c-987e-c9c6645be576\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h" Jul 06 20:34:36.567085 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:36.566991 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h" Jul 06 20:34:36.705998 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:36.705972 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h"] Jul 06 20:34:36.707747 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:34:36.707720 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aaefe21_1f90_476c_987e_c9c6645be576.slice/crio-0cc766390dfacd124ce35849f9515dec48e0659b0e04a3d568f96cb96be1bb02 WatchSource:0}: Error finding container 0cc766390dfacd124ce35849f9515dec48e0659b0e04a3d568f96cb96be1bb02: Status 404 returned error can't find the container with id 0cc766390dfacd124ce35849f9515dec48e0659b0e04a3d568f96cb96be1bb02 Jul 06 20:34:37.338982 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:37.338943 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h" event={"ID":"4aaefe21-1f90-476c-987e-c9c6645be576","Type":"ContainerStarted","Data":"8d59e244c32c6cefe7231597a142f306c403906e4033f7e358d24ef69676bed4"} Jul 06 20:34:37.339158 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:37.339004 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h" event={"ID":"4aaefe21-1f90-476c-987e-c9c6645be576","Type":"ContainerStarted","Data":"0cc766390dfacd124ce35849f9515dec48e0659b0e04a3d568f96cb96be1bb02"} Jul 06 20:34:37.339158 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:37.339030 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h" Jul 06 20:34:37.358222 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:37.358172 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h" podStartSLOduration=1.826657591 podStartE2EDuration="2.358155432s" totalImagesPullingTime="531.497841ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:34:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:34:37.356844685 +0000 UTC m=+1728.567992904" watchObservedRunningTime="2026-07-06 20:34:37.358155432 +0000 UTC m=+1728.569303650" Jul 06 20:34:39.950801 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:39.950763 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:34:43.348890 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:43.348862 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h" Jul 06 20:34:46.641867 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:34:46.641830 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:35:13.197431 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:35:13.197393 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:35:13.642302 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:35:13.642272 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h"] Jul 06 20:35:13.642532 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:35:13.642507 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h" podUID="4aaefe21-1f90-476c-987e-c9c6645be576" containerName="maas-api" containerID="cri-o://8d59e244c32c6cefe7231597a142f306c403906e4033f7e358d24ef69676bed4" gracePeriod=30 Jul 06 20:35:13.666302 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:35:13.666265 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:35:13.881944 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:35:13.881917 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h" Jul 06 20:35:14.036886 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:35:14.036797 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aaefe21-1f90-476c-987e-c9c6645be576-openshift-service-ca\") pod \"4aaefe21-1f90-476c-987e-c9c6645be576\" (UID: \"4aaefe21-1f90-476c-987e-c9c6645be576\") " Jul 06 20:35:14.036886 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:35:14.036882 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aaefe21-1f90-476c-987e-c9c6645be576-maas-api-tls\") pod \"4aaefe21-1f90-476c-987e-c9c6645be576\" (UID: \"4aaefe21-1f90-476c-987e-c9c6645be576\") " Jul 06 20:35:14.037130 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:35:14.036915 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79b2t\" (UniqueName: \"kubernetes.io/projected/4aaefe21-1f90-476c-987e-c9c6645be576-kube-api-access-79b2t\") pod \"4aaefe21-1f90-476c-987e-c9c6645be576\" (UID: \"4aaefe21-1f90-476c-987e-c9c6645be576\") " Jul 06 20:35:14.037260 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:35:14.037234 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aaefe21-1f90-476c-987e-c9c6645be576-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4aaefe21-1f90-476c-987e-c9c6645be576" (UID: "4aaefe21-1f90-476c-987e-c9c6645be576"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:35:14.039038 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:35:14.039000 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aaefe21-1f90-476c-987e-c9c6645be576-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4aaefe21-1f90-476c-987e-c9c6645be576" (UID: "4aaefe21-1f90-476c-987e-c9c6645be576"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:35:14.039149 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:35:14.039128 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aaefe21-1f90-476c-987e-c9c6645be576-kube-api-access-79b2t" (OuterVolumeSpecName: "kube-api-access-79b2t") pod "4aaefe21-1f90-476c-987e-c9c6645be576" (UID: "4aaefe21-1f90-476c-987e-c9c6645be576"). InnerVolumeSpecName "kube-api-access-79b2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:35:14.138054 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:35:14.138017 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79b2t\" (UniqueName: \"kubernetes.io/projected/4aaefe21-1f90-476c-987e-c9c6645be576-kube-api-access-79b2t\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:35:14.138054 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:35:14.138046 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aaefe21-1f90-476c-987e-c9c6645be576-openshift-service-ca\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:35:14.138259 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:35:14.138058 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aaefe21-1f90-476c-987e-c9c6645be576-maas-api-tls\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:35:14.496435 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:35:14.496407 2578 generic.go:356] "Generic (PLEG): container finished" podID="4aaefe21-1f90-476c-987e-c9c6645be576" containerID="8d59e244c32c6cefe7231597a142f306c403906e4033f7e358d24ef69676bed4" exitCode=0 Jul 06 20:35:14.496851 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:35:14.496474 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h" Jul 06 20:35:14.496851 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:35:14.496498 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h" event={"ID":"4aaefe21-1f90-476c-987e-c9c6645be576","Type":"ContainerDied","Data":"8d59e244c32c6cefe7231597a142f306c403906e4033f7e358d24ef69676bed4"} Jul 06 20:35:14.496851 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:35:14.496549 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h" event={"ID":"4aaefe21-1f90-476c-987e-c9c6645be576","Type":"ContainerDied","Data":"0cc766390dfacd124ce35849f9515dec48e0659b0e04a3d568f96cb96be1bb02"} Jul 06 20:35:14.496851 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:35:14.496569 2578 scope.go:117] "RemoveContainer" containerID="8d59e244c32c6cefe7231597a142f306c403906e4033f7e358d24ef69676bed4" Jul 06 20:35:14.505324 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:35:14.505307 2578 scope.go:117] "RemoveContainer" containerID="8d59e244c32c6cefe7231597a142f306c403906e4033f7e358d24ef69676bed4" Jul 06 20:35:14.505584 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:35:14.505567 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d59e244c32c6cefe7231597a142f306c403906e4033f7e358d24ef69676bed4\": container with ID starting with 8d59e244c32c6cefe7231597a142f306c403906e4033f7e358d24ef69676bed4 not found: ID does not exist" containerID="8d59e244c32c6cefe7231597a142f306c403906e4033f7e358d24ef69676bed4" Jul 06 20:35:14.505647 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:35:14.505600 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d59e244c32c6cefe7231597a142f306c403906e4033f7e358d24ef69676bed4"} err="failed to get container status \"8d59e244c32c6cefe7231597a142f306c403906e4033f7e358d24ef69676bed4\": rpc error: code = NotFound desc = could not find container \"8d59e244c32c6cefe7231597a142f306c403906e4033f7e358d24ef69676bed4\": container with ID starting with 8d59e244c32c6cefe7231597a142f306c403906e4033f7e358d24ef69676bed4 not found: ID does not exist" Jul 06 20:35:14.520209 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:35:14.520179 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h"] Jul 06 20:35:14.525891 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:35:14.525860 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-080e3c37-5dcf854b79-8dd8h"] Jul 06 20:35:15.298060 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:35:15.298023 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4aaefe21-1f90-476c-987e-c9c6645be576" path="/var/lib/kubelet/pods/4aaefe21-1f90-476c-987e-c9c6645be576/volumes" Jul 06 20:35:19.269045 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:35:19.269014 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:35:24.344423 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:35:24.344388 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:35:49.299015 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:35:49.298981 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dnfgp_0f4c3a8a-30c4-428e-88a6-dcf086ef8615/ovn-acl-logging/0.log" Jul 06 20:35:49.301980 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:35:49.301958 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dnfgp_0f4c3a8a-30c4-428e-88a6-dcf086ef8615/ovn-acl-logging/0.log" Jul 06 20:36:01.588640 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:36:01.588601 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:36:01.990838 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:36:01.990806 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:36:08.319857 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:36:08.319825 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:36:08.427422 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:36:08.427383 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:36:08.648046 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:36:08.648015 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:36:09.758308 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:36:09.758270 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:36:09.857156 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:36:09.857118 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:36:09.957467 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:36:09.957407 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:36:10.057301 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:36:10.057220 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:36:10.158095 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:36:10.158059 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:36:10.925991 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:36:10.925961 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:36:11.017533 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:36:11.017500 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:36:11.176477 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:36:11.176366 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:36:12.534604 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:36:12.534567 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:36:19.306168 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:36:19.306133 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:36:19.391566 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:36:19.391530 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:36:20.062140 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:36:20.062105 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:36:23.756883 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:36:23.756849 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:36:23.823701 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:36:23.823665 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:36:26.541780 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:36:26.541746 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:36:26.677060 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:36:26.677029 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:36:26.775778 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:36:26.775747 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:36:37.353273 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:36:37.353228 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:36:37.686019 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:36:37.685991 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:36:58.639978 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:36:58.639893 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:37:05.134642 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:05.134610 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:37:05.230358 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:05.230320 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:37:05.481222 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:05.481184 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:37:05.871254 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:05.871161 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:37:05.972035 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:05.971979 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:37:08.470040 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:08.470000 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:37:08.540180 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:08.540140 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd"] Jul 06 20:37:08.540516 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:08.540503 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aaefe21-1f90-476c-987e-c9c6645be576" containerName="maas-api" Jul 06 20:37:08.540574 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:08.540518 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aaefe21-1f90-476c-987e-c9c6645be576" containerName="maas-api" Jul 06 20:37:08.540617 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:08.540584 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aaefe21-1f90-476c-987e-c9c6645be576" containerName="maas-api" Jul 06 20:37:08.543355 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:08.543336 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd" Jul 06 20:37:08.546213 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:08.546193 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-cb2bb9-serving-cert\"" Jul 06 20:37:08.546964 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:08.546943 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:37:08.558234 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:08.558211 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd"] Jul 06 20:37:08.616521 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:08.616483 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab189dcf-b1c3-4937-b5f3-b9eed27efe74-maas-api-tls\") pod \"maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd\" (UID: \"ab189dcf-b1c3-4937-b5f3-b9eed27efe74\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd" Jul 06 20:37:08.616696 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:08.616530 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab189dcf-b1c3-4937-b5f3-b9eed27efe74-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd\" (UID: \"ab189dcf-b1c3-4937-b5f3-b9eed27efe74\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd" Jul 06 20:37:08.616696 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:08.616560 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swdd5\" (UniqueName: \"kubernetes.io/projected/ab189dcf-b1c3-4937-b5f3-b9eed27efe74-kube-api-access-swdd5\") pod \"maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd\" (UID: \"ab189dcf-b1c3-4937-b5f3-b9eed27efe74\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd" Jul 06 20:37:08.631893 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:08.631830 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:37:08.717172 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:08.717134 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab189dcf-b1c3-4937-b5f3-b9eed27efe74-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd\" (UID: \"ab189dcf-b1c3-4937-b5f3-b9eed27efe74\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd" Jul 06 20:37:08.717172 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:08.717179 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swdd5\" (UniqueName: \"kubernetes.io/projected/ab189dcf-b1c3-4937-b5f3-b9eed27efe74-kube-api-access-swdd5\") pod \"maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd\" (UID: \"ab189dcf-b1c3-4937-b5f3-b9eed27efe74\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd" Jul 06 20:37:08.717404 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:08.717248 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab189dcf-b1c3-4937-b5f3-b9eed27efe74-maas-api-tls\") pod \"maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd\" (UID: \"ab189dcf-b1c3-4937-b5f3-b9eed27efe74\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd" Jul 06 20:37:08.717404 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:37:08.717349 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-cb2bb9-serving-cert: secret "maas-api-e2e-inf-a-cb2bb9-serving-cert" not found Jul 06 20:37:08.717521 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:37:08.717415 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab189dcf-b1c3-4937-b5f3-b9eed27efe74-maas-api-tls podName:ab189dcf-b1c3-4937-b5f3-b9eed27efe74 nodeName:}" failed. No retries permitted until 2026-07-06 20:37:09.217397678 +0000 UTC m=+1880.428545875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ab189dcf-b1c3-4937-b5f3-b9eed27efe74-maas-api-tls") pod "maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd" (UID: "ab189dcf-b1c3-4937-b5f3-b9eed27efe74") : secret "maas-api-e2e-inf-a-cb2bb9-serving-cert" not found Jul 06 20:37:08.717822 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:08.717803 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab189dcf-b1c3-4937-b5f3-b9eed27efe74-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd\" (UID: \"ab189dcf-b1c3-4937-b5f3-b9eed27efe74\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd" Jul 06 20:37:08.729036 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:08.728974 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swdd5\" (UniqueName: \"kubernetes.io/projected/ab189dcf-b1c3-4937-b5f3-b9eed27efe74-kube-api-access-swdd5\") pod \"maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd\" (UID: \"ab189dcf-b1c3-4937-b5f3-b9eed27efe74\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd" Jul 06 20:37:08.843173 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:08.843143 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:37:09.157013 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:09.156980 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:37:09.222348 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:09.222308 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab189dcf-b1c3-4937-b5f3-b9eed27efe74-maas-api-tls\") pod \"maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd\" (UID: \"ab189dcf-b1c3-4937-b5f3-b9eed27efe74\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd" Jul 06 20:37:09.224880 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:09.224859 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab189dcf-b1c3-4937-b5f3-b9eed27efe74-maas-api-tls\") pod \"maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd\" (UID: \"ab189dcf-b1c3-4937-b5f3-b9eed27efe74\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd" Jul 06 20:37:09.257103 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:09.257066 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:37:09.454032 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:09.453911 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd" Jul 06 20:37:09.605979 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:09.605946 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd"] Jul 06 20:37:09.609715 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:37:09.609683 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab189dcf_b1c3_4937_b5f3_b9eed27efe74.slice/crio-ce89149692aae5f2ce672f2d399fce9cb3d08972ddfdaf127fe8821e06bf6dee WatchSource:0}: Error finding container ce89149692aae5f2ce672f2d399fce9cb3d08972ddfdaf127fe8821e06bf6dee: Status 404 returned error can't find the container with id ce89149692aae5f2ce672f2d399fce9cb3d08972ddfdaf127fe8821e06bf6dee Jul 06 20:37:09.611225 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:09.611205 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 20:37:09.998677 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:09.998641 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd" event={"ID":"ab189dcf-b1c3-4937-b5f3-b9eed27efe74","Type":"ContainerStarted","Data":"ce89149692aae5f2ce672f2d399fce9cb3d08972ddfdaf127fe8821e06bf6dee"} Jul 06 20:37:11.005043 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:11.005007 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd" event={"ID":"ab189dcf-b1c3-4937-b5f3-b9eed27efe74","Type":"ContainerStarted","Data":"b4f5dac77a8c3f9da2c07edac49c48cce990037a461d7cb3164afa2b7b3da987"} Jul 06 20:37:11.005569 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:11.005095 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd" Jul 06 20:37:11.024019 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:11.023936 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd" podStartSLOduration=2.536653821 podStartE2EDuration="3.02391661s" totalImagesPullingTime="487.262789ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:11.023239712 +0000 UTC m=+1882.234387929" watchObservedRunningTime="2026-07-06 20:37:11.02391661 +0000 UTC m=+1882.235064830" Jul 06 20:37:12.900253 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:12.900215 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:37:13.244987 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:13.244897 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:37:13.639735 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:13.639649 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:37:13.838185 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:13.838148 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:37:16.757727 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:16.757690 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:37:16.842862 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:16.842830 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:37:16.959008 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:16.958977 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv"] Jul 06 20:37:16.962745 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:16.962727 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv" Jul 06 20:37:16.969220 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:16.969089 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-844a47-serving-cert\"" Jul 06 20:37:16.977418 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:16.977393 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv"] Jul 06 20:37:16.987740 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:16.987714 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a9b0c97-72c8-4975-b521-c66d45035ac2-maas-api-tls\") pod \"maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv\" (UID: \"6a9b0c97-72c8-4975-b521-c66d45035ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv" Jul 06 20:37:16.987882 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:16.987759 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mwcm\" (UniqueName: \"kubernetes.io/projected/6a9b0c97-72c8-4975-b521-c66d45035ac2-kube-api-access-9mwcm\") pod \"maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv\" (UID: \"6a9b0c97-72c8-4975-b521-c66d45035ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv" Jul 06 20:37:16.987882 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:16.987783 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9b0c97-72c8-4975-b521-c66d45035ac2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv\" (UID: \"6a9b0c97-72c8-4975-b521-c66d45035ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv" Jul 06 20:37:16.997066 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:16.997027 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:37:17.015249 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:17.015184 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd" Jul 06 20:37:17.089218 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:17.089178 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9b0c97-72c8-4975-b521-c66d45035ac2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv\" (UID: \"6a9b0c97-72c8-4975-b521-c66d45035ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv" Jul 06 20:37:17.089419 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:17.089273 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a9b0c97-72c8-4975-b521-c66d45035ac2-maas-api-tls\") pod \"maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv\" (UID: \"6a9b0c97-72c8-4975-b521-c66d45035ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv" Jul 06 20:37:17.089419 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:17.089324 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mwcm\" (UniqueName: \"kubernetes.io/projected/6a9b0c97-72c8-4975-b521-c66d45035ac2-kube-api-access-9mwcm\") pod \"maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv\" (UID: \"6a9b0c97-72c8-4975-b521-c66d45035ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv" Jul 06 20:37:17.089914 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:17.089894 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9b0c97-72c8-4975-b521-c66d45035ac2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv\" (UID: \"6a9b0c97-72c8-4975-b521-c66d45035ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv" Jul 06 20:37:17.091943 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:17.091911 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a9b0c97-72c8-4975-b521-c66d45035ac2-maas-api-tls\") pod \"maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv\" (UID: \"6a9b0c97-72c8-4975-b521-c66d45035ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv" Jul 06 20:37:17.102131 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:17.102105 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:37:17.105994 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:17.105972 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mwcm\" (UniqueName: \"kubernetes.io/projected/6a9b0c97-72c8-4975-b521-c66d45035ac2-kube-api-access-9mwcm\") pod \"maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv\" (UID: \"6a9b0c97-72c8-4975-b521-c66d45035ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv" Jul 06 20:37:17.272880 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:17.272793 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv" Jul 06 20:37:17.417100 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:17.417074 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv"] Jul 06 20:37:17.419840 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:37:17.419764 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a9b0c97_72c8_4975_b521_c66d45035ac2.slice/crio-2d1f855ed4f42af0de19a2daaa659c391d47f07843232134ca36980406f2144c WatchSource:0}: Error finding container 2d1f855ed4f42af0de19a2daaa659c391d47f07843232134ca36980406f2144c: Status 404 returned error can't find the container with id 2d1f855ed4f42af0de19a2daaa659c391d47f07843232134ca36980406f2144c Jul 06 20:37:17.630487 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:17.630361 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:37:17.732821 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:17.732778 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:37:18.037587 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:18.037538 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv" event={"ID":"6a9b0c97-72c8-4975-b521-c66d45035ac2","Type":"ContainerStarted","Data":"99382bfc1a660c8324318ae9c39fd68adab6eab2c223ab5c61361a4e57707e60"} Jul 06 20:37:18.037979 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:18.037648 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv" event={"ID":"6a9b0c97-72c8-4975-b521-c66d45035ac2","Type":"ContainerStarted","Data":"2d1f855ed4f42af0de19a2daaa659c391d47f07843232134ca36980406f2144c"} Jul 06 20:37:18.037979 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:18.037676 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv" Jul 06 20:37:18.058256 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:18.058198 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv" podStartSLOduration=1.7576311690000002 podStartE2EDuration="2.058179827s" totalImagesPullingTime="300.548658ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:37:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:18.057713897 +0000 UTC m=+1889.268862115" watchObservedRunningTime="2026-07-06 20:37:18.058179827 +0000 UTC m=+1889.269328045" Jul 06 20:37:21.124544 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:21.124506 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:37:22.387967 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:22.387937 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:37:22.466543 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:22.466513 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:37:24.047465 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:24.047433 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv" Jul 06 20:37:31.262885 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:31.262854 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:37:31.361920 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:31.361882 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:37:31.777840 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:31.777806 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:37:39.696774 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:39.696680 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:37:39.797306 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:39.797258 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:37:52.904094 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:52.904054 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:37:58.883587 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:37:58.883535 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:38:12.973956 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:38:12.973920 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:38:19.741860 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:38:19.741825 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:38:20.699830 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:38:20.699796 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:38:20.849607 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:38:20.849576 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:38:21.240234 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:38:21.240201 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:38:21.342081 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:38:21.342048 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:38:21.443321 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:38:21.443286 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:38:21.563713 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:38:21.563586 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd"] Jul 06 20:38:21.563952 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:38:21.563923 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd" podUID="ab189dcf-b1c3-4937-b5f3-b9eed27efe74" containerName="maas-api" containerID="cri-o://b4f5dac77a8c3f9da2c07edac49c48cce990037a461d7cb3164afa2b7b3da987" gracePeriod=30 Jul 06 20:38:21.594759 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:38:21.594732 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:38:22.008908 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:38:22.008885 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd" Jul 06 20:38:22.044940 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:38:22.044907 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab189dcf-b1c3-4937-b5f3-b9eed27efe74-maas-api-tls\") pod \"ab189dcf-b1c3-4937-b5f3-b9eed27efe74\" (UID: \"ab189dcf-b1c3-4937-b5f3-b9eed27efe74\") " Jul 06 20:38:22.045116 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:38:22.044949 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swdd5\" (UniqueName: \"kubernetes.io/projected/ab189dcf-b1c3-4937-b5f3-b9eed27efe74-kube-api-access-swdd5\") pod \"ab189dcf-b1c3-4937-b5f3-b9eed27efe74\" (UID: \"ab189dcf-b1c3-4937-b5f3-b9eed27efe74\") " Jul 06 20:38:22.045116 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:38:22.045017 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab189dcf-b1c3-4937-b5f3-b9eed27efe74-openshift-service-ca\") pod \"ab189dcf-b1c3-4937-b5f3-b9eed27efe74\" (UID: \"ab189dcf-b1c3-4937-b5f3-b9eed27efe74\") " Jul 06 20:38:22.045374 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:38:22.045344 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab189dcf-b1c3-4937-b5f3-b9eed27efe74-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ab189dcf-b1c3-4937-b5f3-b9eed27efe74" (UID: "ab189dcf-b1c3-4937-b5f3-b9eed27efe74"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:38:22.047130 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:38:22.047104 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab189dcf-b1c3-4937-b5f3-b9eed27efe74-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ab189dcf-b1c3-4937-b5f3-b9eed27efe74" (UID: "ab189dcf-b1c3-4937-b5f3-b9eed27efe74"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:38:22.047227 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:38:22.047155 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab189dcf-b1c3-4937-b5f3-b9eed27efe74-kube-api-access-swdd5" (OuterVolumeSpecName: "kube-api-access-swdd5") pod "ab189dcf-b1c3-4937-b5f3-b9eed27efe74" (UID: "ab189dcf-b1c3-4937-b5f3-b9eed27efe74"). InnerVolumeSpecName "kube-api-access-swdd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:38:22.146522 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:38:22.146494 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab189dcf-b1c3-4937-b5f3-b9eed27efe74-maas-api-tls\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:38:22.146522 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:38:22.146523 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swdd5\" (UniqueName: \"kubernetes.io/projected/ab189dcf-b1c3-4937-b5f3-b9eed27efe74-kube-api-access-swdd5\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:38:22.146708 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:38:22.146534 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab189dcf-b1c3-4937-b5f3-b9eed27efe74-openshift-service-ca\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:38:22.317886 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:38:22.317840 2578 generic.go:356] "Generic (PLEG): container finished" podID="ab189dcf-b1c3-4937-b5f3-b9eed27efe74" containerID="b4f5dac77a8c3f9da2c07edac49c48cce990037a461d7cb3164afa2b7b3da987" exitCode=0 Jul 06 20:38:22.318081 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:38:22.317900 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd" Jul 06 20:38:22.318081 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:38:22.317932 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd" event={"ID":"ab189dcf-b1c3-4937-b5f3-b9eed27efe74","Type":"ContainerDied","Data":"b4f5dac77a8c3f9da2c07edac49c48cce990037a461d7cb3164afa2b7b3da987"} Jul 06 20:38:22.318081 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:38:22.317991 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd" event={"ID":"ab189dcf-b1c3-4937-b5f3-b9eed27efe74","Type":"ContainerDied","Data":"ce89149692aae5f2ce672f2d399fce9cb3d08972ddfdaf127fe8821e06bf6dee"} Jul 06 20:38:22.318081 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:38:22.318021 2578 scope.go:117] "RemoveContainer" containerID="b4f5dac77a8c3f9da2c07edac49c48cce990037a461d7cb3164afa2b7b3da987" Jul 06 20:38:22.326848 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:38:22.326832 2578 scope.go:117] "RemoveContainer" containerID="b4f5dac77a8c3f9da2c07edac49c48cce990037a461d7cb3164afa2b7b3da987" Jul 06 20:38:22.327108 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:38:22.327090 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4f5dac77a8c3f9da2c07edac49c48cce990037a461d7cb3164afa2b7b3da987\": container with ID starting with b4f5dac77a8c3f9da2c07edac49c48cce990037a461d7cb3164afa2b7b3da987 not found: ID does not exist" containerID="b4f5dac77a8c3f9da2c07edac49c48cce990037a461d7cb3164afa2b7b3da987" Jul 06 20:38:22.327157 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:38:22.327116 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4f5dac77a8c3f9da2c07edac49c48cce990037a461d7cb3164afa2b7b3da987"} err="failed to get container status \"b4f5dac77a8c3f9da2c07edac49c48cce990037a461d7cb3164afa2b7b3da987\": rpc error: code = NotFound desc = could not find container \"b4f5dac77a8c3f9da2c07edac49c48cce990037a461d7cb3164afa2b7b3da987\": container with ID starting with b4f5dac77a8c3f9da2c07edac49c48cce990037a461d7cb3164afa2b7b3da987 not found: ID does not exist" Jul 06 20:38:22.341421 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:38:22.341398 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd"] Jul 06 20:38:22.344249 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:38:22.344230 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cb2bb9-54547c4b54-dk7sd"] Jul 06 20:38:23.298473 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:38:23.298428 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab189dcf-b1c3-4937-b5f3-b9eed27efe74" path="/var/lib/kubelet/pods/ab189dcf-b1c3-4937-b5f3-b9eed27efe74/volumes" Jul 06 20:38:26.317355 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:38:26.317300 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:03.528487 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:03.528424 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:04.969055 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:04.969021 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv"] Jul 06 20:39:04.969484 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:04.969273 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv" podUID="6a9b0c97-72c8-4975-b521-c66d45035ac2" containerName="maas-api" containerID="cri-o://99382bfc1a660c8324318ae9c39fd68adab6eab2c223ab5c61361a4e57707e60" gracePeriod=30 Jul 06 20:39:04.976163 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:04.976137 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:05.045394 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:05.045361 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:05.218840 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:05.218818 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv" Jul 06 20:39:05.322313 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:05.322235 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a9b0c97-72c8-4975-b521-c66d45035ac2-maas-api-tls\") pod \"6a9b0c97-72c8-4975-b521-c66d45035ac2\" (UID: \"6a9b0c97-72c8-4975-b521-c66d45035ac2\") " Jul 06 20:39:05.322313 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:05.322274 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mwcm\" (UniqueName: \"kubernetes.io/projected/6a9b0c97-72c8-4975-b521-c66d45035ac2-kube-api-access-9mwcm\") pod \"6a9b0c97-72c8-4975-b521-c66d45035ac2\" (UID: \"6a9b0c97-72c8-4975-b521-c66d45035ac2\") " Jul 06 20:39:05.322313 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:05.322303 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9b0c97-72c8-4975-b521-c66d45035ac2-openshift-service-ca\") pod \"6a9b0c97-72c8-4975-b521-c66d45035ac2\" (UID: \"6a9b0c97-72c8-4975-b521-c66d45035ac2\") " Jul 06 20:39:05.322820 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:05.322787 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a9b0c97-72c8-4975-b521-c66d45035ac2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6a9b0c97-72c8-4975-b521-c66d45035ac2" (UID: "6a9b0c97-72c8-4975-b521-c66d45035ac2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:39:05.324603 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:05.324579 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a9b0c97-72c8-4975-b521-c66d45035ac2-kube-api-access-9mwcm" (OuterVolumeSpecName: "kube-api-access-9mwcm") pod "6a9b0c97-72c8-4975-b521-c66d45035ac2" (UID: "6a9b0c97-72c8-4975-b521-c66d45035ac2"). InnerVolumeSpecName "kube-api-access-9mwcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:39:05.324706 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:05.324610 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a9b0c97-72c8-4975-b521-c66d45035ac2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6a9b0c97-72c8-4975-b521-c66d45035ac2" (UID: "6a9b0c97-72c8-4975-b521-c66d45035ac2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:39:05.423097 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:05.423058 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9b0c97-72c8-4975-b521-c66d45035ac2-openshift-service-ca\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:39:05.423097 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:05.423090 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a9b0c97-72c8-4975-b521-c66d45035ac2-maas-api-tls\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:39:05.423097 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:05.423100 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mwcm\" (UniqueName: \"kubernetes.io/projected/6a9b0c97-72c8-4975-b521-c66d45035ac2-kube-api-access-9mwcm\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:39:05.506632 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:05.506588 2578 generic.go:356] "Generic (PLEG): container finished" podID="6a9b0c97-72c8-4975-b521-c66d45035ac2" containerID="99382bfc1a660c8324318ae9c39fd68adab6eab2c223ab5c61361a4e57707e60" exitCode=0 Jul 06 20:39:05.506815 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:05.506659 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv" Jul 06 20:39:05.506815 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:05.506667 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv" event={"ID":"6a9b0c97-72c8-4975-b521-c66d45035ac2","Type":"ContainerDied","Data":"99382bfc1a660c8324318ae9c39fd68adab6eab2c223ab5c61361a4e57707e60"} Jul 06 20:39:05.506815 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:05.506720 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv" event={"ID":"6a9b0c97-72c8-4975-b521-c66d45035ac2","Type":"ContainerDied","Data":"2d1f855ed4f42af0de19a2daaa659c391d47f07843232134ca36980406f2144c"} Jul 06 20:39:05.506815 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:05.506742 2578 scope.go:117] "RemoveContainer" containerID="99382bfc1a660c8324318ae9c39fd68adab6eab2c223ab5c61361a4e57707e60" Jul 06 20:39:05.515620 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:05.515604 2578 scope.go:117] "RemoveContainer" containerID="99382bfc1a660c8324318ae9c39fd68adab6eab2c223ab5c61361a4e57707e60" Jul 06 20:39:05.515851 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:39:05.515834 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99382bfc1a660c8324318ae9c39fd68adab6eab2c223ab5c61361a4e57707e60\": container with ID starting with 99382bfc1a660c8324318ae9c39fd68adab6eab2c223ab5c61361a4e57707e60 not found: ID does not exist" containerID="99382bfc1a660c8324318ae9c39fd68adab6eab2c223ab5c61361a4e57707e60" Jul 06 20:39:05.515898 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:05.515859 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99382bfc1a660c8324318ae9c39fd68adab6eab2c223ab5c61361a4e57707e60"} err="failed to get container status \"99382bfc1a660c8324318ae9c39fd68adab6eab2c223ab5c61361a4e57707e60\": rpc error: code = NotFound desc = could not find container \"99382bfc1a660c8324318ae9c39fd68adab6eab2c223ab5c61361a4e57707e60\": container with ID starting with 99382bfc1a660c8324318ae9c39fd68adab6eab2c223ab5c61361a4e57707e60 not found: ID does not exist" Jul 06 20:39:05.536794 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:05.536765 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv"] Jul 06 20:39:05.552638 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:05.552607 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-844a47-6d448cdf65-zs9rv"] Jul 06 20:39:07.300917 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:07.300880 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a9b0c97-72c8-4975-b521-c66d45035ac2" path="/var/lib/kubelet/pods/6a9b0c97-72c8-4975-b521-c66d45035ac2/volumes" Jul 06 20:39:10.840231 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:10.840192 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:10.895244 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:10.895213 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:11.117305 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:11.117220 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:11.696910 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:11.696878 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:12.011059 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:12.010987 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:12.304847 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:12.304760 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:15.182578 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:15.182539 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:15.287581 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:15.287548 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6"] Jul 06 20:39:15.287955 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:15.287928 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab189dcf-b1c3-4937-b5f3-b9eed27efe74" containerName="maas-api" Jul 06 20:39:15.287955 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:15.287950 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab189dcf-b1c3-4937-b5f3-b9eed27efe74" containerName="maas-api" Jul 06 20:39:15.288211 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:15.287964 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a9b0c97-72c8-4975-b521-c66d45035ac2" containerName="maas-api" Jul 06 20:39:15.288211 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:15.287973 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a9b0c97-72c8-4975-b521-c66d45035ac2" containerName="maas-api" Jul 06 20:39:15.288211 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:15.288049 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab189dcf-b1c3-4937-b5f3-b9eed27efe74" containerName="maas-api" Jul 06 20:39:15.288211 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:15.288060 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a9b0c97-72c8-4975-b521-c66d45035ac2" containerName="maas-api" Jul 06 20:39:15.288211 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:15.288190 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:15.292227 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:15.292209 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6" Jul 06 20:39:15.294683 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:15.294664 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-3d7651-serving-cert\"" Jul 06 20:39:15.307026 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:15.307001 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6"] Jul 06 20:39:15.385937 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:15.385872 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:15.408016 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:15.407979 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhzc6\" (UniqueName: \"kubernetes.io/projected/56223f26-9288-45cd-867b-76f4b35f47d8-kube-api-access-bhzc6\") pod \"maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6\" (UID: \"56223f26-9288-45cd-867b-76f4b35f47d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6" Jul 06 20:39:15.408212 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:15.408144 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56223f26-9288-45cd-867b-76f4b35f47d8-openshift-service-ca\") pod \"maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6\" (UID: \"56223f26-9288-45cd-867b-76f4b35f47d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6" Jul 06 20:39:15.408212 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:15.408178 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56223f26-9288-45cd-867b-76f4b35f47d8-maas-api-tls\") pod \"maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6\" (UID: \"56223f26-9288-45cd-867b-76f4b35f47d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6" Jul 06 20:39:15.509072 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:15.508984 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56223f26-9288-45cd-867b-76f4b35f47d8-openshift-service-ca\") pod \"maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6\" (UID: \"56223f26-9288-45cd-867b-76f4b35f47d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6" Jul 06 20:39:15.509072 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:15.509023 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56223f26-9288-45cd-867b-76f4b35f47d8-maas-api-tls\") pod \"maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6\" (UID: \"56223f26-9288-45cd-867b-76f4b35f47d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6" Jul 06 20:39:15.509072 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:15.509050 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhzc6\" (UniqueName: \"kubernetes.io/projected/56223f26-9288-45cd-867b-76f4b35f47d8-kube-api-access-bhzc6\") pod \"maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6\" (UID: \"56223f26-9288-45cd-867b-76f4b35f47d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6" Jul 06 20:39:15.509299 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:39:15.509150 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-3d7651-serving-cert: secret "maas-api-e2e-api-a-3d7651-serving-cert" not found Jul 06 20:39:15.509299 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:39:15.509218 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56223f26-9288-45cd-867b-76f4b35f47d8-maas-api-tls podName:56223f26-9288-45cd-867b-76f4b35f47d8 nodeName:}" failed. No retries permitted until 2026-07-06 20:39:16.009199965 +0000 UTC m=+2007.220348162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/56223f26-9288-45cd-867b-76f4b35f47d8-maas-api-tls") pod "maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6" (UID: "56223f26-9288-45cd-867b-76f4b35f47d8") : secret "maas-api-e2e-api-a-3d7651-serving-cert" not found Jul 06 20:39:15.509754 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:15.509730 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56223f26-9288-45cd-867b-76f4b35f47d8-openshift-service-ca\") pod \"maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6\" (UID: \"56223f26-9288-45cd-867b-76f4b35f47d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6" Jul 06 20:39:15.522219 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:15.522193 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhzc6\" (UniqueName: \"kubernetes.io/projected/56223f26-9288-45cd-867b-76f4b35f47d8-kube-api-access-bhzc6\") pod \"maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6\" (UID: \"56223f26-9288-45cd-867b-76f4b35f47d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6" Jul 06 20:39:15.704031 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:15.703988 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:15.894325 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:15.894299 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:15.993062 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:15.993028 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:16.013758 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:16.013730 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56223f26-9288-45cd-867b-76f4b35f47d8-maas-api-tls\") pod \"maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6\" (UID: \"56223f26-9288-45cd-867b-76f4b35f47d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6" Jul 06 20:39:16.016281 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:16.016262 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56223f26-9288-45cd-867b-76f4b35f47d8-maas-api-tls\") pod \"maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6\" (UID: \"56223f26-9288-45cd-867b-76f4b35f47d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6" Jul 06 20:39:16.203156 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:16.203060 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6" Jul 06 20:39:16.344508 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:16.344485 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6"] Jul 06 20:39:16.347215 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:39:16.347179 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56223f26_9288_45cd_867b_76f4b35f47d8.slice/crio-6348eabc7101134e52ff80c5cd5a6306dc1c71f710032afa70fefc6d7bdda35c WatchSource:0}: Error finding container 6348eabc7101134e52ff80c5cd5a6306dc1c71f710032afa70fefc6d7bdda35c: Status 404 returned error can't find the container with id 6348eabc7101134e52ff80c5cd5a6306dc1c71f710032afa70fefc6d7bdda35c Jul 06 20:39:16.556547 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:16.556448 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6" event={"ID":"56223f26-9288-45cd-867b-76f4b35f47d8","Type":"ContainerStarted","Data":"6348eabc7101134e52ff80c5cd5a6306dc1c71f710032afa70fefc6d7bdda35c"} Jul 06 20:39:17.561813 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:17.561778 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6" event={"ID":"56223f26-9288-45cd-867b-76f4b35f47d8","Type":"ContainerStarted","Data":"4395226f52b8f96b26dad1a75f1700e018ab4e1402e731f94c87db0f067dcc06"} Jul 06 20:39:17.562276 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:17.561855 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6" Jul 06 20:39:17.584961 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:17.584905 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6" podStartSLOduration=2.122482469 podStartE2EDuration="2.584884604s" totalImagesPullingTime="462.402135ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:39:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:39:17.582378606 +0000 UTC m=+2008.793526824" watchObservedRunningTime="2026-07-06 20:39:17.584884604 +0000 UTC m=+2008.796032823" Jul 06 20:39:19.664779 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:19.664731 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:20.131009 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:20.130913 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:20.519145 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:20.519105 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:20.814312 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:20.814214 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:23.255147 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:23.255111 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:23.344865 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:23.344826 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9"] Jul 06 20:39:23.345802 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:23.345764 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:23.348631 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:23.348609 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9" Jul 06 20:39:23.353399 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:23.353377 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-86cb5d-serving-cert\"" Jul 06 20:39:23.360584 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:23.360557 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9"] Jul 06 20:39:23.412709 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:23.412626 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:23.481946 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:23.481914 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5d879cd-c380-4810-9599-093a58694200-openshift-service-ca\") pod \"maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9\" (UID: \"f5d879cd-c380-4810-9599-093a58694200\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9" Jul 06 20:39:23.482119 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:23.481959 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fklj2\" (UniqueName: \"kubernetes.io/projected/f5d879cd-c380-4810-9599-093a58694200-kube-api-access-fklj2\") pod \"maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9\" (UID: \"f5d879cd-c380-4810-9599-093a58694200\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9" Jul 06 20:39:23.482119 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:23.481998 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5d879cd-c380-4810-9599-093a58694200-maas-api-tls\") pod \"maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9\" (UID: \"f5d879cd-c380-4810-9599-093a58694200\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9" Jul 06 20:39:23.572368 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:23.572297 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6" Jul 06 20:39:23.583366 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:23.583341 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5d879cd-c380-4810-9599-093a58694200-openshift-service-ca\") pod \"maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9\" (UID: \"f5d879cd-c380-4810-9599-093a58694200\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9" Jul 06 20:39:23.583553 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:23.583382 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fklj2\" (UniqueName: \"kubernetes.io/projected/f5d879cd-c380-4810-9599-093a58694200-kube-api-access-fklj2\") pod \"maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9\" (UID: \"f5d879cd-c380-4810-9599-093a58694200\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9" Jul 06 20:39:23.583553 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:23.583403 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5d879cd-c380-4810-9599-093a58694200-maas-api-tls\") pod \"maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9\" (UID: \"f5d879cd-c380-4810-9599-093a58694200\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9" Jul 06 20:39:23.583998 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:23.583977 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5d879cd-c380-4810-9599-093a58694200-openshift-service-ca\") pod \"maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9\" (UID: \"f5d879cd-c380-4810-9599-093a58694200\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9" Jul 06 20:39:23.586359 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:23.586340 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5d879cd-c380-4810-9599-093a58694200-maas-api-tls\") pod \"maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9\" (UID: \"f5d879cd-c380-4810-9599-093a58694200\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9" Jul 06 20:39:23.601406 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:23.601380 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fklj2\" (UniqueName: \"kubernetes.io/projected/f5d879cd-c380-4810-9599-093a58694200-kube-api-access-fklj2\") pod \"maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9\" (UID: \"f5d879cd-c380-4810-9599-093a58694200\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9" Jul 06 20:39:23.655448 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:23.655414 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:23.661363 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:23.661343 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9" Jul 06 20:39:23.799990 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:23.799965 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9"] Jul 06 20:39:23.802374 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:39:23.802344 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d879cd_c380_4810_9599_093a58694200.slice/crio-9c1621b541c0d42cd902d917ecf358215c3e829f58165095e0cba1c4c43af094 WatchSource:0}: Error finding container 9c1621b541c0d42cd902d917ecf358215c3e829f58165095e0cba1c4c43af094: Status 404 returned error can't find the container with id 9c1621b541c0d42cd902d917ecf358215c3e829f58165095e0cba1c4c43af094 Jul 06 20:39:24.026722 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:24.026687 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:24.122283 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:24.122249 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:24.592734 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:24.592695 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9" event={"ID":"f5d879cd-c380-4810-9599-093a58694200","Type":"ContainerStarted","Data":"02bada530404f4bede59348ff11660d4d9d1a7c08effbe58b221935b375a615f"} Jul 06 20:39:24.593193 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:24.592763 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9" event={"ID":"f5d879cd-c380-4810-9599-093a58694200","Type":"ContainerStarted","Data":"9c1621b541c0d42cd902d917ecf358215c3e829f58165095e0cba1c4c43af094"} Jul 06 20:39:24.593193 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:24.592803 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9" Jul 06 20:39:24.614445 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:24.614398 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9" podStartSLOduration=1.241965872 podStartE2EDuration="1.614381568s" totalImagesPullingTime="372.415696ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:39:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:39:24.613829612 +0000 UTC m=+2015.824977847" watchObservedRunningTime="2026-07-06 20:39:24.614381568 +0000 UTC m=+2015.825529786" Jul 06 20:39:27.728561 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:27.728524 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:30.601979 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:30.601950 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9" Jul 06 20:39:35.901425 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:35.901387 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:36.248658 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:36.248576 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:36.411625 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:36.411590 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:36.500643 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:36.500562 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:36.727021 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:36.726986 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6"] Jul 06 20:39:36.727498 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:36.727447 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6" podUID="56223f26-9288-45cd-867b-76f4b35f47d8" containerName="maas-api" containerID="cri-o://4395226f52b8f96b26dad1a75f1700e018ab4e1402e731f94c87db0f067dcc06" gracePeriod=30 Jul 06 20:39:36.755172 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:36.755151 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:36.987586 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:36.987564 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6" Jul 06 20:39:37.099837 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:37.099739 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhzc6\" (UniqueName: \"kubernetes.io/projected/56223f26-9288-45cd-867b-76f4b35f47d8-kube-api-access-bhzc6\") pod \"56223f26-9288-45cd-867b-76f4b35f47d8\" (UID: \"56223f26-9288-45cd-867b-76f4b35f47d8\") " Jul 06 20:39:37.099837 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:37.099789 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56223f26-9288-45cd-867b-76f4b35f47d8-openshift-service-ca\") pod \"56223f26-9288-45cd-867b-76f4b35f47d8\" (UID: \"56223f26-9288-45cd-867b-76f4b35f47d8\") " Jul 06 20:39:37.100066 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:37.099868 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56223f26-9288-45cd-867b-76f4b35f47d8-maas-api-tls\") pod \"56223f26-9288-45cd-867b-76f4b35f47d8\" (UID: \"56223f26-9288-45cd-867b-76f4b35f47d8\") " Jul 06 20:39:37.100213 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:37.100177 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56223f26-9288-45cd-867b-76f4b35f47d8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "56223f26-9288-45cd-867b-76f4b35f47d8" (UID: "56223f26-9288-45cd-867b-76f4b35f47d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:39:37.102038 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:37.102013 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56223f26-9288-45cd-867b-76f4b35f47d8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "56223f26-9288-45cd-867b-76f4b35f47d8" (UID: "56223f26-9288-45cd-867b-76f4b35f47d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:39:37.102145 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:37.102128 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56223f26-9288-45cd-867b-76f4b35f47d8-kube-api-access-bhzc6" (OuterVolumeSpecName: "kube-api-access-bhzc6") pod "56223f26-9288-45cd-867b-76f4b35f47d8" (UID: "56223f26-9288-45cd-867b-76f4b35f47d8"). InnerVolumeSpecName "kube-api-access-bhzc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:39:37.201562 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:37.201522 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56223f26-9288-45cd-867b-76f4b35f47d8-maas-api-tls\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:39:37.201562 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:37.201556 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhzc6\" (UniqueName: \"kubernetes.io/projected/56223f26-9288-45cd-867b-76f4b35f47d8-kube-api-access-bhzc6\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:39:37.201562 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:37.201569 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56223f26-9288-45cd-867b-76f4b35f47d8-openshift-service-ca\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:39:37.651688 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:37.651656 2578 generic.go:356] "Generic (PLEG): container finished" podID="56223f26-9288-45cd-867b-76f4b35f47d8" containerID="4395226f52b8f96b26dad1a75f1700e018ab4e1402e731f94c87db0f067dcc06" exitCode=0 Jul 06 20:39:37.651857 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:37.651691 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6" event={"ID":"56223f26-9288-45cd-867b-76f4b35f47d8","Type":"ContainerDied","Data":"4395226f52b8f96b26dad1a75f1700e018ab4e1402e731f94c87db0f067dcc06"} Jul 06 20:39:37.651857 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:37.651706 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6" Jul 06 20:39:37.651857 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:37.651745 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6" event={"ID":"56223f26-9288-45cd-867b-76f4b35f47d8","Type":"ContainerDied","Data":"6348eabc7101134e52ff80c5cd5a6306dc1c71f710032afa70fefc6d7bdda35c"} Jul 06 20:39:37.651857 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:37.651768 2578 scope.go:117] "RemoveContainer" containerID="4395226f52b8f96b26dad1a75f1700e018ab4e1402e731f94c87db0f067dcc06" Jul 06 20:39:37.660255 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:37.660238 2578 scope.go:117] "RemoveContainer" containerID="4395226f52b8f96b26dad1a75f1700e018ab4e1402e731f94c87db0f067dcc06" Jul 06 20:39:37.660514 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:39:37.660493 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4395226f52b8f96b26dad1a75f1700e018ab4e1402e731f94c87db0f067dcc06\": container with ID starting with 4395226f52b8f96b26dad1a75f1700e018ab4e1402e731f94c87db0f067dcc06 not found: ID does not exist" containerID="4395226f52b8f96b26dad1a75f1700e018ab4e1402e731f94c87db0f067dcc06" Jul 06 20:39:37.660573 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:37.660526 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4395226f52b8f96b26dad1a75f1700e018ab4e1402e731f94c87db0f067dcc06"} err="failed to get container status \"4395226f52b8f96b26dad1a75f1700e018ab4e1402e731f94c87db0f067dcc06\": rpc error: code = NotFound desc = could not find container \"4395226f52b8f96b26dad1a75f1700e018ab4e1402e731f94c87db0f067dcc06\": container with ID starting with 4395226f52b8f96b26dad1a75f1700e018ab4e1402e731f94c87db0f067dcc06 not found: ID does not exist" Jul 06 20:39:37.680481 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:37.680434 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6"] Jul 06 20:39:37.685593 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:37.685566 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3d7651-865b758d9c-pvqf6"] Jul 06 20:39:39.299826 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:39.299793 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56223f26-9288-45cd-867b-76f4b35f47d8" path="/var/lib/kubelet/pods/56223f26-9288-45cd-867b-76f4b35f47d8/volumes" Jul 06 20:39:42.773111 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:42.773062 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:42.837990 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:42.837961 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:43.035743 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:43.035646 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:43.558578 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:43.558540 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:43.908248 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:43.908213 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:44.024763 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:44.024725 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:44.125194 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:44.125152 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:44.302985 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:44.302899 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9"] Jul 06 20:39:44.303149 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:44.303104 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9" podUID="f5d879cd-c380-4810-9599-093a58694200" containerName="maas-api" containerID="cri-o://02bada530404f4bede59348ff11660d4d9d1a7c08effbe58b221935b375a615f" gracePeriod=30 Jul 06 20:39:44.337496 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:44.337420 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:44.556823 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:44.556758 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9" Jul 06 20:39:44.666911 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:44.666869 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5d879cd-c380-4810-9599-093a58694200-maas-api-tls\") pod \"f5d879cd-c380-4810-9599-093a58694200\" (UID: \"f5d879cd-c380-4810-9599-093a58694200\") " Jul 06 20:39:44.667115 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:44.666981 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fklj2\" (UniqueName: \"kubernetes.io/projected/f5d879cd-c380-4810-9599-093a58694200-kube-api-access-fklj2\") pod \"f5d879cd-c380-4810-9599-093a58694200\" (UID: \"f5d879cd-c380-4810-9599-093a58694200\") " Jul 06 20:39:44.667115 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:44.667029 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5d879cd-c380-4810-9599-093a58694200-openshift-service-ca\") pod \"f5d879cd-c380-4810-9599-093a58694200\" (UID: \"f5d879cd-c380-4810-9599-093a58694200\") " Jul 06 20:39:44.667435 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:44.667400 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5d879cd-c380-4810-9599-093a58694200-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f5d879cd-c380-4810-9599-093a58694200" (UID: "f5d879cd-c380-4810-9599-093a58694200"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:39:44.669176 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:44.669153 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5d879cd-c380-4810-9599-093a58694200-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f5d879cd-c380-4810-9599-093a58694200" (UID: "f5d879cd-c380-4810-9599-093a58694200"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:39:44.669176 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:44.669159 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5d879cd-c380-4810-9599-093a58694200-kube-api-access-fklj2" (OuterVolumeSpecName: "kube-api-access-fklj2") pod "f5d879cd-c380-4810-9599-093a58694200" (UID: "f5d879cd-c380-4810-9599-093a58694200"). InnerVolumeSpecName "kube-api-access-fklj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:39:44.684338 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:44.684308 2578 generic.go:356] "Generic (PLEG): container finished" podID="f5d879cd-c380-4810-9599-093a58694200" containerID="02bada530404f4bede59348ff11660d4d9d1a7c08effbe58b221935b375a615f" exitCode=0 Jul 06 20:39:44.684503 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:44.684373 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9" Jul 06 20:39:44.684503 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:44.684384 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9" event={"ID":"f5d879cd-c380-4810-9599-093a58694200","Type":"ContainerDied","Data":"02bada530404f4bede59348ff11660d4d9d1a7c08effbe58b221935b375a615f"} Jul 06 20:39:44.684503 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:44.684483 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9" event={"ID":"f5d879cd-c380-4810-9599-093a58694200","Type":"ContainerDied","Data":"9c1621b541c0d42cd902d917ecf358215c3e829f58165095e0cba1c4c43af094"} Jul 06 20:39:44.684667 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:44.684507 2578 scope.go:117] "RemoveContainer" containerID="02bada530404f4bede59348ff11660d4d9d1a7c08effbe58b221935b375a615f" Jul 06 20:39:44.693423 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:44.693407 2578 scope.go:117] "RemoveContainer" containerID="02bada530404f4bede59348ff11660d4d9d1a7c08effbe58b221935b375a615f" Jul 06 20:39:44.693691 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:39:44.693671 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02bada530404f4bede59348ff11660d4d9d1a7c08effbe58b221935b375a615f\": container with ID starting with 02bada530404f4bede59348ff11660d4d9d1a7c08effbe58b221935b375a615f not found: ID does not exist" containerID="02bada530404f4bede59348ff11660d4d9d1a7c08effbe58b221935b375a615f" Jul 06 20:39:44.693747 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:44.693698 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02bada530404f4bede59348ff11660d4d9d1a7c08effbe58b221935b375a615f"} err="failed to get container status \"02bada530404f4bede59348ff11660d4d9d1a7c08effbe58b221935b375a615f\": rpc error: code = NotFound desc = could not find container \"02bada530404f4bede59348ff11660d4d9d1a7c08effbe58b221935b375a615f\": container with ID starting with 02bada530404f4bede59348ff11660d4d9d1a7c08effbe58b221935b375a615f not found: ID does not exist" Jul 06 20:39:44.708297 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:44.708269 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9"] Jul 06 20:39:44.714479 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:44.714443 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-86cb5d-6d95f9888b-7bmf9"] Jul 06 20:39:44.768517 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:44.768486 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fklj2\" (UniqueName: \"kubernetes.io/projected/f5d879cd-c380-4810-9599-093a58694200-kube-api-access-fklj2\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:39:44.768517 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:44.768513 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5d879cd-c380-4810-9599-093a58694200-openshift-service-ca\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:39:44.768517 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:44.768525 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5d879cd-c380-4810-9599-093a58694200-maas-api-tls\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:39:45.299326 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:45.299290 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5d879cd-c380-4810-9599-093a58694200" path="/var/lib/kubelet/pods/f5d879cd-c380-4810-9599-093a58694200/volumes" Jul 06 20:39:50.416574 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:50.416531 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:50.474064 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:50.474031 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:50.687738 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:50.687668 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:51.226396 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:51.226363 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:51.572743 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:51.572651 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:51.788046 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:51.788010 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:54.418242 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:54.418201 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:54.497609 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:54.497573 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:54.498390 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:54.498371 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t"] Jul 06 20:39:54.498724 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:54.498711 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56223f26-9288-45cd-867b-76f4b35f47d8" containerName="maas-api" Jul 06 20:39:54.498771 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:54.498728 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="56223f26-9288-45cd-867b-76f4b35f47d8" containerName="maas-api" Jul 06 20:39:54.498771 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:54.498739 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5d879cd-c380-4810-9599-093a58694200" containerName="maas-api" Jul 06 20:39:54.498771 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:54.498745 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5d879cd-c380-4810-9599-093a58694200" containerName="maas-api" Jul 06 20:39:54.498904 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:54.498793 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5d879cd-c380-4810-9599-093a58694200" containerName="maas-api" Jul 06 20:39:54.498904 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:54.498805 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="56223f26-9288-45cd-867b-76f4b35f47d8" containerName="maas-api" Jul 06 20:39:54.501650 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:54.501629 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t" Jul 06 20:39:54.505642 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:54.505624 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-37e178-serving-cert\"" Jul 06 20:39:54.513860 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:54.513838 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t"] Jul 06 20:39:54.553650 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:54.553621 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70a3994a-2423-4402-a6ef-b92e75569998-maas-api-tls\") pod \"maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t\" (UID: \"70a3994a-2423-4402-a6ef-b92e75569998\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t" Jul 06 20:39:54.553801 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:54.553706 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70a3994a-2423-4402-a6ef-b92e75569998-openshift-service-ca\") pod \"maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t\" (UID: \"70a3994a-2423-4402-a6ef-b92e75569998\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t" Jul 06 20:39:54.553801 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:54.553734 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l97gj\" (UniqueName: \"kubernetes.io/projected/70a3994a-2423-4402-a6ef-b92e75569998-kube-api-access-l97gj\") pod \"maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t\" (UID: \"70a3994a-2423-4402-a6ef-b92e75569998\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t" Jul 06 20:39:54.572325 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:54.572289 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:54.654586 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:54.654545 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70a3994a-2423-4402-a6ef-b92e75569998-openshift-service-ca\") pod \"maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t\" (UID: \"70a3994a-2423-4402-a6ef-b92e75569998\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t" Jul 06 20:39:54.654586 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:54.654592 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l97gj\" (UniqueName: \"kubernetes.io/projected/70a3994a-2423-4402-a6ef-b92e75569998-kube-api-access-l97gj\") pod \"maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t\" (UID: \"70a3994a-2423-4402-a6ef-b92e75569998\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t" Jul 06 20:39:54.654795 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:54.654618 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70a3994a-2423-4402-a6ef-b92e75569998-maas-api-tls\") pod \"maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t\" (UID: \"70a3994a-2423-4402-a6ef-b92e75569998\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t" Jul 06 20:39:54.654795 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:39:54.654713 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-37e178-serving-cert: secret "maas-api-e2e-shared-a-37e178-serving-cert" not found Jul 06 20:39:54.654795 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:39:54.654778 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70a3994a-2423-4402-a6ef-b92e75569998-maas-api-tls podName:70a3994a-2423-4402-a6ef-b92e75569998 nodeName:}" failed. No retries permitted until 2026-07-06 20:39:55.154763331 +0000 UTC m=+2046.365911527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/70a3994a-2423-4402-a6ef-b92e75569998-maas-api-tls") pod "maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t" (UID: "70a3994a-2423-4402-a6ef-b92e75569998") : secret "maas-api-e2e-shared-a-37e178-serving-cert" not found Jul 06 20:39:54.655195 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:54.655177 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70a3994a-2423-4402-a6ef-b92e75569998-openshift-service-ca\") pod \"maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t\" (UID: \"70a3994a-2423-4402-a6ef-b92e75569998\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t" Jul 06 20:39:54.664728 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:54.664700 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l97gj\" (UniqueName: \"kubernetes.io/projected/70a3994a-2423-4402-a6ef-b92e75569998-kube-api-access-l97gj\") pod \"maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t\" (UID: \"70a3994a-2423-4402-a6ef-b92e75569998\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t" Jul 06 20:39:55.007344 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:55.007305 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:55.107255 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:55.107220 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:55.158929 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:55.158896 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70a3994a-2423-4402-a6ef-b92e75569998-maas-api-tls\") pod \"maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t\" (UID: \"70a3994a-2423-4402-a6ef-b92e75569998\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t" Jul 06 20:39:55.161429 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:55.161408 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70a3994a-2423-4402-a6ef-b92e75569998-maas-api-tls\") pod \"maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t\" (UID: \"70a3994a-2423-4402-a6ef-b92e75569998\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t" Jul 06 20:39:55.412981 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:55.412947 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t" Jul 06 20:39:55.550108 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:55.550076 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t"] Jul 06 20:39:55.551523 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:39:55.551496 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70a3994a_2423_4402_a6ef_b92e75569998.slice/crio-9ba2fcca60560d55a788ed6a6250f534a3af1c83d710b8ec1026eeb67f711519 WatchSource:0}: Error finding container 9ba2fcca60560d55a788ed6a6250f534a3af1c83d710b8ec1026eeb67f711519: Status 404 returned error can't find the container with id 9ba2fcca60560d55a788ed6a6250f534a3af1c83d710b8ec1026eeb67f711519 Jul 06 20:39:55.731780 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:55.731681 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t" event={"ID":"70a3994a-2423-4402-a6ef-b92e75569998","Type":"ContainerStarted","Data":"9ba2fcca60560d55a788ed6a6250f534a3af1c83d710b8ec1026eeb67f711519"} Jul 06 20:39:56.737930 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:56.737900 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t" event={"ID":"70a3994a-2423-4402-a6ef-b92e75569998","Type":"ContainerStarted","Data":"f9a9b8b547aa7bd0d8fe2f0a5cfaa38f62eabf881544ab626b8b174a63ec864d"} Jul 06 20:39:56.738330 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:56.737962 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t" Jul 06 20:39:56.769201 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:56.769150 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t" podStartSLOduration=2.319515268 podStartE2EDuration="2.769134741s" totalImagesPullingTime="449.619473ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:39:56.768327515 +0000 UTC m=+2047.979475734" watchObservedRunningTime="2026-07-06 20:39:56.769134741 +0000 UTC m=+2047.980282958" Jul 06 20:39:58.916002 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:58.915966 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:59.249562 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:59.249413 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:59.652696 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:39:59.652650 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:00.038880 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:00.038776 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:02.461305 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:02.461275 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:02.528605 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:02.528556 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:02.675281 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:02.675249 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6"] Jul 06 20:40:02.678819 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:02.678777 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6" Jul 06 20:40:02.692221 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:02.692199 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-67755c-serving-cert\"" Jul 06 20:40:02.698484 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:02.698437 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6"] Jul 06 20:40:02.724525 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:02.724427 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f23a0f93-b30c-4f6a-bcec-192e94541eac-maas-api-tls\") pod \"maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6\" (UID: \"f23a0f93-b30c-4f6a-bcec-192e94541eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6" Jul 06 20:40:02.724682 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:02.724555 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f23a0f93-b30c-4f6a-bcec-192e94541eac-openshift-service-ca\") pod \"maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6\" (UID: \"f23a0f93-b30c-4f6a-bcec-192e94541eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6" Jul 06 20:40:02.724682 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:02.724608 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhwl7\" (UniqueName: \"kubernetes.io/projected/f23a0f93-b30c-4f6a-bcec-192e94541eac-kube-api-access-mhwl7\") pod \"maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6\" (UID: \"f23a0f93-b30c-4f6a-bcec-192e94541eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6" Jul 06 20:40:02.725031 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:02.725009 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:02.747539 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:02.747514 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t" Jul 06 20:40:02.825364 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:02.825333 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f23a0f93-b30c-4f6a-bcec-192e94541eac-maas-api-tls\") pod \"maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6\" (UID: \"f23a0f93-b30c-4f6a-bcec-192e94541eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6" Jul 06 20:40:02.825559 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:02.825410 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f23a0f93-b30c-4f6a-bcec-192e94541eac-openshift-service-ca\") pod \"maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6\" (UID: \"f23a0f93-b30c-4f6a-bcec-192e94541eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6" Jul 06 20:40:02.825718 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:02.825688 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhwl7\" (UniqueName: \"kubernetes.io/projected/f23a0f93-b30c-4f6a-bcec-192e94541eac-kube-api-access-mhwl7\") pod \"maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6\" (UID: \"f23a0f93-b30c-4f6a-bcec-192e94541eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6" Jul 06 20:40:02.826395 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:02.826371 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f23a0f93-b30c-4f6a-bcec-192e94541eac-openshift-service-ca\") pod \"maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6\" (UID: \"f23a0f93-b30c-4f6a-bcec-192e94541eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6" Jul 06 20:40:02.828342 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:02.828316 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f23a0f93-b30c-4f6a-bcec-192e94541eac-maas-api-tls\") pod \"maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6\" (UID: \"f23a0f93-b30c-4f6a-bcec-192e94541eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6" Jul 06 20:40:02.842417 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:02.842390 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:02.848810 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:02.848784 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhwl7\" (UniqueName: \"kubernetes.io/projected/f23a0f93-b30c-4f6a-bcec-192e94541eac-kube-api-access-mhwl7\") pod \"maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6\" (UID: \"f23a0f93-b30c-4f6a-bcec-192e94541eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6" Jul 06 20:40:02.990184 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:02.990081 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6" Jul 06 20:40:03.127236 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:03.127213 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6"] Jul 06 20:40:03.129343 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:40:03.129311 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf23a0f93_b30c_4f6a_bcec_192e94541eac.slice/crio-4f2fced15ef2a6deaadd6c9f179ab448d87316e034a77ebc0b7bd4ce2a7da671 WatchSource:0}: Error finding container 4f2fced15ef2a6deaadd6c9f179ab448d87316e034a77ebc0b7bd4ce2a7da671: Status 404 returned error can't find the container with id 4f2fced15ef2a6deaadd6c9f179ab448d87316e034a77ebc0b7bd4ce2a7da671 Jul 06 20:40:03.464708 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:03.464666 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:03.563273 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:03.563223 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:03.771015 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:03.770918 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6" event={"ID":"f23a0f93-b30c-4f6a-bcec-192e94541eac","Type":"ContainerStarted","Data":"72ec774c8377b9228975b3081b8aaed8da6b1221f1a8554e19cf94ebebd59e11"} Jul 06 20:40:03.771015 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:03.770968 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6" event={"ID":"f23a0f93-b30c-4f6a-bcec-192e94541eac","Type":"ContainerStarted","Data":"4f2fced15ef2a6deaadd6c9f179ab448d87316e034a77ebc0b7bd4ce2a7da671"} Jul 06 20:40:03.771015 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:03.771006 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6" Jul 06 20:40:03.802435 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:03.802372 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6" podStartSLOduration=1.352629936 podStartE2EDuration="1.802353389s" totalImagesPullingTime="449.723453ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:40:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:40:03.800597363 +0000 UTC m=+2055.011745571" watchObservedRunningTime="2026-07-06 20:40:03.802353389 +0000 UTC m=+2055.013501606" Jul 06 20:40:06.948347 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:06.948313 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:09.780839 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:09.780760 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6" Jul 06 20:40:11.925889 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:11.925834 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:15.776085 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:15.776050 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:15.876024 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:15.875971 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:25.371594 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:25.371560 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:25.469602 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:25.469557 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:30.742124 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:30.742077 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:49.329701 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:49.329672 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dnfgp_0f4c3a8a-30c4-428e-88a6-dcf086ef8615/ovn-acl-logging/0.log" Jul 06 20:40:49.333115 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:40:49.333095 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dnfgp_0f4c3a8a-30c4-428e-88a6-dcf086ef8615/ovn-acl-logging/0.log" Jul 06 20:41:10.728847 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:41:10.728803 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:41:10.831732 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:41:10.831694 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:41:29.339442 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:41:29.339404 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:27.264094 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:42:27.264053 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:38.730798 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:42:38.730762 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:39.627427 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:42:39.627385 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:54.158179 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:42:54.158140 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:43:26.523279 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:43:26.523245 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:43:30.004017 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:43:30.003983 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:44:18.452226 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:44:18.452174 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:44:19.823579 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:44:19.823541 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:44:34.534495 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:44:34.534432 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:00.150398 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:45:00.150363 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29722845-7h4gq"] Jul 06 20:45:00.153807 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:45:00.153783 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29722845-7h4gq" Jul 06 20:45:00.170134 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:45:00.170106 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29722845-7h4gq"] Jul 06 20:45:00.181424 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:45:00.181392 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hnms\" (UniqueName: \"kubernetes.io/projected/4e68323e-1267-4637-8956-c7dcd4385496-kube-api-access-6hnms\") pod \"maas-api-key-cleanup-29722845-7h4gq\" (UID: \"4e68323e-1267-4637-8956-c7dcd4385496\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29722845-7h4gq" Jul 06 20:45:00.282199 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:45:00.282153 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hnms\" (UniqueName: \"kubernetes.io/projected/4e68323e-1267-4637-8956-c7dcd4385496-kube-api-access-6hnms\") pod \"maas-api-key-cleanup-29722845-7h4gq\" (UID: \"4e68323e-1267-4637-8956-c7dcd4385496\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29722845-7h4gq" Jul 06 20:45:00.292739 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:45:00.292710 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hnms\" (UniqueName: \"kubernetes.io/projected/4e68323e-1267-4637-8956-c7dcd4385496-kube-api-access-6hnms\") pod \"maas-api-key-cleanup-29722845-7h4gq\" (UID: \"4e68323e-1267-4637-8956-c7dcd4385496\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29722845-7h4gq" Jul 06 20:45:00.463128 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:45:00.463037 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29722845-7h4gq" Jul 06 20:45:00.600757 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:45:00.600729 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29722845-7h4gq"] Jul 06 20:45:00.603221 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:45:00.603185 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e68323e_1267_4637_8956_c7dcd4385496.slice/crio-4271ead7021c70bde9bdc4654d8fbca41b0df677a617d09f64e5b5989f575d8f WatchSource:0}: Error finding container 4271ead7021c70bde9bdc4654d8fbca41b0df677a617d09f64e5b5989f575d8f: Status 404 returned error can't find the container with id 4271ead7021c70bde9bdc4654d8fbca41b0df677a617d09f64e5b5989f575d8f Jul 06 20:45:00.604917 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:45:00.604901 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 20:45:01.054772 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:45:01.054742 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29722845-7h4gq" event={"ID":"4e68323e-1267-4637-8956-c7dcd4385496","Type":"ContainerStarted","Data":"4271ead7021c70bde9bdc4654d8fbca41b0df677a617d09f64e5b5989f575d8f"} Jul 06 20:45:03.065951 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:45:03.065847 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29722845-7h4gq" event={"ID":"4e68323e-1267-4637-8956-c7dcd4385496","Type":"ContainerStarted","Data":"665cee1a318bcf8c6bb415245b4f93de856b662a3add1008fab3cc424bbbfcb8"} Jul 06 20:45:03.088878 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:45:03.088830 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29722845-7h4gq" podStartSLOduration=0.97353844 podStartE2EDuration="3.088811509s" totalImagesPullingTime="2.115273069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:45:03.086635591 +0000 UTC m=+2354.297783809" watchObservedRunningTime="2026-07-06 20:45:03.088811509 +0000 UTC m=+2354.299959727" Jul 06 20:45:16.159367 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:45:16.159328 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:16.578510 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:45:16.578397 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:16.636426 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:45:16.636384 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:26.362261 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:45:26.362226 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:26.466483 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:45:26.466426 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:49.355189 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:45:49.355158 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dnfgp_0f4c3a8a-30c4-428e-88a6-dcf086ef8615/ovn-acl-logging/0.log" Jul 06 20:45:49.361941 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:45:49.361919 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dnfgp_0f4c3a8a-30c4-428e-88a6-dcf086ef8615/ovn-acl-logging/0.log" Jul 06 20:45:58.730027 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:45:58.729988 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:59.599097 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:45:59.594448 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:00.017209 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:00.017170 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29722845-7h4gq"] Jul 06 20:46:00.017753 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:00.017472 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29722845-7h4gq" podUID="4e68323e-1267-4637-8956-c7dcd4385496" containerName="cleanup" containerID="cri-o://665cee1a318bcf8c6bb415245b4f93de856b662a3add1008fab3cc424bbbfcb8" gracePeriod=30 Jul 06 20:46:00.634097 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:00.634061 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:01.031448 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:01.031411 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:01.150087 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:01.150056 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:01.247595 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:01.247563 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:01.380154 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:01.380043 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t"] Jul 06 20:46:01.380326 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:01.380286 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t" podUID="70a3994a-2423-4402-a6ef-b92e75569998" containerName="maas-api" containerID="cri-o://f9a9b8b547aa7bd0d8fe2f0a5cfaa38f62eabf881544ab626b8b174a63ec864d" gracePeriod=30 Jul 06 20:46:01.406266 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:01.406227 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:01.681676 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:46:01.681633 2578 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70a3994a_2423_4402_a6ef_b92e75569998.slice/crio-f9a9b8b547aa7bd0d8fe2f0a5cfaa38f62eabf881544ab626b8b174a63ec864d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70a3994a_2423_4402_a6ef_b92e75569998.slice/crio-conmon-f9a9b8b547aa7bd0d8fe2f0a5cfaa38f62eabf881544ab626b8b174a63ec864d.scope\": RecentStats: unable to find data in memory cache]" Jul 06 20:46:01.822362 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:01.822339 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t" Jul 06 20:46:01.949310 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:01.949213 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70a3994a-2423-4402-a6ef-b92e75569998-maas-api-tls\") pod \"70a3994a-2423-4402-a6ef-b92e75569998\" (UID: \"70a3994a-2423-4402-a6ef-b92e75569998\") " Jul 06 20:46:01.949310 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:01.949271 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l97gj\" (UniqueName: \"kubernetes.io/projected/70a3994a-2423-4402-a6ef-b92e75569998-kube-api-access-l97gj\") pod \"70a3994a-2423-4402-a6ef-b92e75569998\" (UID: \"70a3994a-2423-4402-a6ef-b92e75569998\") " Jul 06 20:46:01.949546 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:01.949322 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70a3994a-2423-4402-a6ef-b92e75569998-openshift-service-ca\") pod \"70a3994a-2423-4402-a6ef-b92e75569998\" (UID: \"70a3994a-2423-4402-a6ef-b92e75569998\") " Jul 06 20:46:01.949770 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:01.949744 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70a3994a-2423-4402-a6ef-b92e75569998-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "70a3994a-2423-4402-a6ef-b92e75569998" (UID: "70a3994a-2423-4402-a6ef-b92e75569998"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:46:01.951598 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:01.951570 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70a3994a-2423-4402-a6ef-b92e75569998-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "70a3994a-2423-4402-a6ef-b92e75569998" (UID: "70a3994a-2423-4402-a6ef-b92e75569998"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:46:01.951598 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:01.951585 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70a3994a-2423-4402-a6ef-b92e75569998-kube-api-access-l97gj" (OuterVolumeSpecName: "kube-api-access-l97gj") pod "70a3994a-2423-4402-a6ef-b92e75569998" (UID: "70a3994a-2423-4402-a6ef-b92e75569998"). InnerVolumeSpecName "kube-api-access-l97gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:46:02.050350 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:02.050303 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70a3994a-2423-4402-a6ef-b92e75569998-openshift-service-ca\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:46:02.050350 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:02.050336 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70a3994a-2423-4402-a6ef-b92e75569998-maas-api-tls\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:46:02.050350 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:02.050348 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l97gj\" (UniqueName: \"kubernetes.io/projected/70a3994a-2423-4402-a6ef-b92e75569998-kube-api-access-l97gj\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:46:02.327401 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:02.327313 2578 generic.go:356] "Generic (PLEG): container finished" podID="70a3994a-2423-4402-a6ef-b92e75569998" containerID="f9a9b8b547aa7bd0d8fe2f0a5cfaa38f62eabf881544ab626b8b174a63ec864d" exitCode=0 Jul 06 20:46:02.327401 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:02.327359 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t" Jul 06 20:46:02.327658 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:02.327409 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t" event={"ID":"70a3994a-2423-4402-a6ef-b92e75569998","Type":"ContainerDied","Data":"f9a9b8b547aa7bd0d8fe2f0a5cfaa38f62eabf881544ab626b8b174a63ec864d"} Jul 06 20:46:02.327658 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:02.327488 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t" event={"ID":"70a3994a-2423-4402-a6ef-b92e75569998","Type":"ContainerDied","Data":"9ba2fcca60560d55a788ed6a6250f534a3af1c83d710b8ec1026eeb67f711519"} Jul 06 20:46:02.327658 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:02.327520 2578 scope.go:117] "RemoveContainer" containerID="f9a9b8b547aa7bd0d8fe2f0a5cfaa38f62eabf881544ab626b8b174a63ec864d" Jul 06 20:46:02.336536 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:02.336517 2578 scope.go:117] "RemoveContainer" containerID="f9a9b8b547aa7bd0d8fe2f0a5cfaa38f62eabf881544ab626b8b174a63ec864d" Jul 06 20:46:02.336837 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:46:02.336817 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9a9b8b547aa7bd0d8fe2f0a5cfaa38f62eabf881544ab626b8b174a63ec864d\": container with ID starting with f9a9b8b547aa7bd0d8fe2f0a5cfaa38f62eabf881544ab626b8b174a63ec864d not found: ID does not exist" containerID="f9a9b8b547aa7bd0d8fe2f0a5cfaa38f62eabf881544ab626b8b174a63ec864d" Jul 06 20:46:02.336887 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:02.336849 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9a9b8b547aa7bd0d8fe2f0a5cfaa38f62eabf881544ab626b8b174a63ec864d"} err="failed to get container status \"f9a9b8b547aa7bd0d8fe2f0a5cfaa38f62eabf881544ab626b8b174a63ec864d\": rpc error: code = NotFound desc = could not find container \"f9a9b8b547aa7bd0d8fe2f0a5cfaa38f62eabf881544ab626b8b174a63ec864d\": container with ID starting with f9a9b8b547aa7bd0d8fe2f0a5cfaa38f62eabf881544ab626b8b174a63ec864d not found: ID does not exist" Jul 06 20:46:02.350411 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:02.350380 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t"] Jul 06 20:46:02.357584 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:02.357560 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-37e178-b7c9dc549-tqp4t"] Jul 06 20:46:03.299016 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:03.298983 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70a3994a-2423-4402-a6ef-b92e75569998" path="/var/lib/kubelet/pods/70a3994a-2423-4402-a6ef-b92e75569998/volumes" Jul 06 20:46:06.109933 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:06.109891 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:11.503308 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:11.503231 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:11.603596 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:11.603567 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:30.207105 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:30.207070 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29722845-7h4gq" Jul 06 20:46:30.307903 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:30.307822 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hnms\" (UniqueName: \"kubernetes.io/projected/4e68323e-1267-4637-8956-c7dcd4385496-kube-api-access-6hnms\") pod \"4e68323e-1267-4637-8956-c7dcd4385496\" (UID: \"4e68323e-1267-4637-8956-c7dcd4385496\") " Jul 06 20:46:30.309970 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:30.309939 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e68323e-1267-4637-8956-c7dcd4385496-kube-api-access-6hnms" (OuterVolumeSpecName: "kube-api-access-6hnms") pod "4e68323e-1267-4637-8956-c7dcd4385496" (UID: "4e68323e-1267-4637-8956-c7dcd4385496"). InnerVolumeSpecName "kube-api-access-6hnms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:46:30.409168 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:30.409128 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hnms\" (UniqueName: \"kubernetes.io/projected/4e68323e-1267-4637-8956-c7dcd4385496-kube-api-access-6hnms\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:46:30.450828 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:30.450795 2578 generic.go:356] "Generic (PLEG): container finished" podID="4e68323e-1267-4637-8956-c7dcd4385496" containerID="665cee1a318bcf8c6bb415245b4f93de856b662a3add1008fab3cc424bbbfcb8" exitCode=137 Jul 06 20:46:30.450986 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:30.450835 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29722845-7h4gq" Jul 06 20:46:30.450986 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:30.450853 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29722845-7h4gq" event={"ID":"4e68323e-1267-4637-8956-c7dcd4385496","Type":"ContainerDied","Data":"665cee1a318bcf8c6bb415245b4f93de856b662a3add1008fab3cc424bbbfcb8"} Jul 06 20:46:30.450986 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:30.450908 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29722845-7h4gq" event={"ID":"4e68323e-1267-4637-8956-c7dcd4385496","Type":"ContainerDied","Data":"4271ead7021c70bde9bdc4654d8fbca41b0df677a617d09f64e5b5989f575d8f"} Jul 06 20:46:30.450986 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:30.450928 2578 scope.go:117] "RemoveContainer" containerID="665cee1a318bcf8c6bb415245b4f93de856b662a3add1008fab3cc424bbbfcb8" Jul 06 20:46:30.461498 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:30.461470 2578 scope.go:117] "RemoveContainer" containerID="665cee1a318bcf8c6bb415245b4f93de856b662a3add1008fab3cc424bbbfcb8" Jul 06 20:46:30.461893 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:46:30.461867 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"665cee1a318bcf8c6bb415245b4f93de856b662a3add1008fab3cc424bbbfcb8\": container with ID starting with 665cee1a318bcf8c6bb415245b4f93de856b662a3add1008fab3cc424bbbfcb8 not found: ID does not exist" containerID="665cee1a318bcf8c6bb415245b4f93de856b662a3add1008fab3cc424bbbfcb8" Jul 06 20:46:30.461971 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:30.461903 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"665cee1a318bcf8c6bb415245b4f93de856b662a3add1008fab3cc424bbbfcb8"} err="failed to get container status \"665cee1a318bcf8c6bb415245b4f93de856b662a3add1008fab3cc424bbbfcb8\": rpc error: code = NotFound desc = could not find container \"665cee1a318bcf8c6bb415245b4f93de856b662a3add1008fab3cc424bbbfcb8\": container with ID starting with 665cee1a318bcf8c6bb415245b4f93de856b662a3add1008fab3cc424bbbfcb8 not found: ID does not exist" Jul 06 20:46:30.477443 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:30.477417 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29722845-7h4gq"] Jul 06 20:46:30.489502 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:30.489477 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29722845-7h4gq"] Jul 06 20:46:31.298739 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:31.298707 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e68323e-1267-4637-8956-c7dcd4385496" path="/var/lib/kubelet/pods/4e68323e-1267-4637-8956-c7dcd4385496/volumes" Jul 06 20:46:43.606954 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:43.606917 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:43.901978 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:43.901945 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:44.383188 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:44.383100 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:44.795091 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:44.795033 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:44.899593 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:44.899562 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:44.995768 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:44.995728 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:45.168371 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:45.168336 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6"] Jul 06 20:46:45.168754 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:45.168722 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6" podUID="f23a0f93-b30c-4f6a-bcec-192e94541eac" containerName="maas-api" containerID="cri-o://72ec774c8377b9228975b3081b8aaed8da6b1221f1a8554e19cf94ebebd59e11" gracePeriod=30 Jul 06 20:46:45.193062 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:45.193040 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:45.415169 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:45.415146 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6" Jul 06 20:46:45.518988 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:45.518898 2578 generic.go:356] "Generic (PLEG): container finished" podID="f23a0f93-b30c-4f6a-bcec-192e94541eac" containerID="72ec774c8377b9228975b3081b8aaed8da6b1221f1a8554e19cf94ebebd59e11" exitCode=0 Jul 06 20:46:45.518988 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:45.518946 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6" Jul 06 20:46:45.518988 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:45.518982 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6" event={"ID":"f23a0f93-b30c-4f6a-bcec-192e94541eac","Type":"ContainerDied","Data":"72ec774c8377b9228975b3081b8aaed8da6b1221f1a8554e19cf94ebebd59e11"} Jul 06 20:46:45.519248 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:45.519031 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6" event={"ID":"f23a0f93-b30c-4f6a-bcec-192e94541eac","Type":"ContainerDied","Data":"4f2fced15ef2a6deaadd6c9f179ab448d87316e034a77ebc0b7bd4ce2a7da671"} Jul 06 20:46:45.519248 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:45.519054 2578 scope.go:117] "RemoveContainer" containerID="72ec774c8377b9228975b3081b8aaed8da6b1221f1a8554e19cf94ebebd59e11" Jul 06 20:46:45.527230 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:45.527216 2578 scope.go:117] "RemoveContainer" containerID="72ec774c8377b9228975b3081b8aaed8da6b1221f1a8554e19cf94ebebd59e11" Jul 06 20:46:45.527543 ip-10-0-131-218 kubenswrapper[2578]: E0706 20:46:45.527519 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72ec774c8377b9228975b3081b8aaed8da6b1221f1a8554e19cf94ebebd59e11\": container with ID starting with 72ec774c8377b9228975b3081b8aaed8da6b1221f1a8554e19cf94ebebd59e11 not found: ID does not exist" containerID="72ec774c8377b9228975b3081b8aaed8da6b1221f1a8554e19cf94ebebd59e11" Jul 06 20:46:45.527641 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:45.527547 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72ec774c8377b9228975b3081b8aaed8da6b1221f1a8554e19cf94ebebd59e11"} err="failed to get container status \"72ec774c8377b9228975b3081b8aaed8da6b1221f1a8554e19cf94ebebd59e11\": rpc error: code = NotFound desc = could not find container \"72ec774c8377b9228975b3081b8aaed8da6b1221f1a8554e19cf94ebebd59e11\": container with ID starting with 72ec774c8377b9228975b3081b8aaed8da6b1221f1a8554e19cf94ebebd59e11 not found: ID does not exist" Jul 06 20:46:45.549501 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:45.549444 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f23a0f93-b30c-4f6a-bcec-192e94541eac-openshift-service-ca\") pod \"f23a0f93-b30c-4f6a-bcec-192e94541eac\" (UID: \"f23a0f93-b30c-4f6a-bcec-192e94541eac\") " Jul 06 20:46:45.549620 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:45.549540 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhwl7\" (UniqueName: \"kubernetes.io/projected/f23a0f93-b30c-4f6a-bcec-192e94541eac-kube-api-access-mhwl7\") pod \"f23a0f93-b30c-4f6a-bcec-192e94541eac\" (UID: \"f23a0f93-b30c-4f6a-bcec-192e94541eac\") " Jul 06 20:46:45.549620 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:45.549563 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f23a0f93-b30c-4f6a-bcec-192e94541eac-maas-api-tls\") pod \"f23a0f93-b30c-4f6a-bcec-192e94541eac\" (UID: \"f23a0f93-b30c-4f6a-bcec-192e94541eac\") " Jul 06 20:46:45.549792 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:45.549767 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f23a0f93-b30c-4f6a-bcec-192e94541eac-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f23a0f93-b30c-4f6a-bcec-192e94541eac" (UID: "f23a0f93-b30c-4f6a-bcec-192e94541eac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:46:45.551676 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:45.551652 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f23a0f93-b30c-4f6a-bcec-192e94541eac-kube-api-access-mhwl7" (OuterVolumeSpecName: "kube-api-access-mhwl7") pod "f23a0f93-b30c-4f6a-bcec-192e94541eac" (UID: "f23a0f93-b30c-4f6a-bcec-192e94541eac"). InnerVolumeSpecName "kube-api-access-mhwl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:46:45.551775 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:45.551709 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f23a0f93-b30c-4f6a-bcec-192e94541eac-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f23a0f93-b30c-4f6a-bcec-192e94541eac" (UID: "f23a0f93-b30c-4f6a-bcec-192e94541eac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:46:45.650119 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:45.650086 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f23a0f93-b30c-4f6a-bcec-192e94541eac-openshift-service-ca\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:46:45.650119 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:45.650113 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhwl7\" (UniqueName: \"kubernetes.io/projected/f23a0f93-b30c-4f6a-bcec-192e94541eac-kube-api-access-mhwl7\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:46:45.650119 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:45.650125 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f23a0f93-b30c-4f6a-bcec-192e94541eac-maas-api-tls\") on node \"ip-10-0-131-218.ec2.internal\" DevicePath \"\"" Jul 06 20:46:45.842804 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:45.842772 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6"] Jul 06 20:46:45.847217 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:45.847195 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-67755c-67d6dc7b56-wv5h6"] Jul 06 20:46:47.298398 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:47.298365 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f23a0f93-b30c-4f6a-bcec-192e94541eac" path="/var/lib/kubelet/pods/f23a0f93-b30c-4f6a-bcec-192e94541eac/volumes" Jul 06 20:46:51.574057 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:51.574024 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:51.636234 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:51.636195 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:51.855624 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:51.855530 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:57.456486 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:57.456439 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65954644c5-g5sgh_a886baf9-c25d-4a4e-8eb5-2a2b48234466/manager/1.log" Jul 06 20:46:57.817057 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:57.816980 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68cbf59454-4sslp_630178c4-0b36-42bb-bbda-a53f32ec4fe0/copy-manifests/0.log" Jul 06 20:46:57.831191 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:46:57.831165 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68cbf59454-4sslp_630178c4-0b36-42bb-bbda-a53f32ec4fe0/manager/0.log" Jul 06 20:47:00.676372 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:00.676334 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-vblkn_a3be73c6-e989-4ccc-aacf-a9ef3d8f1378/discovery/0.log" Jul 06 20:47:01.161603 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:01.161572 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-ptpvt_f2d06033-e757-4936-bce6-3b23f1a033d7/payload-pre-processing/0.log" Jul 06 20:47:01.277008 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:01.276968 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-h7qvz_d03334da-8cb5-45dd-a1d2-9076cbaba240/payload-processing/0.log" Jul 06 20:47:01.751724 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:01.751676 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-v6z76_917e277b-543c-4010-8e59-a2fc6461ec76/main/0.log" Jul 06 20:47:02.109114 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:02.108992 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7mhbn_c4c33e1b-3286-4f90-92be-b2bcd94e3385/main/0.log" Jul 06 20:47:06.993839 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:06.993789 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:47:07.065810 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:07.065775 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" limit=1024 Jul 06 20:47:09.917033 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:09.916996 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jmp82_4ef938dd-1047-4283-a4f5-d0d0abb4eda0/global-pull-secret-syncer/0.log" Jul 06 20:47:10.062136 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:10.062102 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5wjbn_24dd4f5b-0cda-4bb0-b776-72712fb30f3c/konnectivity-agent/0.log" Jul 06 20:47:10.182852 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:10.182759 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-218.ec2.internal_8c77d522b11ae15bf067473201a7e234/haproxy/0.log" Jul 06 20:47:17.870904 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:17.870870 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-5czzx_57888923-ec2b-4511-8590-bdd4fd3ed9ce/monitoring-plugin/0.log" Jul 06 20:47:17.905501 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:17.905476 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h45hc_40f67c9c-cf00-4f97-a5b3-b4698737b55f/node-exporter/0.log" Jul 06 20:47:17.934608 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:17.934579 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h45hc_40f67c9c-cf00-4f97-a5b3-b4698737b55f/kube-rbac-proxy/0.log" Jul 06 20:47:17.960294 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:17.960268 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h45hc_40f67c9c-cf00-4f97-a5b3-b4698737b55f/init-textfile/0.log" Jul 06 20:47:18.257252 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:18.257218 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a5f1feda-1270-4213-b7c2-568b7f28c74c/prometheus/0.log" Jul 06 20:47:18.286865 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:18.286840 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a5f1feda-1270-4213-b7c2-568b7f28c74c/config-reloader/0.log" Jul 06 20:47:18.313336 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:18.313307 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a5f1feda-1270-4213-b7c2-568b7f28c74c/thanos-sidecar/0.log" Jul 06 20:47:18.340282 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:18.340249 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a5f1feda-1270-4213-b7c2-568b7f28c74c/kube-rbac-proxy-web/0.log" Jul 06 20:47:18.367414 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:18.367387 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a5f1feda-1270-4213-b7c2-568b7f28c74c/kube-rbac-proxy/0.log" Jul 06 20:47:18.391519 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:18.391489 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a5f1feda-1270-4213-b7c2-568b7f28c74c/kube-rbac-proxy-thanos/0.log" Jul 06 20:47:18.415496 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:18.415445 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a5f1feda-1270-4213-b7c2-568b7f28c74c/init-config-reloader/0.log" Jul 06 20:47:19.704858 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:19.704826 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rsttk/perf-node-gather-daemonset-n87ns"] Jul 06 20:47:19.705236 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:19.705132 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70a3994a-2423-4402-a6ef-b92e75569998" containerName="maas-api" Jul 06 20:47:19.705236 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:19.705144 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="70a3994a-2423-4402-a6ef-b92e75569998" containerName="maas-api" Jul 06 20:47:19.705236 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:19.705154 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e68323e-1267-4637-8956-c7dcd4385496" containerName="cleanup" Jul 06 20:47:19.705236 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:19.705159 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e68323e-1267-4637-8956-c7dcd4385496" containerName="cleanup" Jul 06 20:47:19.705236 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:19.705167 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f23a0f93-b30c-4f6a-bcec-192e94541eac" containerName="maas-api" Jul 06 20:47:19.705236 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:19.705172 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f23a0f93-b30c-4f6a-bcec-192e94541eac" containerName="maas-api" Jul 06 20:47:19.705236 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:19.705227 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f23a0f93-b30c-4f6a-bcec-192e94541eac" containerName="maas-api" Jul 06 20:47:19.705236 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:19.705234 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="70a3994a-2423-4402-a6ef-b92e75569998" containerName="maas-api" Jul 06 20:47:19.705236 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:19.705241 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e68323e-1267-4637-8956-c7dcd4385496" containerName="cleanup" Jul 06 20:47:19.708340 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:19.708322 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rsttk/perf-node-gather-daemonset-n87ns" Jul 06 20:47:19.710668 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:19.710643 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rsttk\"/\"kube-root-ca.crt\"" Jul 06 20:47:19.711638 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:19.711616 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rsttk\"/\"openshift-service-ca.crt\"" Jul 06 20:47:19.711736 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:19.711683 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rsttk\"/\"default-dockercfg-tvshg\"" Jul 06 20:47:19.716296 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:19.716272 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rsttk/perf-node-gather-daemonset-n87ns"] Jul 06 20:47:19.821738 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:19.821706 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9804eaef-90a1-4644-870a-1ce9e8a2b065-lib-modules\") pod \"perf-node-gather-daemonset-n87ns\" (UID: \"9804eaef-90a1-4644-870a-1ce9e8a2b065\") " pod="openshift-must-gather-rsttk/perf-node-gather-daemonset-n87ns" Jul 06 20:47:19.821917 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:19.821762 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bv69\" (UniqueName: \"kubernetes.io/projected/9804eaef-90a1-4644-870a-1ce9e8a2b065-kube-api-access-9bv69\") pod \"perf-node-gather-daemonset-n87ns\" (UID: \"9804eaef-90a1-4644-870a-1ce9e8a2b065\") " pod="openshift-must-gather-rsttk/perf-node-gather-daemonset-n87ns" Jul 06 20:47:19.821917 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:19.821792 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9804eaef-90a1-4644-870a-1ce9e8a2b065-podres\") pod \"perf-node-gather-daemonset-n87ns\" (UID: \"9804eaef-90a1-4644-870a-1ce9e8a2b065\") " pod="openshift-must-gather-rsttk/perf-node-gather-daemonset-n87ns" Jul 06 20:47:19.821917 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:19.821827 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9804eaef-90a1-4644-870a-1ce9e8a2b065-sys\") pod \"perf-node-gather-daemonset-n87ns\" (UID: \"9804eaef-90a1-4644-870a-1ce9e8a2b065\") " pod="openshift-must-gather-rsttk/perf-node-gather-daemonset-n87ns" Jul 06 20:47:19.821917 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:19.821840 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9804eaef-90a1-4644-870a-1ce9e8a2b065-proc\") pod \"perf-node-gather-daemonset-n87ns\" (UID: \"9804eaef-90a1-4644-870a-1ce9e8a2b065\") " pod="openshift-must-gather-rsttk/perf-node-gather-daemonset-n87ns" Jul 06 20:47:19.923018 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:19.922977 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9804eaef-90a1-4644-870a-1ce9e8a2b065-sys\") pod \"perf-node-gather-daemonset-n87ns\" (UID: \"9804eaef-90a1-4644-870a-1ce9e8a2b065\") " pod="openshift-must-gather-rsttk/perf-node-gather-daemonset-n87ns" Jul 06 20:47:19.923231 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:19.923028 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9804eaef-90a1-4644-870a-1ce9e8a2b065-proc\") pod \"perf-node-gather-daemonset-n87ns\" (UID: \"9804eaef-90a1-4644-870a-1ce9e8a2b065\") " pod="openshift-must-gather-rsttk/perf-node-gather-daemonset-n87ns" Jul 06 20:47:19.923231 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:19.923053 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9804eaef-90a1-4644-870a-1ce9e8a2b065-lib-modules\") pod \"perf-node-gather-daemonset-n87ns\" (UID: \"9804eaef-90a1-4644-870a-1ce9e8a2b065\") " pod="openshift-must-gather-rsttk/perf-node-gather-daemonset-n87ns" Jul 06 20:47:19.923231 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:19.923081 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9804eaef-90a1-4644-870a-1ce9e8a2b065-sys\") pod \"perf-node-gather-daemonset-n87ns\" (UID: \"9804eaef-90a1-4644-870a-1ce9e8a2b065\") " pod="openshift-must-gather-rsttk/perf-node-gather-daemonset-n87ns" Jul 06 20:47:19.923231 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:19.923106 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bv69\" (UniqueName: \"kubernetes.io/projected/9804eaef-90a1-4644-870a-1ce9e8a2b065-kube-api-access-9bv69\") pod \"perf-node-gather-daemonset-n87ns\" (UID: \"9804eaef-90a1-4644-870a-1ce9e8a2b065\") " pod="openshift-must-gather-rsttk/perf-node-gather-daemonset-n87ns" Jul 06 20:47:19.923231 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:19.923145 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9804eaef-90a1-4644-870a-1ce9e8a2b065-proc\") pod \"perf-node-gather-daemonset-n87ns\" (UID: \"9804eaef-90a1-4644-870a-1ce9e8a2b065\") " pod="openshift-must-gather-rsttk/perf-node-gather-daemonset-n87ns" Jul 06 20:47:19.923231 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:19.923165 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9804eaef-90a1-4644-870a-1ce9e8a2b065-podres\") pod \"perf-node-gather-daemonset-n87ns\" (UID: \"9804eaef-90a1-4644-870a-1ce9e8a2b065\") " pod="openshift-must-gather-rsttk/perf-node-gather-daemonset-n87ns" Jul 06 20:47:19.923231 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:19.923217 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9804eaef-90a1-4644-870a-1ce9e8a2b065-lib-modules\") pod \"perf-node-gather-daemonset-n87ns\" (UID: \"9804eaef-90a1-4644-870a-1ce9e8a2b065\") " pod="openshift-must-gather-rsttk/perf-node-gather-daemonset-n87ns" Jul 06 20:47:19.923523 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:19.923286 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9804eaef-90a1-4644-870a-1ce9e8a2b065-podres\") pod \"perf-node-gather-daemonset-n87ns\" (UID: \"9804eaef-90a1-4644-870a-1ce9e8a2b065\") " pod="openshift-must-gather-rsttk/perf-node-gather-daemonset-n87ns" Jul 06 20:47:19.932435 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:19.932412 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bv69\" (UniqueName: \"kubernetes.io/projected/9804eaef-90a1-4644-870a-1ce9e8a2b065-kube-api-access-9bv69\") pod \"perf-node-gather-daemonset-n87ns\" (UID: \"9804eaef-90a1-4644-870a-1ce9e8a2b065\") " pod="openshift-must-gather-rsttk/perf-node-gather-daemonset-n87ns" Jul 06 20:47:20.018631 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:20.018541 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rsttk/perf-node-gather-daemonset-n87ns" Jul 06 20:47:20.152961 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:20.152929 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rsttk/perf-node-gather-daemonset-n87ns"] Jul 06 20:47:20.156956 ip-10-0-131-218 kubenswrapper[2578]: W0706 20:47:20.156923 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9804eaef_90a1_4644_870a_1ce9e8a2b065.slice/crio-62a5e98370ea034a346265ec99751be7d17ed9b2e004a17be6ae3d772ebfdef7 WatchSource:0}: Error finding container 62a5e98370ea034a346265ec99751be7d17ed9b2e004a17be6ae3d772ebfdef7: Status 404 returned error can't find the container with id 62a5e98370ea034a346265ec99751be7d17ed9b2e004a17be6ae3d772ebfdef7 Jul 06 20:47:20.679734 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:20.679699 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rsttk/perf-node-gather-daemonset-n87ns" event={"ID":"9804eaef-90a1-4644-870a-1ce9e8a2b065","Type":"ContainerStarted","Data":"be988285bc067304b0d95163f1d45dab1b822bd57cbfeab9f99da83ef965dcd4"} Jul 06 20:47:20.680111 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:20.679749 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rsttk/perf-node-gather-daemonset-n87ns" event={"ID":"9804eaef-90a1-4644-870a-1ce9e8a2b065","Type":"ContainerStarted","Data":"62a5e98370ea034a346265ec99751be7d17ed9b2e004a17be6ae3d772ebfdef7"} Jul 06 20:47:20.680111 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:20.679828 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rsttk/perf-node-gather-daemonset-n87ns" Jul 06 20:47:20.701393 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:20.701339 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rsttk/perf-node-gather-daemonset-n87ns" podStartSLOduration=1.70132445 podStartE2EDuration="1.70132445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:47:20.69999799 +0000 UTC m=+2491.911146202" watchObservedRunningTime="2026-07-06 20:47:20.70132445 +0000 UTC m=+2491.912472705" Jul 06 20:47:22.850788 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:22.850765 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pwlxq_71661bcb-74d1-48e3-afa1-4acec1af79ae/dns/0.log" Jul 06 20:47:22.879928 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:22.879902 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pwlxq_71661bcb-74d1-48e3-afa1-4acec1af79ae/kube-rbac-proxy/0.log" Jul 06 20:47:22.982546 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:22.982519 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j4d6m_f0b6b2dc-73c6-453c-9941-8738e48b4976/dns-node-resolver/0.log" Jul 06 20:47:23.624966 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:23.624942 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r6xsf_4e950de4-fc27-492d-ab51-69255ef36a84/node-ca/0.log" Jul 06 20:47:24.738982 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:24.738941 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-vblkn_a3be73c6-e989-4ccc-aacf-a9ef3d8f1378/discovery/0.log" Jul 06 20:47:24.891009 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:24.890980 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-ptpvt_f2d06033-e757-4936-bce6-3b23f1a033d7/payload-pre-processing/0.log" Jul 06 20:47:24.945499 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:24.945472 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-h7qvz_d03334da-8cb5-45dd-a1d2-9076cbaba240/payload-processing/0.log" Jul 06 20:47:25.755177 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:25.755146 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9vmsh_ba49337f-cb8d-4c73-9a76-7945951eac79/serve-healthcheck-canary/0.log" Jul 06 20:47:26.411202 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:26.411170 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s2vnx_c2e525b8-01f4-4894-9604-e72935da6856/kube-rbac-proxy/0.log" Jul 06 20:47:26.436342 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:26.436317 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s2vnx_c2e525b8-01f4-4894-9604-e72935da6856/exporter/0.log" Jul 06 20:47:26.461659 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:26.461629 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s2vnx_c2e525b8-01f4-4894-9604-e72935da6856/extractor/0.log" Jul 06 20:47:26.693710 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:26.693631 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rsttk/perf-node-gather-daemonset-n87ns" Jul 06 20:47:29.092342 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:29.092240 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65954644c5-g5sgh_a886baf9-c25d-4a4e-8eb5-2a2b48234466/manager/1.log" Jul 06 20:47:29.104100 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:29.104076 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65954644c5-g5sgh_a886baf9-c25d-4a4e-8eb5-2a2b48234466/manager/0.log" Jul 06 20:47:29.262926 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:29.262892 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68cbf59454-4sslp_630178c4-0b36-42bb-bbda-a53f32ec4fe0/manager/0.log" Jul 06 20:47:29.288597 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:29.288568 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68cbf59454-4sslp_630178c4-0b36-42bb-bbda-a53f32ec4fe0/copy-manifests/0.log" Jul 06 20:47:30.097033 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:30.096941 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-vblkn_a3be73c6-e989-4ccc-aacf-a9ef3d8f1378/discovery/0.log" Jul 06 20:47:30.211999 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:30.211973 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-ptpvt_f2d06033-e757-4936-bce6-3b23f1a033d7/payload-pre-processing/0.log" Jul 06 20:47:30.246118 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:30.246086 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-h7qvz_d03334da-8cb5-45dd-a1d2-9076cbaba240/payload-processing/0.log" Jul 06 20:47:30.744806 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:30.744725 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-79c95fdbc9-rhh4g_0def7154-736a-4490-acf2-93ccb9ae1a35/manager/1.log" Jul 06 20:47:30.745098 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:30.745083 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-79c95fdbc9-rhh4g_0def7154-736a-4490-acf2-93ccb9ae1a35/manager/0.log" Jul 06 20:47:31.724075 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:31.724041 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-vblkn_a3be73c6-e989-4ccc-aacf-a9ef3d8f1378/discovery/0.log" Jul 06 20:47:31.828615 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:31.828587 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-ptpvt_f2d06033-e757-4936-bce6-3b23f1a033d7/payload-pre-processing/0.log" Jul 06 20:47:31.866629 ip-10-0-131-218 kubenswrapper[2578]: I0706 20:47:31.866592 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-h7qvz_d03334da-8cb5-45dd-a1d2-9076cbaba240/payload-processing/0.log"