Jul 14 22:07:20.320290 ip-10-0-133-118 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 14 22:07:20.320300 ip-10-0-133-118 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 14 22:07:20.320307 ip-10-0-133-118 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 14 22:07:20.320527 ip-10-0-133-118 systemd[1]: Failed to start Kubernetes Kubelet. Jul 14 22:07:30.402688 ip-10-0-133-118 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 14 22:07:30.402705 ip-10-0-133-118 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot d63d1e423bfe407ea200d1e513755e87 -- Jul 14 22:09:27.611319 ip-10-0-133-118 systemd[1]: Starting Kubernetes Kubelet... Jul 14 22:09:27.981595 ip-10-0-133-118 kubenswrapper[2550]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 22:09:27.981595 ip-10-0-133-118 kubenswrapper[2550]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 14 22:09:27.981595 ip-10-0-133-118 kubenswrapper[2550]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 22:09:27.981595 ip-10-0-133-118 kubenswrapper[2550]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 14 22:09:27.981595 ip-10-0-133-118 kubenswrapper[2550]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 22:09:27.983445 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.983363 2550 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 14 22:09:27.987388 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987373 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 22:09:27.987388 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987388 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 22:09:27.987447 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987392 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 22:09:27.987447 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987398 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 22:09:27.987447 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987402 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 22:09:27.987447 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987406 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 22:09:27.987447 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987409 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 22:09:27.987447 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987413 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 22:09:27.987447 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987416 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 22:09:27.987447 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987419 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 22:09:27.987447 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987421 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 22:09:27.987447 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987424 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 22:09:27.987447 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987427 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 22:09:27.987447 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987429 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 22:09:27.987447 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987432 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 22:09:27.987447 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987435 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 22:09:27.987447 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987437 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 22:09:27.987447 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987440 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 22:09:27.987447 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987444 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 22:09:27.987447 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987446 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 22:09:27.987447 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987449 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 22:09:27.987899 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987458 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 22:09:27.987899 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987461 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 22:09:27.987899 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987464 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 22:09:27.987899 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987467 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 22:09:27.987899 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987469 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 22:09:27.987899 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987471 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 22:09:27.987899 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987474 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 22:09:27.987899 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987477 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 22:09:27.987899 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987479 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 22:09:27.987899 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987482 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 22:09:27.987899 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987484 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 22:09:27.987899 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987487 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 22:09:27.987899 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987489 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 22:09:27.987899 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987492 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 22:09:27.987899 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987494 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 22:09:27.987899 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987497 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 22:09:27.987899 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987499 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 22:09:27.987899 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987501 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 22:09:27.987899 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987504 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 22:09:27.987899 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987507 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 22:09:27.988389 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987509 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 22:09:27.988389 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987511 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 22:09:27.988389 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987514 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 22:09:27.988389 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987516 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 22:09:27.988389 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987519 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 22:09:27.988389 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987521 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 22:09:27.988389 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987523 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 22:09:27.988389 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987526 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 22:09:27.988389 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987528 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 22:09:27.988389 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987530 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 22:09:27.988389 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987533 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 22:09:27.988389 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987535 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 22:09:27.988389 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987538 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 22:09:27.988389 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987547 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 22:09:27.988389 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987550 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 22:09:27.988389 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987552 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 22:09:27.988389 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987555 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 22:09:27.988389 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987559 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 22:09:27.988389 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987562 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 22:09:27.988867 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987565 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 22:09:27.988867 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987567 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 22:09:27.988867 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987569 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 22:09:27.988867 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987572 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 22:09:27.988867 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987575 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 22:09:27.988867 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987578 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 22:09:27.988867 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987581 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 22:09:27.988867 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987583 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 22:09:27.988867 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987586 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 22:09:27.988867 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987588 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 22:09:27.988867 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987590 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 22:09:27.988867 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987593 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 22:09:27.988867 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987595 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 22:09:27.988867 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987598 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 22:09:27.988867 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987600 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 22:09:27.988867 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987603 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 22:09:27.988867 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987605 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 22:09:27.988867 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987608 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 22:09:27.988867 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987610 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 22:09:27.988867 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987612 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 22:09:27.989341 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987615 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 22:09:27.989341 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987617 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 22:09:27.989341 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987620 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 22:09:27.989341 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987622 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 22:09:27.989341 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987625 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 22:09:27.989341 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987627 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 22:09:27.989341 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987631 2550 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 22:09:27.989341 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987640 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 22:09:27.989341 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987642 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 22:09:27.989341 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987644 2550 feature_gate.go:328] unrecognized feature gate: Example Jul 14 22:09:27.989341 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987647 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 22:09:27.989341 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987650 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 22:09:27.989341 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987653 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 22:09:27.989341 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987656 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 22:09:27.989341 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987659 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 22:09:27.989341 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987661 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 22:09:27.989341 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987664 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 22:09:27.989341 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987666 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 22:09:27.989341 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.987668 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 22:09:27.989341 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988367 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 22:09:27.989871 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988374 2550 feature_gate.go:328] unrecognized feature gate: Example Jul 14 22:09:27.989871 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988377 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 22:09:27.989871 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988380 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 22:09:27.989871 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988383 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 22:09:27.989871 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988386 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 22:09:27.989871 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988388 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 22:09:27.989871 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988391 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 22:09:27.989871 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988394 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 22:09:27.989871 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988397 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 22:09:27.989871 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988399 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 22:09:27.989871 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988402 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 22:09:27.989871 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988404 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 22:09:27.989871 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988407 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 22:09:27.989871 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988410 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 22:09:27.989871 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988412 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 22:09:27.989871 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988415 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 22:09:27.989871 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988417 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 22:09:27.989871 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988421 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 22:09:27.989871 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988424 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 22:09:27.989871 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988427 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 22:09:27.990378 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988429 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 22:09:27.990378 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988432 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 22:09:27.990378 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988435 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 22:09:27.990378 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988438 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 22:09:27.990378 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988440 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 22:09:27.990378 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988442 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 22:09:27.990378 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988447 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 22:09:27.990378 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988450 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 22:09:27.990378 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988454 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 22:09:27.990378 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988457 2550 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 22:09:27.990378 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988459 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 22:09:27.990378 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988462 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 22:09:27.990378 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988465 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 22:09:27.990378 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988468 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 22:09:27.990378 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988471 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 22:09:27.990378 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988474 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 22:09:27.990378 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988476 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 22:09:27.990378 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988479 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 22:09:27.990378 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988481 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 22:09:27.990378 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988484 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 22:09:27.990903 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988487 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 22:09:27.990903 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988490 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 22:09:27.990903 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988492 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 22:09:27.990903 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988495 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 22:09:27.990903 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988498 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 22:09:27.990903 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988500 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 22:09:27.990903 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988503 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 22:09:27.990903 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988505 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 22:09:27.990903 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988508 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 22:09:27.990903 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988510 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 22:09:27.990903 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988513 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 22:09:27.990903 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988515 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 22:09:27.990903 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988518 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 22:09:27.990903 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988524 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 22:09:27.990903 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988526 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 22:09:27.990903 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988529 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 22:09:27.990903 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988531 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 22:09:27.990903 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988534 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 22:09:27.990903 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988537 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 22:09:27.990903 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988540 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 22:09:27.991407 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988542 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 22:09:27.991407 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988545 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 22:09:27.991407 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988547 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 22:09:27.991407 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988550 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 22:09:27.991407 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988552 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 22:09:27.991407 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988555 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 22:09:27.991407 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988557 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 22:09:27.991407 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988560 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 22:09:27.991407 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988562 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 22:09:27.991407 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988565 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 22:09:27.991407 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988569 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 22:09:27.991407 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988572 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 22:09:27.991407 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988575 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 22:09:27.991407 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988579 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 22:09:27.991407 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988582 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 22:09:27.991407 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988584 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 22:09:27.991407 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988587 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 22:09:27.991407 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988589 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 22:09:27.991407 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988592 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 22:09:27.991407 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988594 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 22:09:27.991875 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988597 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 22:09:27.991875 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988599 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 22:09:27.991875 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988602 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 22:09:27.991875 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988604 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 22:09:27.991875 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988607 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 22:09:27.991875 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988609 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 22:09:27.991875 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988618 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 22:09:27.991875 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988620 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 22:09:27.991875 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988623 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 22:09:27.991875 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988629 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 22:09:27.991875 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988631 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 22:09:27.991875 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988634 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 22:09:27.991875 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988637 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 22:09:27.991875 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988639 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 22:09:27.991875 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988642 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 22:09:27.991875 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988644 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 22:09:27.991875 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988647 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 22:09:27.991875 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988649 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 22:09:27.991875 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.988999 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 22:09:27.992338 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989003 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 22:09:27.992338 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989006 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 22:09:27.992338 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989009 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 22:09:27.992338 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989012 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 22:09:27.992338 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989014 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 22:09:27.992338 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989017 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 22:09:27.992338 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989021 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 22:09:27.992338 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989025 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 22:09:27.992338 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989029 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 22:09:27.992338 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989033 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 22:09:27.992338 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989036 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 22:09:27.992338 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989039 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 22:09:27.992338 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989042 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 22:09:27.992338 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989045 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 22:09:27.992338 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989048 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 22:09:27.992338 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989051 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 22:09:27.992338 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989054 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 22:09:27.992338 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989056 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 22:09:27.992338 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989059 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 22:09:27.992794 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989062 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 22:09:27.992794 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989064 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 22:09:27.992794 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989067 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 22:09:27.992794 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989071 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 22:09:27.992794 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989073 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 22:09:27.992794 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989076 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 22:09:27.992794 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989078 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 22:09:27.992794 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989096 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 22:09:27.992794 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989099 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 22:09:27.992794 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989102 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 22:09:27.992794 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989105 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 22:09:27.992794 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989107 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 22:09:27.992794 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989110 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 22:09:27.992794 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989112 2550 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 22:09:27.992794 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989115 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 22:09:27.992794 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989118 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 22:09:27.992794 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989121 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 22:09:27.992794 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989124 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 22:09:27.992794 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989126 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 22:09:27.992794 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989129 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 22:09:27.993286 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989131 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 22:09:27.993286 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989134 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 22:09:27.993286 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989139 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 22:09:27.993286 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989142 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 22:09:27.993286 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989145 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 22:09:27.993286 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989147 2550 feature_gate.go:328] unrecognized feature gate: Example Jul 14 22:09:27.993286 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989150 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 22:09:27.993286 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989152 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 22:09:27.993286 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989155 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 22:09:27.993286 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989157 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 22:09:27.993286 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989160 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 22:09:27.993286 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989162 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 22:09:27.993286 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989164 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 22:09:27.993286 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989167 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 22:09:27.993286 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989170 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 22:09:27.993286 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989173 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 22:09:27.993286 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989176 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 22:09:27.993286 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989178 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 22:09:27.993286 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989181 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 22:09:27.993286 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989183 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 22:09:27.993748 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989186 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 22:09:27.993748 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989188 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 22:09:27.993748 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989191 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 22:09:27.993748 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989193 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 22:09:27.993748 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989195 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 22:09:27.993748 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989198 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 22:09:27.993748 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989200 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 22:09:27.993748 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989203 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 22:09:27.993748 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989205 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 22:09:27.993748 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989208 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 22:09:27.993748 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989211 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 22:09:27.993748 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989214 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 22:09:27.993748 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989218 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 22:09:27.993748 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989221 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 22:09:27.993748 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989225 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 22:09:27.993748 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989228 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 22:09:27.993748 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989231 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 22:09:27.993748 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989233 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 22:09:27.993748 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989236 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 22:09:27.994234 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989238 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 22:09:27.994234 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989241 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 22:09:27.994234 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989244 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 22:09:27.994234 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989246 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 22:09:27.994234 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989268 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 22:09:27.994234 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989271 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 22:09:27.994234 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989274 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 22:09:27.994234 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989277 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 22:09:27.994234 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989281 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 22:09:27.994234 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989283 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 22:09:27.994234 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989286 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 22:09:27.994234 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989288 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 22:09:27.994234 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989291 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 22:09:27.994234 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989293 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 22:09:27.994234 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989296 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 22:09:27.994234 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989298 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 22:09:27.994234 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989301 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 22:09:27.994234 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989303 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 22:09:27.994234 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989306 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 22:09:27.994234 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.989308 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 22:09:27.994716 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989377 2550 flags.go:64] FLAG: --address="0.0.0.0" Jul 14 22:09:27.994716 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989386 2550 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 14 22:09:27.994716 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989395 2550 flags.go:64] FLAG: --anonymous-auth="true" Jul 14 22:09:27.994716 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989402 2550 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 14 22:09:27.994716 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989409 2550 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 14 22:09:27.994716 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989413 2550 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 14 22:09:27.994716 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989417 2550 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 14 22:09:27.994716 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989422 2550 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 14 22:09:27.994716 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989425 2550 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 14 22:09:27.994716 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989428 2550 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 14 22:09:27.994716 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989431 2550 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 14 22:09:27.994716 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989434 2550 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 14 22:09:27.994716 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989437 2550 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 14 22:09:27.994716 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989440 2550 flags.go:64] FLAG: --cgroup-root="" Jul 14 22:09:27.994716 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989443 2550 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 14 22:09:27.994716 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989446 2550 flags.go:64] FLAG: --client-ca-file="" Jul 14 22:09:27.994716 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989448 2550 flags.go:64] FLAG: --cloud-provider="external" Jul 14 22:09:27.994716 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989451 2550 flags.go:64] FLAG: --cluster-dns="[]" Jul 14 22:09:27.994716 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989455 2550 flags.go:64] FLAG: --cluster-domain="" Jul 14 22:09:27.994716 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989458 2550 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 14 22:09:27.994716 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989462 2550 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 14 22:09:27.994716 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989466 2550 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 14 22:09:27.994716 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989469 2550 flags.go:64] FLAG: --container-log-max-files="5" Jul 14 22:09:27.994716 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989472 2550 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 14 22:09:27.995423 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989475 2550 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 14 22:09:27.995423 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989478 2550 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 14 22:09:27.995423 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989481 2550 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 14 22:09:27.995423 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989484 2550 flags.go:64] FLAG: --contention-profiling="false" Jul 14 22:09:27.995423 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989487 2550 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 14 22:09:27.995423 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989489 2550 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 14 22:09:27.995423 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989492 2550 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 14 22:09:27.995423 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989495 2550 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 14 22:09:27.995423 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989499 2550 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 14 22:09:27.995423 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989508 2550 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 14 22:09:27.995423 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989511 2550 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 14 22:09:27.995423 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989514 2550 flags.go:64] FLAG: --enable-load-reader="false" Jul 14 22:09:27.995423 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989516 2550 flags.go:64] FLAG: --enable-server="true" Jul 14 22:09:27.995423 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989519 2550 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 14 22:09:27.995423 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989524 2550 flags.go:64] FLAG: --event-burst="100" Jul 14 22:09:27.995423 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989527 2550 flags.go:64] FLAG: --event-qps="50" Jul 14 22:09:27.995423 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989530 2550 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 14 22:09:27.995423 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989533 2550 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 14 22:09:27.995423 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989536 2550 flags.go:64] FLAG: --eviction-hard="" Jul 14 22:09:27.995423 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989539 2550 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 14 22:09:27.995423 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989542 2550 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 14 22:09:27.995423 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989545 2550 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 14 22:09:27.995423 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989548 2550 flags.go:64] FLAG: --eviction-soft="" Jul 14 22:09:27.995423 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989551 2550 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 14 22:09:27.995423 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989554 2550 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 14 22:09:27.996494 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989557 2550 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 14 22:09:27.996494 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989560 2550 flags.go:64] FLAG: --experimental-mounter-path="" Jul 14 22:09:27.996494 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989562 2550 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 14 22:09:27.996494 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989567 2550 flags.go:64] FLAG: --fail-swap-on="true" Jul 14 22:09:27.996494 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989570 2550 flags.go:64] FLAG: --feature-gates="" Jul 14 22:09:27.996494 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989574 2550 flags.go:64] FLAG: --file-check-frequency="20s" Jul 14 22:09:27.996494 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989577 2550 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 14 22:09:27.996494 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989579 2550 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 14 22:09:27.996494 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989582 2550 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 14 22:09:27.996494 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989585 2550 flags.go:64] FLAG: --healthz-port="10248" Jul 14 22:09:27.996494 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989588 2550 flags.go:64] FLAG: --help="false" Jul 14 22:09:27.996494 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989591 2550 flags.go:64] FLAG: --hostname-override="ip-10-0-133-118.ec2.internal" Jul 14 22:09:27.996494 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989594 2550 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 14 22:09:27.996494 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989597 2550 flags.go:64] FLAG: --http-check-frequency="20s" Jul 14 22:09:27.996494 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989600 2550 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 14 22:09:27.996494 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989603 2550 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 14 22:09:27.996494 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989606 2550 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 14 22:09:27.996494 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989609 2550 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 14 22:09:27.996494 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989612 2550 flags.go:64] FLAG: --image-service-endpoint="" Jul 14 22:09:27.996494 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989615 2550 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 14 22:09:27.996494 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989618 2550 flags.go:64] FLAG: --kube-api-burst="100" Jul 14 22:09:27.996494 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989621 2550 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 14 22:09:27.996494 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989624 2550 flags.go:64] FLAG: --kube-api-qps="50" Jul 14 22:09:27.996494 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989627 2550 flags.go:64] FLAG: --kube-reserved="" Jul 14 22:09:27.997542 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989630 2550 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 14 22:09:27.997542 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989632 2550 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 14 22:09:27.997542 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989636 2550 flags.go:64] FLAG: --kubelet-cgroups="" Jul 14 22:09:27.997542 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989639 2550 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 14 22:09:27.997542 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989641 2550 flags.go:64] FLAG: --lock-file="" Jul 14 22:09:27.997542 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989644 2550 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 14 22:09:27.997542 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989647 2550 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 14 22:09:27.997542 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989650 2550 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 14 22:09:27.997542 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989655 2550 flags.go:64] FLAG: --log-json-split-stream="false" Jul 14 22:09:27.997542 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989657 2550 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 14 22:09:27.997542 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989660 2550 flags.go:64] FLAG: --log-text-split-stream="false" Jul 14 22:09:27.997542 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989664 2550 flags.go:64] FLAG: --logging-format="text" Jul 14 22:09:27.997542 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989667 2550 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 14 22:09:27.997542 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989670 2550 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 14 22:09:27.997542 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989673 2550 flags.go:64] FLAG: --manifest-url="" Jul 14 22:09:27.997542 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989676 2550 flags.go:64] FLAG: --manifest-url-header="" Jul 14 22:09:27.997542 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989680 2550 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 14 22:09:27.997542 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989683 2550 flags.go:64] FLAG: --max-open-files="1000000" Jul 14 22:09:27.997542 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989687 2550 flags.go:64] FLAG: --max-pods="110" Jul 14 22:09:27.997542 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989690 2550 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 14 22:09:27.997542 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989693 2550 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 14 22:09:27.997542 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989696 2550 flags.go:64] FLAG: --memory-manager-policy="None" Jul 14 22:09:27.997542 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989698 2550 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 14 22:09:27.997542 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989701 2550 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 14 22:09:27.997542 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989704 2550 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 14 22:09:27.998283 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989707 2550 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 14 22:09:27.998283 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989726 2550 flags.go:64] FLAG: --node-status-max-images="50" Jul 14 22:09:27.998283 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989730 2550 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 14 22:09:27.998283 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989733 2550 flags.go:64] FLAG: --oom-score-adj="-999" Jul 14 22:09:27.998283 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989736 2550 flags.go:64] FLAG: --pod-cidr="" Jul 14 22:09:27.998283 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989739 2550 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 14 22:09:27.998283 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989744 2550 flags.go:64] FLAG: --pod-manifest-path="" Jul 14 22:09:27.998283 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989747 2550 flags.go:64] FLAG: --pod-max-pids="-1" Jul 14 22:09:27.998283 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989750 2550 flags.go:64] FLAG: --pods-per-core="0" Jul 14 22:09:27.998283 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989753 2550 flags.go:64] FLAG: --port="10250" Jul 14 22:09:27.998283 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989756 2550 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 14 22:09:27.998283 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989759 2550 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-083ad0243e7d38709" Jul 14 22:09:27.998283 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989762 2550 flags.go:64] FLAG: --qos-reserved="" Jul 14 22:09:27.998283 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989764 2550 flags.go:64] FLAG: --read-only-port="10255" Jul 14 22:09:27.998283 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989767 2550 flags.go:64] FLAG: --register-node="true" Jul 14 22:09:27.998283 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989770 2550 flags.go:64] FLAG: --register-with-taints="" Jul 14 22:09:27.998283 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989773 2550 flags.go:64] FLAG: --registry-burst="10" Jul 14 22:09:27.998283 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989776 2550 flags.go:64] FLAG: --registry-qps="5" Jul 14 22:09:27.998283 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989781 2550 flags.go:64] FLAG: --reserved-cpus="" Jul 14 22:09:27.998283 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989784 2550 flags.go:64] FLAG: --reserved-memory="" Jul 14 22:09:27.998283 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989787 2550 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 14 22:09:27.998283 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989791 2550 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 14 22:09:27.998283 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989794 2550 flags.go:64] FLAG: --rotate-certificates="false" Jul 14 22:09:27.998283 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989797 2550 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 14 22:09:27.998283 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989799 2550 flags.go:64] FLAG: --runonce="false" Jul 14 22:09:27.998896 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989802 2550 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 14 22:09:27.998896 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989805 2550 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 14 22:09:27.998896 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989808 2550 flags.go:64] FLAG: --seccomp-default="false" Jul 14 22:09:27.998896 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989810 2550 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 14 22:09:27.998896 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989813 2550 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 14 22:09:27.998896 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989816 2550 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 14 22:09:27.998896 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989819 2550 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 14 22:09:27.998896 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989822 2550 flags.go:64] FLAG: --storage-driver-password="root" Jul 14 22:09:27.998896 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989824 2550 flags.go:64] FLAG: --storage-driver-secure="false" Jul 14 22:09:27.998896 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989835 2550 flags.go:64] FLAG: --storage-driver-table="stats" Jul 14 22:09:27.998896 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989838 2550 flags.go:64] FLAG: --storage-driver-user="root" Jul 14 22:09:27.998896 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989841 2550 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 14 22:09:27.998896 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989845 2550 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 14 22:09:27.998896 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989848 2550 flags.go:64] FLAG: --system-cgroups="" Jul 14 22:09:27.998896 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989851 2550 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 14 22:09:27.998896 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989856 2550 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 14 22:09:27.998896 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989859 2550 flags.go:64] FLAG: --tls-cert-file="" Jul 14 22:09:27.998896 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989862 2550 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 14 22:09:27.998896 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989866 2550 flags.go:64] FLAG: --tls-min-version="" Jul 14 22:09:27.998896 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989868 2550 flags.go:64] FLAG: --tls-private-key-file="" Jul 14 22:09:27.998896 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989871 2550 flags.go:64] FLAG: --topology-manager-policy="none" Jul 14 22:09:27.998896 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989874 2550 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 14 22:09:27.998896 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989877 2550 flags.go:64] FLAG: --topology-manager-scope="container" Jul 14 22:09:27.998896 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989880 2550 flags.go:64] FLAG: --v="2" Jul 14 22:09:27.998896 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989884 2550 flags.go:64] FLAG: --version="false" Jul 14 22:09:27.999516 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989891 2550 flags.go:64] FLAG: --vmodule="" Jul 14 22:09:27.999516 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989895 2550 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 14 22:09:27.999516 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.989898 2550 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 14 22:09:27.999516 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990015 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 22:09:27.999516 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990019 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 22:09:27.999516 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990022 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 22:09:27.999516 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990025 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 22:09:27.999516 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990028 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 22:09:27.999516 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990031 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 22:09:27.999516 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990034 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 22:09:27.999516 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990038 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 22:09:27.999516 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990041 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 22:09:27.999516 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990044 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 22:09:27.999516 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990047 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 22:09:27.999516 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990050 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 22:09:27.999516 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990052 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 22:09:27.999516 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990055 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 22:09:27.999516 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990058 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 22:09:27.999516 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990061 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 22:09:27.999516 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990065 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 22:09:28.000000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990068 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 22:09:28.000000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990071 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 22:09:28.000000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990074 2550 feature_gate.go:328] unrecognized feature gate: Example Jul 14 22:09:28.000000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990077 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 22:09:28.000000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990093 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 22:09:28.000000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990096 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 22:09:28.000000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990098 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 22:09:28.000000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990102 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 22:09:28.000000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990105 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 22:09:28.000000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990107 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 22:09:28.000000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990110 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 22:09:28.000000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990113 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 22:09:28.000000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990116 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 22:09:28.000000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990118 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 22:09:28.000000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990121 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 22:09:28.000000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990123 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 22:09:28.000000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990126 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 22:09:28.000000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990129 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 22:09:28.000000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990132 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 22:09:28.000000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990134 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 22:09:28.000508 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990137 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 22:09:28.000508 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990139 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 22:09:28.000508 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990141 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 22:09:28.000508 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990144 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 22:09:28.000508 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990146 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 22:09:28.000508 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990149 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 22:09:28.000508 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990153 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 22:09:28.000508 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990156 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 22:09:28.000508 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990160 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 22:09:28.000508 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990163 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 22:09:28.000508 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990165 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 22:09:28.000508 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990170 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 22:09:28.000508 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990173 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 22:09:28.000508 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990176 2550 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 22:09:28.000508 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990179 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 22:09:28.000508 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990182 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 22:09:28.000508 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990184 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 22:09:28.000508 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990187 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 22:09:28.000508 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990189 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 22:09:28.001003 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990192 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 22:09:28.001003 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990195 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 22:09:28.001003 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990197 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 22:09:28.001003 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990199 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 22:09:28.001003 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990203 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 22:09:28.001003 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990207 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 22:09:28.001003 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990210 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 22:09:28.001003 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990213 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 22:09:28.001003 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990215 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 22:09:28.001003 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990218 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 22:09:28.001003 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990221 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 22:09:28.001003 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990224 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 22:09:28.001003 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990227 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 22:09:28.001003 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990229 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 22:09:28.001003 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990232 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 22:09:28.001003 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990234 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 22:09:28.001003 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990237 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 22:09:28.001003 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990240 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 22:09:28.001003 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990242 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 22:09:28.001003 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990244 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 22:09:28.001520 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990247 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 22:09:28.001520 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990250 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 22:09:28.001520 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990252 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 22:09:28.001520 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990254 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 22:09:28.001520 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990258 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 22:09:28.001520 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990261 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 22:09:28.001520 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990264 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 22:09:28.001520 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990266 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 22:09:28.001520 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990269 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 22:09:28.001520 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990271 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 22:09:28.001520 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990274 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 22:09:28.001520 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990277 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 22:09:28.001520 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990280 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 22:09:28.001520 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990282 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 22:09:28.001520 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990285 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 22:09:28.001520 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990288 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 22:09:28.001520 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990291 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 22:09:28.001520 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990293 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 22:09:28.001520 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990296 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 22:09:28.002064 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990298 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 22:09:28.002064 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990301 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 22:09:28.002064 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990303 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 22:09:28.002064 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.990306 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 22:09:28.002064 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.990314 2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 22:09:28.002064 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.998045 2550 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 14 22:09:28.002064 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.998059 2550 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 14 22:09:28.002064 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.998824 2550 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 14 22:09:28.002064 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.998832 2550 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 14 22:09:28.002064 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998886 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 22:09:28.002064 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998891 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 22:09:28.002064 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998894 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 22:09:28.002064 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998897 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 22:09:28.002064 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998900 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 22:09:28.002064 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998903 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 22:09:28.002488 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998906 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 22:09:28.002488 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998909 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 22:09:28.002488 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998912 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 22:09:28.002488 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998914 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 22:09:28.002488 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998918 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 22:09:28.002488 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998920 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 22:09:28.002488 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998923 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 22:09:28.002488 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998925 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 22:09:28.002488 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998928 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 22:09:28.002488 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998931 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 22:09:28.002488 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998934 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 22:09:28.002488 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998938 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 22:09:28.002488 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998942 2550 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 22:09:28.002488 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998945 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 22:09:28.002488 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998948 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 22:09:28.002488 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998951 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 22:09:28.002488 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998953 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 22:09:28.002488 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998957 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 22:09:28.002488 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998959 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 22:09:28.003010 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998962 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 22:09:28.003010 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998964 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 22:09:28.003010 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998967 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 22:09:28.003010 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998970 2550 feature_gate.go:328] unrecognized feature gate: Example Jul 14 22:09:28.003010 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998973 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 22:09:28.003010 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998976 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 22:09:28.003010 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998979 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 22:09:28.003010 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998982 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 22:09:28.003010 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998984 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 22:09:28.003010 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998987 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 22:09:28.003010 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998990 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 22:09:28.003010 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998993 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 22:09:28.003010 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998995 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 22:09:28.003010 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.998998 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 22:09:28.003010 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999001 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 22:09:28.003010 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999003 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 22:09:28.003010 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999006 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 22:09:28.003010 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999008 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 22:09:28.003010 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999011 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 22:09:28.003010 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999013 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 22:09:28.003613 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999016 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 22:09:28.003613 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999020 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 22:09:28.003613 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999023 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 22:09:28.003613 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999025 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 22:09:28.003613 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999028 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 22:09:28.003613 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999031 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 22:09:28.003613 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999033 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 22:09:28.003613 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999036 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 22:09:28.003613 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999038 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 22:09:28.003613 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999041 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 22:09:28.003613 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999043 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 22:09:28.003613 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999047 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 22:09:28.003613 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999051 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 22:09:28.003613 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999054 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 22:09:28.003613 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999057 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 22:09:28.003613 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999060 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 22:09:28.003613 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999063 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 22:09:28.003613 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999066 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 22:09:28.003613 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999069 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 22:09:28.004100 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999072 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 22:09:28.004100 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999075 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 22:09:28.004100 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999078 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 22:09:28.004100 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999093 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 22:09:28.004100 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999097 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 22:09:28.004100 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999099 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 22:09:28.004100 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999103 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 22:09:28.004100 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999105 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 22:09:28.004100 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999108 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 22:09:28.004100 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999110 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 22:09:28.004100 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999113 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 22:09:28.004100 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999117 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 22:09:28.004100 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999120 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 22:09:28.004100 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999123 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 22:09:28.004100 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999126 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 22:09:28.004100 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999130 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 22:09:28.004100 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999133 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 22:09:28.004100 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999135 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 22:09:28.004100 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999137 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 22:09:28.004585 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999140 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 22:09:28.004585 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999143 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 22:09:28.004585 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999145 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 22:09:28.004585 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999148 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 22:09:28.004585 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999150 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 22:09:28.004585 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999153 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 22:09:28.004585 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999155 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 22:09:28.004585 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999158 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 22:09:28.004585 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999160 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 22:09:28.004585 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999163 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 22:09:28.004585 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999166 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 22:09:28.004585 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999169 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 22:09:28.004585 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999172 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 22:09:28.004585 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999174 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 22:09:28.004585 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999177 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 22:09:28.004585 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999180 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 22:09:28.004987 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.999185 2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 22:09:28.004987 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999308 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 22:09:28.004987 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999313 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 22:09:28.004987 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999316 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 22:09:28.004987 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999319 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 22:09:28.004987 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999321 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 22:09:28.004987 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999324 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 22:09:28.004987 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999326 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 22:09:28.004987 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999329 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 22:09:28.004987 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999332 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 22:09:28.004987 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999334 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 22:09:28.004987 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999337 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 22:09:28.004987 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999341 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 22:09:28.004987 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999345 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 22:09:28.005587 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999348 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 22:09:28.005587 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999350 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 22:09:28.005587 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999353 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 22:09:28.005587 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999356 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 22:09:28.005587 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999358 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 22:09:28.005587 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999361 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 22:09:28.005587 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999363 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 22:09:28.005587 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999366 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 22:09:28.005587 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999368 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 22:09:28.005587 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999371 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 22:09:28.005587 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999373 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 22:09:28.005587 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999376 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 22:09:28.005587 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999378 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 22:09:28.005587 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999381 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 22:09:28.005587 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999384 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 22:09:28.005587 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999387 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 22:09:28.005587 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999389 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 22:09:28.005587 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999392 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 22:09:28.005587 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999394 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 22:09:28.005587 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999397 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 22:09:28.006435 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999400 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 22:09:28.006435 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999402 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 22:09:28.006435 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999405 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 22:09:28.006435 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999407 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 22:09:28.006435 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999410 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 22:09:28.006435 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999412 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 22:09:28.006435 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999415 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 22:09:28.006435 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999417 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 22:09:28.006435 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999419 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 22:09:28.006435 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999422 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 22:09:28.006435 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999424 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 22:09:28.006435 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999428 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 22:09:28.006435 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999432 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 22:09:28.006435 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999435 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 22:09:28.006435 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999438 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 22:09:28.006435 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999440 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 22:09:28.006435 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999443 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 22:09:28.006435 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999445 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 22:09:28.006435 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999448 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 22:09:28.007000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999450 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 22:09:28.007000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999453 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 22:09:28.007000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999455 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 22:09:28.007000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999458 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 22:09:28.007000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999460 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 22:09:28.007000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999464 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 22:09:28.007000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999466 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 22:09:28.007000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999469 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 22:09:28.007000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999471 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 22:09:28.007000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999474 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 22:09:28.007000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999477 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 22:09:28.007000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999480 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 22:09:28.007000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999483 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 22:09:28.007000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999485 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 22:09:28.007000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999488 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 22:09:28.007000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999490 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 22:09:28.007000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999492 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 22:09:28.007000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999495 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 22:09:28.007000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999497 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 22:09:28.007000 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999499 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 22:09:28.007603 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999502 2550 feature_gate.go:328] unrecognized feature gate: Example Jul 14 22:09:28.007603 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999504 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 22:09:28.007603 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999507 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 22:09:28.007603 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999509 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 22:09:28.007603 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999512 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 22:09:28.007603 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999514 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 22:09:28.007603 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999518 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 22:09:28.007603 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999520 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 22:09:28.007603 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999523 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 22:09:28.007603 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999525 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 22:09:28.007603 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999528 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 22:09:28.007603 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999530 2550 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 22:09:28.007603 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999533 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 22:09:28.007603 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999535 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 22:09:28.007603 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999538 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 22:09:28.007603 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999541 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 22:09:28.007603 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999543 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 22:09:28.007603 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999546 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 22:09:28.007603 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999549 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 22:09:28.007603 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999551 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 22:09:28.008128 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999554 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 22:09:28.008128 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999556 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 22:09:28.008128 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999559 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 22:09:28.008128 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999561 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 22:09:28.008128 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999564 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 22:09:28.008128 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999566 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 22:09:28.008128 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:27.999569 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 22:09:28.008128 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.999573 2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 22:09:28.008128 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:27.999665 2550 server.go:962] "Client rotation is on, will bootstrap in background" Jul 14 22:09:28.008128 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.001506 2550 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 14 22:09:28.008128 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.002811 2550 server.go:1019] "Starting client certificate rotation" Jul 14 22:09:28.008128 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.002927 2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 22:09:28.008128 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.003669 2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 22:09:28.008492 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.005641 2550 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 22:09:28.009014 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.008993 2550 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 22:09:28.016725 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.016708 2550 log.go:25] "Validated CRI v1 runtime API" Jul 14 22:09:28.018646 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.018632 2550 log.go:25] "Validated CRI v1 image API" Jul 14 22:09:28.019452 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.019437 2550 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 14 22:09:28.021790 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.021773 2550 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d25894f9-2d18-4fb6-b7cd-9ae372942f85:/dev/nvme0n1p3 fa9ebd27-848d-482f-b93b-24344d23f2c6:/dev/nvme0n1p4] Jul 14 22:09:28.021856 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.021790 2550 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 14 22:09:28.027647 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.027538 2550 manager.go:216] Machine: {Timestamp:2026-07-14 22:09:28.026227732 +0000 UTC m=+0.304843154 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097979 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c9c6e1924bf15dae5bd0e5f628c1b SystemUUID:ec2c9c6e-1924-bf15-dae5-bd0e5f628c1b BootID:d63d1e42-3bfe-407e-a200-d1e513755e87 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e1:bd:83:cf:a9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e1:bd:83:cf:a9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:a8:1c:22:4c:1c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 14 22:09:28.027647 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.027642 2550 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 14 22:09:28.027768 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.027711 2550 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 14 22:09:28.028379 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.028364 2550 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 22:09:28.031141 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.030680 2550 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 14 22:09:28.031141 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.030783 2550 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-118.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 14 22:09:28.032218 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.032194 2550 topology_manager.go:138] "Creating topology manager with none policy" Jul 14 22:09:28.032218 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.032217 2550 container_manager_linux.go:309] "Creating device plugin manager" Jul 14 22:09:28.032360 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.032228 2550 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 22:09:28.033042 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.033022 2550 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 22:09:28.033882 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.033868 2550 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 14 22:09:28.036539 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.036527 2550 state_mem.go:36] "Initialized new in-memory state store" Jul 14 22:09:28.036819 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.036809 2550 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 14 22:09:28.036859 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.036841 2550 kubelet.go:486] "Attempting to sync node with API server" Jul 14 22:09:28.036887 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.036859 2550 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 14 22:09:28.036887 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.036873 2550 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 14 22:09:28.036887 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.036883 2550 kubelet.go:398] "Adding apiserver pod source" Jul 14 22:09:28.036978 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.036892 2550 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 14 22:09:28.037274 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.037261 2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 22:09:28.037336 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.037283 2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 22:09:28.039350 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.039335 2550 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 14 22:09:28.040552 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.040539 2550 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 14 22:09:28.040602 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.040569 2550 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 14 22:09:28.041146 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.041133 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 14 22:09:28.041176 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.041152 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 14 22:09:28.041176 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.041159 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 14 22:09:28.041176 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.041164 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 14 22:09:28.041176 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.041172 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 14 22:09:28.041176 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.041177 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 14 22:09:28.041305 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.041206 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 14 22:09:28.041305 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.041211 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 14 22:09:28.041305 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.041217 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 14 22:09:28.041305 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.041222 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 14 22:09:28.041305 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.041230 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 14 22:09:28.041305 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.041238 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 14 22:09:28.041904 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.041895 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 14 22:09:28.041904 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.041904 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 14 22:09:28.044711 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.044672 2550 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-118.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 14 22:09:28.045106 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.045075 2550 server.go:1268] "Started kubelet" Jul 14 22:09:28.045315 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:28.045254 2550 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 14 22:09:28.045315 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:28.045260 2550 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-118.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 14 22:09:28.045441 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.045352 2550 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 14 22:09:28.045441 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.045340 2550 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 14 22:09:28.045550 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.045522 2550 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 14 22:09:28.046130 ip-10-0-133-118 systemd[1]: Started Kubernetes Kubelet. Jul 14 22:09:28.046750 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.046733 2550 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 14 22:09:28.047180 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.047169 2550 server.go:310] "Adding debug handlers to kubelet server" Jul 14 22:09:28.050497 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:28.050478 2550 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 14 22:09:28.051651 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.051634 2550 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 14 22:09:28.051749 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.051649 2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 14 22:09:28.051749 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:28.051736 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-118.ec2.internal\" not found" Jul 14 22:09:28.052209 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.052194 2550 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 14 22:09:28.052209 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.052196 2550 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 14 22:09:28.052358 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.052221 2550 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 14 22:09:28.052358 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:28.051201 2550 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-118.ec2.internal.18c247747dd54d5f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-118.ec2.internal,UID:ip-10-0-133-118.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-118.ec2.internal,},FirstTimestamp:2026-07-14 22:09:28.045047135 +0000 UTC m=+0.323662559,LastTimestamp:2026-07-14 22:09:28.045047135 +0000 UTC m=+0.323662559,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-118.ec2.internal,}" Jul 14 22:09:28.052481 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.052382 2550 reconstruct.go:97] "Volume reconstruction finished" Jul 14 22:09:28.052481 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.052390 2550 reconciler.go:29] "Reconciler: start to sync state" Jul 14 22:09:28.052869 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.052855 2550 factory.go:55] Registering systemd factory Jul 14 22:09:28.052937 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.052874 2550 factory.go:223] Registration of the systemd container factory successfully Jul 14 22:09:28.053098 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.053063 2550 factory.go:153] Registering CRI-O factory Jul 14 22:09:28.053098 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.053092 2550 factory.go:223] Registration of the crio container factory successfully Jul 14 22:09:28.053211 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.053152 2550 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 14 22:09:28.053211 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.053173 2550 factory.go:103] Registering Raw factory Jul 14 22:09:28.053211 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.053188 2550 manager.go:1196] Started watching for new ooms in manager Jul 14 22:09:28.053531 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.053519 2550 manager.go:349] Starting recovery of all containers Jul 14 22:09:28.069855 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:28.069664 2550 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 14 22:09:28.069855 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:28.069778 2550 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-118.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 14 22:09:28.070157 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.070136 2550 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b7njh" Jul 14 22:09:28.070211 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.070167 2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b7njh" Jul 14 22:09:28.076853 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.076836 2550 manager.go:354] Recovery completed Jul 14 22:09:28.081253 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.081241 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 22:09:28.082909 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.082889 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasSufficientMemory" Jul 14 22:09:28.082981 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.082915 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 22:09:28.082981 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.082928 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasSufficientPID" Jul 14 22:09:28.083419 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.083406 2550 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 14 22:09:28.083419 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.083416 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 22:09:28.083496 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.083418 2550 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 14 22:09:28.083496 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.083490 2550 state_mem.go:36] "Initialized new in-memory state store" Jul 14 22:09:28.084036 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.084023 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasSufficientMemory" Jul 14 22:09:28.084075 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.084048 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 22:09:28.084075 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.084059 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasSufficientPID" Jul 14 22:09:28.084649 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.084631 2550 policy_none.go:49] "None policy: Start" Jul 14 22:09:28.084649 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.084652 2550 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 14 22:09:28.084767 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.084666 2550 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 14 22:09:28.084893 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:28.084820 2550 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-118.ec2.internal.18c247748016ebbf default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-118.ec2.internal,UID:ip-10-0-133-118.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-118.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-118.ec2.internal,},FirstTimestamp:2026-07-14 22:09:28.082901951 +0000 UTC m=+0.361517372,LastTimestamp:2026-07-14 22:09:28.082901951 +0000 UTC m=+0.361517372,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-118.ec2.internal,}" Jul 14 22:09:28.085723 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.085711 2550 policy_none.go:47] "Start" Jul 14 22:09:28.126596 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.126582 2550 manager.go:337] "Starting Device Plugin manager" Jul 14 22:09:28.126683 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:28.126622 2550 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 14 22:09:28.126683 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.126631 2550 server.go:87] "Starting device plugin registration server" Jul 14 22:09:28.126873 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.126860 2550 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 14 22:09:28.126973 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.126876 2550 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 14 22:09:28.126973 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.126955 2550 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 14 22:09:28.127063 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.127043 2550 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 14 22:09:28.127063 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.127052 2550 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 14 22:09:28.127581 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:28.127562 2550 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 14 22:09:28.127663 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:28.127616 2550 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-118.ec2.internal\" not found" Jul 14 22:09:28.227322 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.227291 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 22:09:28.228457 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.228440 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasSufficientMemory" Jul 14 22:09:28.228523 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.228473 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 22:09:28.228523 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.228489 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasSufficientPID" Jul 14 22:09:28.228588 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.228524 2550 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-118.ec2.internal" Jul 14 22:09:28.235010 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.234970 2550 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-118.ec2.internal" Jul 14 22:09:28.235010 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:28.234992 2550 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-118.ec2.internal\": node \"ip-10-0-133-118.ec2.internal\" not found" Jul 14 22:09:28.252093 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:28.252057 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-118.ec2.internal\" not found" Jul 14 22:09:28.268276 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.268245 2550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 14 22:09:28.269429 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.269405 2550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 14 22:09:28.269516 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.269449 2550 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 14 22:09:28.269516 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.269476 2550 kubelet.go:2448] "Starting kubelet main sync loop" Jul 14 22:09:28.269616 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:28.269518 2550 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 14 22:09:28.271801 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.271785 2550 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 22:09:28.352652 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:28.352635 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-118.ec2.internal\" not found" Jul 14 22:09:28.369952 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.369931 2550 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal"] Jul 14 22:09:28.369996 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.369989 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 22:09:28.371503 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.371490 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasSufficientMemory" Jul 14 22:09:28.371556 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.371518 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 22:09:28.371556 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.371528 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasSufficientPID" Jul 14 22:09:28.372938 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.372927 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 22:09:28.373073 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.373041 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal" Jul 14 22:09:28.373187 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.373094 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 22:09:28.373660 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.373647 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasSufficientMemory" Jul 14 22:09:28.373712 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.373668 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 22:09:28.373712 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.373680 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasSufficientPID" Jul 14 22:09:28.373712 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.373704 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasSufficientMemory" Jul 14 22:09:28.373816 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.373724 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 22:09:28.373816 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.373733 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasSufficientPID" Jul 14 22:09:28.375533 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.375519 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal" Jul 14 22:09:28.375603 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.375542 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 22:09:28.376239 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.376223 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasSufficientMemory" Jul 14 22:09:28.376313 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.376249 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 22:09:28.376313 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.376259 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeHasSufficientPID" Jul 14 22:09:28.397729 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:28.397710 2550 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-118.ec2.internal\" not found" node="ip-10-0-133-118.ec2.internal" Jul 14 22:09:28.402931 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:28.402915 2550 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-118.ec2.internal\" not found" node="ip-10-0-133-118.ec2.internal" Jul 14 22:09:28.453548 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:28.453527 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-118.ec2.internal\" not found" Jul 14 22:09:28.454894 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.454876 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/da6abbb4b54c9a3e9a35bc6fff12e4c3-config\") pod \"kube-apiserver-proxy-ip-10-0-133-118.ec2.internal\" (UID: \"da6abbb4b54c9a3e9a35bc6fff12e4c3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal" Jul 14 22:09:28.454945 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.454905 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/70253776f2fd132651fc1b4e46ac7c7d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal\" (UID: \"70253776f2fd132651fc1b4e46ac7c7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal" Jul 14 22:09:28.454945 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.454922 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70253776f2fd132651fc1b4e46ac7c7d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal\" (UID: \"70253776f2fd132651fc1b4e46ac7c7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal" Jul 14 22:09:28.554288 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:28.554224 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-118.ec2.internal\" not found" Jul 14 22:09:28.555330 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.555312 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/da6abbb4b54c9a3e9a35bc6fff12e4c3-config\") pod \"kube-apiserver-proxy-ip-10-0-133-118.ec2.internal\" (UID: \"da6abbb4b54c9a3e9a35bc6fff12e4c3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal" Jul 14 22:09:28.555387 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.555339 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/70253776f2fd132651fc1b4e46ac7c7d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal\" (UID: \"70253776f2fd132651fc1b4e46ac7c7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal" Jul 14 22:09:28.555387 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.555356 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70253776f2fd132651fc1b4e46ac7c7d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal\" (UID: \"70253776f2fd132651fc1b4e46ac7c7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal" Jul 14 22:09:28.555448 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.555395 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70253776f2fd132651fc1b4e46ac7c7d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal\" (UID: \"70253776f2fd132651fc1b4e46ac7c7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal" Jul 14 22:09:28.555448 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.555416 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/da6abbb4b54c9a3e9a35bc6fff12e4c3-config\") pod \"kube-apiserver-proxy-ip-10-0-133-118.ec2.internal\" (UID: \"da6abbb4b54c9a3e9a35bc6fff12e4c3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal" Jul 14 22:09:28.555448 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.555420 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/70253776f2fd132651fc1b4e46ac7c7d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal\" (UID: \"70253776f2fd132651fc1b4e46ac7c7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal" Jul 14 22:09:28.654656 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:28.654620 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-118.ec2.internal\" not found" Jul 14 22:09:28.700907 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.700885 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal" Jul 14 22:09:28.705703 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:28.705683 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal" Jul 14 22:09:28.755487 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:28.755460 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-118.ec2.internal\" not found" Jul 14 22:09:28.855917 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:28.855854 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-118.ec2.internal\" not found" Jul 14 22:09:28.956351 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:28.956324 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-118.ec2.internal\" not found" Jul 14 22:09:29.002549 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:29.002532 2550 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 14 22:09:29.003065 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:29.002669 2550 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 14 22:09:29.057205 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:29.057189 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-118.ec2.internal\" not found" Jul 14 22:09:29.074229 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:29.074200 2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-13 22:04:29 +0000 UTC" deadline="2028-03-04 09:40:34.133294506 +0000 UTC" Jul 14 22:09:29.074229 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:29.074225 2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14363h31m5.059072058s" Jul 14 22:09:29.113063 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:29.113010 2550 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 22:09:29.157913 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:29.157880 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-118.ec2.internal\" not found" Jul 14 22:09:29.247898 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:29.247871 2550 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 22:09:29.258346 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:29.258313 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-118.ec2.internal\" not found" Jul 14 22:09:29.302237 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:29.302207 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6abbb4b54c9a3e9a35bc6fff12e4c3.slice/crio-769a7b973ec1444f2f7483cd1c53f51ef9f994edf511cd1b07e15564fb84c801 WatchSource:0}: Error finding container 769a7b973ec1444f2f7483cd1c53f51ef9f994edf511cd1b07e15564fb84c801: Status 404 returned error can't find the container with id 769a7b973ec1444f2f7483cd1c53f51ef9f994edf511cd1b07e15564fb84c801 Jul 14 22:09:29.302487 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:29.302473 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70253776f2fd132651fc1b4e46ac7c7d.slice/crio-7650875d57173e4a2a3684f5fdbdfa2f85190e6ab613230a5e64d603258894ff WatchSource:0}: Error finding container 7650875d57173e4a2a3684f5fdbdfa2f85190e6ab613230a5e64d603258894ff: Status 404 returned error can't find the container with id 7650875d57173e4a2a3684f5fdbdfa2f85190e6ab613230a5e64d603258894ff Jul 14 22:09:29.306382 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:29.306367 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 22:09:29.358580 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:29.358542 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-118.ec2.internal\" not found" Jul 14 22:09:31.378409 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:31.378391 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-118.ec2.internal\" not found" Jul 14 22:09:31.420131 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.420113 2550 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 22:09:31.472329 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.472310 2550 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal" Jul 14 22:09:31.486806 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.486787 2550 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 22:09:31.487926 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.487914 2550 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal" Jul 14 22:09:31.497262 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.497247 2550 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 22:09:31.761150 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.761081 2550 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 22:09:31.957650 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.957623 2550 apiserver.go:52] "Watching apiserver" Jul 14 22:09:31.963818 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.963792 2550 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 14 22:09:31.964246 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.964214 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal","openshift-multus/multus-wtrck","openshift-multus/network-metrics-daemon-55gw6","kube-system/global-pull-secret-syncer-6g6bp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsf6v","openshift-dns/node-resolver-6tt4w","openshift-image-registry/node-ca-dj68k","openshift-multus/multus-additional-cni-plugins-l9qrh","openshift-network-diagnostics/network-check-target-6p45d","openshift-network-operator/iptables-alerter-kws8r","openshift-ovn-kubernetes/ovnkube-node-zj4sb","kube-system/konnectivity-agent-gmmnx","kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal","openshift-cluster-node-tuning-operator/tuned-jdnlk"] Jul 14 22:09:31.965914 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.965867 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dj68k" Jul 14 22:09:31.968398 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.968379 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lkqr6\"" Jul 14 22:09:31.968488 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.968440 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 14 22:09:31.968896 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.968877 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:31.969640 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.969428 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 14 22:09:31.969737 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.969665 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 14 22:09:31.971018 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.970998 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2rqrb\"" Jul 14 22:09:31.971157 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.971139 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:09:31.971241 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:31.971210 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55gw6" podUID="7f9358e0-96a8-473f-896b-907e7129d452" Jul 14 22:09:31.971604 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.971470 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 14 22:09:31.971604 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.971524 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 14 22:09:31.971604 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.971537 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 14 22:09:31.971799 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.971781 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 14 22:09:31.971953 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.971841 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 14 22:09:31.971953 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.971938 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 14 22:09:31.973039 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.972484 2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 14 22:09:31.973039 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.972860 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:09:31.973039 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:31.972943 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6bp" podUID="ce5ffbb6-99ac-4817-acd9-f6cd4577f724" Jul 14 22:09:31.977853 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.977832 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsf6v" Jul 14 22:09:31.979409 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.979389 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6tt4w" Jul 14 22:09:31.980468 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.980450 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 14 22:09:31.980701 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.980683 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-59qb8\"" Jul 14 22:09:31.980701 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.980695 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 14 22:09:31.980846 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.980576 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 14 22:09:31.981084 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.981067 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6p45d" Jul 14 22:09:31.981181 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:31.981145 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6p45d" podUID="a549c30a-9937-422b-9f9b-17d10c48ab25" Jul 14 22:09:31.981872 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.981852 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 14 22:09:31.981962 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.981856 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vn8qq\"" Jul 14 22:09:31.982125 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.982107 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 14 22:09:31.982225 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.982104 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lcbg\" (UniqueName: \"kubernetes.io/projected/7f9358e0-96a8-473f-896b-907e7129d452-kube-api-access-8lcbg\") pod \"network-metrics-daemon-55gw6\" (UID: \"7f9358e0-96a8-473f-896b-907e7129d452\") " pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:09:31.982225 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.982153 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0884baa5-9206-4196-a7c9-b5686623311b-ovnkube-script-lib\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:31.982225 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.982180 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-host-cni-netd\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:31.982225 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.982204 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8qvr\" (UniqueName: \"kubernetes.io/projected/8c265d22-e8f9-4516-aaf7-b274420c1b23-kube-api-access-z8qvr\") pod \"node-ca-dj68k\" (UID: \"8c265d22-e8f9-4516-aaf7-b274420c1b23\") " pod="openshift-image-registry/node-ca-dj68k" Jul 14 22:09:31.982431 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.982228 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-host-kubelet\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:31.982431 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.982256 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:31.982431 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.982283 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f9358e0-96a8-473f-896b-907e7129d452-metrics-certs\") pod \"network-metrics-daemon-55gw6\" (UID: \"7f9358e0-96a8-473f-896b-907e7129d452\") " pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:09:31.982431 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.982317 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2g6p\" (UniqueName: \"kubernetes.io/projected/0884baa5-9206-4196-a7c9-b5686623311b-kube-api-access-v2g6p\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:31.982431 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.982358 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8c265d22-e8f9-4516-aaf7-b274420c1b23-serviceca\") pod \"node-ca-dj68k\" (UID: \"8c265d22-e8f9-4516-aaf7-b274420c1b23\") " pod="openshift-image-registry/node-ca-dj68k" Jul 14 22:09:31.982431 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.982387 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-systemd-units\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:31.982431 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.982410 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-host-slash\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:31.982431 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.982431 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-run-systemd\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:31.983035 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.982452 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-run-ovn\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:31.983035 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.982475 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0884baa5-9206-4196-a7c9-b5686623311b-ovn-node-metrics-cert\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:31.983035 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.982507 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce5ffbb6-99ac-4817-acd9-f6cd4577f724-original-pull-secret\") pod \"global-pull-secret-syncer-6g6bp\" (UID: \"ce5ffbb6-99ac-4817-acd9-f6cd4577f724\") " pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:09:31.983035 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.982569 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8c265d22-e8f9-4516-aaf7-b274420c1b23-host\") pod \"node-ca-dj68k\" (UID: \"8c265d22-e8f9-4516-aaf7-b274420c1b23\") " pod="openshift-image-registry/node-ca-dj68k" Jul 14 22:09:31.983035 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.982597 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-run-openvswitch\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:31.983035 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.982636 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-host-run-ovn-kubernetes\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:31.983035 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.982664 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-host-cni-bin\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:31.983035 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.982695 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ce5ffbb6-99ac-4817-acd9-f6cd4577f724-dbus\") pod \"global-pull-secret-syncer-6g6bp\" (UID: \"ce5ffbb6-99ac-4817-acd9-f6cd4577f724\") " pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:09:31.983035 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.982716 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-var-lib-openvswitch\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:31.983035 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.982738 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-host-run-netns\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:31.983035 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.982763 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-etc-openvswitch\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:31.983035 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.982809 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-node-log\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:31.983035 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.982873 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-log-socket\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:31.983035 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.982902 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0884baa5-9206-4196-a7c9-b5686623311b-env-overrides\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:31.983035 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.982929 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ce5ffbb6-99ac-4817-acd9-f6cd4577f724-kubelet-config\") pod \"global-pull-secret-syncer-6g6bp\" (UID: \"ce5ffbb6-99ac-4817-acd9-f6cd4577f724\") " pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:09:31.983035 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.982953 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0884baa5-9206-4196-a7c9-b5686623311b-ovnkube-config\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:31.983721 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.983105 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:31.984338 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.984300 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wtrck" Jul 14 22:09:31.985933 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.985636 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 14 22:09:31.985933 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.985782 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ggb8x\"" Jul 14 22:09:31.986117 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.986048 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kws8r" Jul 14 22:09:31.986355 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.986338 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 14 22:09:31.986981 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.986812 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 14 22:09:31.987208 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.987189 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 14 22:09:31.987283 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.987218 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 14 22:09:31.987283 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.987269 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 14 22:09:31.987372 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.987188 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9f8qp\"" Jul 14 22:09:31.988096 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.988080 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 14 22:09:31.988380 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.988340 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l9qrh" Jul 14 22:09:31.988721 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.988707 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 14 22:09:31.988808 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.988757 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 14 22:09:31.989686 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.989191 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j89dg\"" Jul 14 22:09:31.989775 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.989760 2550 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 22:09:31.990425 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.990287 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gmmnx" Jul 14 22:09:31.990785 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.990766 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 14 22:09:31.990895 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.990851 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-j6txh\"" Jul 14 22:09:31.991455 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.991360 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 14 22:09:31.992823 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.992806 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-sfn4f\"" Jul 14 22:09:31.993059 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.993046 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 14 22:09:31.993479 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:31.993462 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 14 22:09:32.010633 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.010612 2550 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-b8czp" Jul 14 22:09:32.018894 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.018872 2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-b8czp" Jul 14 22:09:32.073168 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.073146 2550 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 14 22:09:32.083693 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.083667 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8qvr\" (UniqueName: \"kubernetes.io/projected/8c265d22-e8f9-4516-aaf7-b274420c1b23-kube-api-access-z8qvr\") pod \"node-ca-dj68k\" (UID: \"8c265d22-e8f9-4516-aaf7-b274420c1b23\") " pod="openshift-image-registry/node-ca-dj68k" Jul 14 22:09:32.083831 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.083698 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-host-kubelet\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.083831 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.083722 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe40e4f0-af16-4743-b435-48a292cd5fc4-etc-kubernetes\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.083831 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.083744 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnqhc\" (UniqueName: \"kubernetes.io/projected/fe40e4f0-af16-4743-b435-48a292cd5fc4-kube-api-access-xnqhc\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.083831 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.083767 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0147c9ad-b474-4fda-8ad1-0661515acfdb-multus-daemon-config\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.083831 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.083789 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kqrw\" (UniqueName: \"kubernetes.io/projected/2cbd6276-0837-4e3e-b5d0-1770b73f48a2-kube-api-access-5kqrw\") pod \"aws-ebs-csi-driver-node-nsf6v\" (UID: \"2cbd6276-0837-4e3e-b5d0-1770b73f48a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsf6v" Jul 14 22:09:32.083831 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.083815 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2g6p\" (UniqueName: \"kubernetes.io/projected/0884baa5-9206-4196-a7c9-b5686623311b-kube-api-access-v2g6p\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.084041 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.083835 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe40e4f0-af16-4743-b435-48a292cd5fc4-sys\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.084041 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.083837 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-host-kubelet\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.084041 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.083855 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe40e4f0-af16-4743-b435-48a292cd5fc4-lib-modules\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.084041 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.083874 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe40e4f0-af16-4743-b435-48a292cd5fc4-tmp\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.084041 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.083910 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca583798-970a-42ec-a335-edb5a50383ad-host-slash\") pod \"iptables-alerter-kws8r\" (UID: \"ca583798-970a-42ec-a335-edb5a50383ad\") " pod="openshift-network-operator/iptables-alerter-kws8r" Jul 14 22:09:32.084041 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.083938 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5f7e8bb-6df3-4c30-84c7-4e30f116545a-system-cni-dir\") pod \"multus-additional-cni-plugins-l9qrh\" (UID: \"d5f7e8bb-6df3-4c30-84c7-4e30f116545a\") " pod="openshift-multus/multus-additional-cni-plugins-l9qrh" Jul 14 22:09:32.084041 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.083963 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5f7e8bb-6df3-4c30-84c7-4e30f116545a-cnibin\") pod \"multus-additional-cni-plugins-l9qrh\" (UID: \"d5f7e8bb-6df3-4c30-84c7-4e30f116545a\") " pod="openshift-multus/multus-additional-cni-plugins-l9qrh" Jul 14 22:09:32.084041 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.083989 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2cbd6276-0837-4e3e-b5d0-1770b73f48a2-etc-selinux\") pod \"aws-ebs-csi-driver-node-nsf6v\" (UID: \"2cbd6276-0837-4e3e-b5d0-1770b73f48a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsf6v" Jul 14 22:09:32.084041 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.084014 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxdn4\" (UniqueName: \"kubernetes.io/projected/ca583798-970a-42ec-a335-edb5a50383ad-kube-api-access-sxdn4\") pod \"iptables-alerter-kws8r\" (UID: \"ca583798-970a-42ec-a335-edb5a50383ad\") " pod="openshift-network-operator/iptables-alerter-kws8r" Jul 14 22:09:32.084377 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.084051 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8c265d22-e8f9-4516-aaf7-b274420c1b23-serviceca\") pod \"node-ca-dj68k\" (UID: \"8c265d22-e8f9-4516-aaf7-b274420c1b23\") " pod="openshift-image-registry/node-ca-dj68k" Jul 14 22:09:32.084377 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.084107 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-host-slash\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.084377 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.084212 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-run-systemd\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.084377 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.084239 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-host-slash\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.084377 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.084279 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fe40e4f0-af16-4743-b435-48a292cd5fc4-etc-sysconfig\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.084377 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.084305 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fe40e4f0-af16-4743-b435-48a292cd5fc4-etc-sysctl-d\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.084377 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.084360 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-run-systemd\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.084660 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.084388 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3a228cf3-be18-4e92-bc1b-0045d1b2c823-konnectivity-ca\") pod \"konnectivity-agent-gmmnx\" (UID: \"3a228cf3-be18-4e92-bc1b-0045d1b2c823\") " pod="kube-system/konnectivity-agent-gmmnx" Jul 14 22:09:32.084660 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.084419 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce5ffbb6-99ac-4817-acd9-f6cd4577f724-original-pull-secret\") pod \"global-pull-secret-syncer-6g6bp\" (UID: \"ce5ffbb6-99ac-4817-acd9-f6cd4577f724\") " pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:09:32.084660 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.084455 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8c265d22-e8f9-4516-aaf7-b274420c1b23-host\") pod \"node-ca-dj68k\" (UID: \"8c265d22-e8f9-4516-aaf7-b274420c1b23\") " pod="openshift-image-registry/node-ca-dj68k" Jul 14 22:09:32.084660 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.084493 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-run-openvswitch\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.084660 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:32.084514 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 22:09:32.084660 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.084568 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8c265d22-e8f9-4516-aaf7-b274420c1b23-host\") pod \"node-ca-dj68k\" (UID: \"8c265d22-e8f9-4516-aaf7-b274420c1b23\") " pod="openshift-image-registry/node-ca-dj68k" Jul 14 22:09:32.084660 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.084589 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8c265d22-e8f9-4516-aaf7-b274420c1b23-serviceca\") pod \"node-ca-dj68k\" (UID: \"8c265d22-e8f9-4516-aaf7-b274420c1b23\") " pod="openshift-image-registry/node-ca-dj68k" Jul 14 22:09:32.084660 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:32.084609 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce5ffbb6-99ac-4817-acd9-f6cd4577f724-original-pull-secret podName:ce5ffbb6-99ac-4817-acd9-f6cd4577f724 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:32.584574435 +0000 UTC m=+2.943720866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce5ffbb6-99ac-4817-acd9-f6cd4577f724-original-pull-secret") pod "global-pull-secret-syncer-6g6bp" (UID: "ce5ffbb6-99ac-4817-acd9-f6cd4577f724") : object "kube-system"/"original-pull-secret" not registered Jul 14 22:09:32.084660 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.084611 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-run-openvswitch\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.084660 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.084516 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fe40e4f0-af16-4743-b435-48a292cd5fc4-run\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.085133 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.084667 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5f7e8bb-6df3-4c30-84c7-4e30f116545a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l9qrh\" (UID: \"d5f7e8bb-6df3-4c30-84c7-4e30f116545a\") " pod="openshift-multus/multus-additional-cni-plugins-l9qrh" Jul 14 22:09:32.085133 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.084688 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3a228cf3-be18-4e92-bc1b-0045d1b2c823-agent-certs\") pod \"konnectivity-agent-gmmnx\" (UID: \"3a228cf3-be18-4e92-bc1b-0045d1b2c823\") " pod="kube-system/konnectivity-agent-gmmnx" Jul 14 22:09:32.085133 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.084710 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2cbd6276-0837-4e3e-b5d0-1770b73f48a2-sys-fs\") pod \"aws-ebs-csi-driver-node-nsf6v\" (UID: \"2cbd6276-0837-4e3e-b5d0-1770b73f48a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsf6v" Jul 14 22:09:32.085133 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.084738 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1159189b-384f-4107-807e-250f243409df-tmp-dir\") pod \"node-resolver-6tt4w\" (UID: \"1159189b-384f-4107-807e-250f243409df\") " pod="openshift-dns/node-resolver-6tt4w" Jul 14 22:09:32.085133 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.084772 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-var-lib-openvswitch\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.085133 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.084808 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-system-cni-dir\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.085133 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.084835 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-host-run-netns\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.085133 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.084867 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-hostroot\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.085133 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.084889 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-multus-conf-dir\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.085133 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.084911 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1159189b-384f-4107-807e-250f243409df-hosts-file\") pod \"node-resolver-6tt4w\" (UID: \"1159189b-384f-4107-807e-250f243409df\") " pod="openshift-dns/node-resolver-6tt4w" Jul 14 22:09:32.085133 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.084954 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-node-log\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.085133 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.084995 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-log-socket\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.085133 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.084990 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-var-lib-openvswitch\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.085133 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.085032 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fe40e4f0-af16-4743-b435-48a292cd5fc4-etc-modprobe-d\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.085133 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.085042 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-log-socket\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.085133 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.085044 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-node-log\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.085133 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.085055 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fe40e4f0-af16-4743-b435-48a292cd5fc4-etc-sysctl-conf\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.085930 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.085076 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe40e4f0-af16-4743-b435-48a292cd5fc4-host\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.085930 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.085109 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2cbd6276-0837-4e3e-b5d0-1770b73f48a2-socket-dir\") pod \"aws-ebs-csi-driver-node-nsf6v\" (UID: \"2cbd6276-0837-4e3e-b5d0-1770b73f48a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsf6v" Jul 14 22:09:32.085930 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.085137 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5gxh\" (UniqueName: \"kubernetes.io/projected/1159189b-384f-4107-807e-250f243409df-kube-api-access-f5gxh\") pod \"node-resolver-6tt4w\" (UID: \"1159189b-384f-4107-807e-250f243409df\") " pod="openshift-dns/node-resolver-6tt4w" Jul 14 22:09:32.085930 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.085169 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0884baa5-9206-4196-a7c9-b5686623311b-env-overrides\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.085930 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.085223 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0884baa5-9206-4196-a7c9-b5686623311b-ovnkube-config\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.085930 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.085248 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lcbg\" (UniqueName: \"kubernetes.io/projected/7f9358e0-96a8-473f-896b-907e7129d452-kube-api-access-8lcbg\") pod \"network-metrics-daemon-55gw6\" (UID: \"7f9358e0-96a8-473f-896b-907e7129d452\") " pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:09:32.085930 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.085272 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-etc-kubernetes\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.085930 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.085304 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krq6v\" (UniqueName: \"kubernetes.io/projected/0147c9ad-b474-4fda-8ad1-0661515acfdb-kube-api-access-krq6v\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.085930 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.085332 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5f7e8bb-6df3-4c30-84c7-4e30f116545a-cni-binary-copy\") pod \"multus-additional-cni-plugins-l9qrh\" (UID: \"d5f7e8bb-6df3-4c30-84c7-4e30f116545a\") " pod="openshift-multus/multus-additional-cni-plugins-l9qrh" Jul 14 22:09:32.085930 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.085365 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d5f7e8bb-6df3-4c30-84c7-4e30f116545a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l9qrh\" (UID: \"d5f7e8bb-6df3-4c30-84c7-4e30f116545a\") " pod="openshift-multus/multus-additional-cni-plugins-l9qrh" Jul 14 22:09:32.085930 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.085391 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-host-cni-netd\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.085930 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.085413 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fe40e4f0-af16-4743-b435-48a292cd5fc4-etc-systemd\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.085930 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.085442 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d5f7e8bb-6df3-4c30-84c7-4e30f116545a-os-release\") pod \"multus-additional-cni-plugins-l9qrh\" (UID: \"d5f7e8bb-6df3-4c30-84c7-4e30f116545a\") " pod="openshift-multus/multus-additional-cni-plugins-l9qrh" Jul 14 22:09:32.085930 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.085465 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2cbd6276-0837-4e3e-b5d0-1770b73f48a2-registration-dir\") pod \"aws-ebs-csi-driver-node-nsf6v\" (UID: \"2cbd6276-0837-4e3e-b5d0-1770b73f48a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsf6v" Jul 14 22:09:32.085930 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.085490 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.085930 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.085515 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f9358e0-96a8-473f-896b-907e7129d452-metrics-certs\") pod \"network-metrics-daemon-55gw6\" (UID: \"7f9358e0-96a8-473f-896b-907e7129d452\") " pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:09:32.086616 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.085538 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-cnibin\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.086616 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.085609 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-host-cni-netd\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.086616 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.085630 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.086616 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.085642 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0884baa5-9206-4196-a7c9-b5686623311b-env-overrides\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.086616 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.085669 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-host-run-k8s-cni-cncf-io\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.086616 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.085733 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-host-var-lib-cni-bin\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.086616 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:32.085703 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 22:09:32.086616 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.085757 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-host-var-lib-cni-multus\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.086616 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:32.085857 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f9358e0-96a8-473f-896b-907e7129d452-metrics-certs podName:7f9358e0-96a8-473f-896b-907e7129d452 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:32.585811778 +0000 UTC m=+2.944958200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f9358e0-96a8-473f-896b-907e7129d452-metrics-certs") pod "network-metrics-daemon-55gw6" (UID: "7f9358e0-96a8-473f-896b-907e7129d452") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 22:09:32.086616 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.085897 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0884baa5-9206-4196-a7c9-b5686623311b-ovnkube-config\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.086616 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.085920 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-multus-cni-dir\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.086616 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.085984 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-os-release\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.086616 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.086010 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-multus-socket-dir-parent\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.086616 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.086052 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ca583798-970a-42ec-a335-edb5a50383ad-iptables-alerter-script\") pod \"iptables-alerter-kws8r\" (UID: \"ca583798-970a-42ec-a335-edb5a50383ad\") " pod="openshift-network-operator/iptables-alerter-kws8r" Jul 14 22:09:32.086616 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.086080 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-systemd-units\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.086616 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.086106 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-run-ovn\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.086616 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.086131 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrtk5\" (UniqueName: \"kubernetes.io/projected/a549c30a-9937-422b-9f9b-17d10c48ab25-kube-api-access-wrtk5\") pod \"network-check-target-6p45d\" (UID: \"a549c30a-9937-422b-9f9b-17d10c48ab25\") " pod="openshift-network-diagnostics/network-check-target-6p45d" Jul 14 22:09:32.087326 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.086154 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0147c9ad-b474-4fda-8ad1-0661515acfdb-cni-binary-copy\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.087326 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.086171 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-systemd-units\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.087326 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.086181 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-run-ovn\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.087326 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.086178 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0884baa5-9206-4196-a7c9-b5686623311b-ovn-node-metrics-cert\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.087326 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.086237 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-host-run-ovn-kubernetes\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.087326 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.086309 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-host-cni-bin\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.087326 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.086333 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-host-run-ovn-kubernetes\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.087326 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.086337 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fe40e4f0-af16-4743-b435-48a292cd5fc4-etc-tuned\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.087326 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.086415 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ce5ffbb6-99ac-4817-acd9-f6cd4577f724-dbus\") pod \"global-pull-secret-syncer-6g6bp\" (UID: \"ce5ffbb6-99ac-4817-acd9-f6cd4577f724\") " pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:09:32.087326 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.086381 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-host-cni-bin\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.087326 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.086442 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe40e4f0-af16-4743-b435-48a292cd5fc4-var-lib-kubelet\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.087326 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.086465 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-host-run-multus-certs\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.087326 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.086487 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2cbd6276-0837-4e3e-b5d0-1770b73f48a2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nsf6v\" (UID: \"2cbd6276-0837-4e3e-b5d0-1770b73f48a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsf6v" Jul 14 22:09:32.087326 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.086508 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2cbd6276-0837-4e3e-b5d0-1770b73f48a2-device-dir\") pod \"aws-ebs-csi-driver-node-nsf6v\" (UID: \"2cbd6276-0837-4e3e-b5d0-1770b73f48a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsf6v" Jul 14 22:09:32.087326 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.086520 2550 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 14 22:09:32.087326 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.086530 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-host-run-netns\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.087326 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.086570 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ce5ffbb6-99ac-4817-acd9-f6cd4577f724-dbus\") pod \"global-pull-secret-syncer-6g6bp\" (UID: \"ce5ffbb6-99ac-4817-acd9-f6cd4577f724\") " pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:09:32.088144 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.086582 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-etc-openvswitch\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.088144 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.086594 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-host-run-netns\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.088144 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.086614 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ce5ffbb6-99ac-4817-acd9-f6cd4577f724-kubelet-config\") pod \"global-pull-secret-syncer-6g6bp\" (UID: \"ce5ffbb6-99ac-4817-acd9-f6cd4577f724\") " pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:09:32.088144 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.086637 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0884baa5-9206-4196-a7c9-b5686623311b-etc-openvswitch\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.088144 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.086643 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv6ds\" (UniqueName: \"kubernetes.io/projected/d5f7e8bb-6df3-4c30-84c7-4e30f116545a-kube-api-access-nv6ds\") pod \"multus-additional-cni-plugins-l9qrh\" (UID: \"d5f7e8bb-6df3-4c30-84c7-4e30f116545a\") " pod="openshift-multus/multus-additional-cni-plugins-l9qrh" Jul 14 22:09:32.088144 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.086671 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0884baa5-9206-4196-a7c9-b5686623311b-ovnkube-script-lib\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.088144 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.086691 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ce5ffbb6-99ac-4817-acd9-f6cd4577f724-kubelet-config\") pod \"global-pull-secret-syncer-6g6bp\" (UID: \"ce5ffbb6-99ac-4817-acd9-f6cd4577f724\") " pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:09:32.088144 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.086705 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-host-var-lib-kubelet\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.088144 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.086734 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d5f7e8bb-6df3-4c30-84c7-4e30f116545a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l9qrh\" (UID: \"d5f7e8bb-6df3-4c30-84c7-4e30f116545a\") " pod="openshift-multus/multus-additional-cni-plugins-l9qrh" Jul 14 22:09:32.088144 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.087785 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0884baa5-9206-4196-a7c9-b5686623311b-ovnkube-script-lib\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.090141 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.090122 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0884baa5-9206-4196-a7c9-b5686623311b-ovn-node-metrics-cert\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.095307 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.095272 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8qvr\" (UniqueName: \"kubernetes.io/projected/8c265d22-e8f9-4516-aaf7-b274420c1b23-kube-api-access-z8qvr\") pod \"node-ca-dj68k\" (UID: \"8c265d22-e8f9-4516-aaf7-b274420c1b23\") " pod="openshift-image-registry/node-ca-dj68k" Jul 14 22:09:32.097182 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.097157 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2g6p\" (UniqueName: \"kubernetes.io/projected/0884baa5-9206-4196-a7c9-b5686623311b-kube-api-access-v2g6p\") pod \"ovnkube-node-zj4sb\" (UID: \"0884baa5-9206-4196-a7c9-b5686623311b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.098847 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.098808 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lcbg\" (UniqueName: \"kubernetes.io/projected/7f9358e0-96a8-473f-896b-907e7129d452-kube-api-access-8lcbg\") pod \"network-metrics-daemon-55gw6\" (UID: \"7f9358e0-96a8-473f-896b-907e7129d452\") " pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:09:32.187012 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.186983 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-host-var-lib-kubelet\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.187012 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187014 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d5f7e8bb-6df3-4c30-84c7-4e30f116545a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l9qrh\" (UID: \"d5f7e8bb-6df3-4c30-84c7-4e30f116545a\") " pod="openshift-multus/multus-additional-cni-plugins-l9qrh" Jul 14 22:09:32.187216 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187035 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe40e4f0-af16-4743-b435-48a292cd5fc4-etc-kubernetes\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.187216 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187058 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnqhc\" (UniqueName: \"kubernetes.io/projected/fe40e4f0-af16-4743-b435-48a292cd5fc4-kube-api-access-xnqhc\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.187216 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187081 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0147c9ad-b474-4fda-8ad1-0661515acfdb-multus-daemon-config\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.187216 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187084 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-host-var-lib-kubelet\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.187216 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187122 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kqrw\" (UniqueName: \"kubernetes.io/projected/2cbd6276-0837-4e3e-b5d0-1770b73f48a2-kube-api-access-5kqrw\") pod \"aws-ebs-csi-driver-node-nsf6v\" (UID: \"2cbd6276-0837-4e3e-b5d0-1770b73f48a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsf6v" Jul 14 22:09:32.187216 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187129 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe40e4f0-af16-4743-b435-48a292cd5fc4-etc-kubernetes\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.187216 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187148 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe40e4f0-af16-4743-b435-48a292cd5fc4-sys\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.187216 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187171 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe40e4f0-af16-4743-b435-48a292cd5fc4-lib-modules\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.187216 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187191 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe40e4f0-af16-4743-b435-48a292cd5fc4-tmp\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.187216 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187214 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca583798-970a-42ec-a335-edb5a50383ad-host-slash\") pod \"iptables-alerter-kws8r\" (UID: \"ca583798-970a-42ec-a335-edb5a50383ad\") " pod="openshift-network-operator/iptables-alerter-kws8r" Jul 14 22:09:32.187721 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187237 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5f7e8bb-6df3-4c30-84c7-4e30f116545a-system-cni-dir\") pod \"multus-additional-cni-plugins-l9qrh\" (UID: \"d5f7e8bb-6df3-4c30-84c7-4e30f116545a\") " pod="openshift-multus/multus-additional-cni-plugins-l9qrh" Jul 14 22:09:32.187721 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187261 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5f7e8bb-6df3-4c30-84c7-4e30f116545a-cnibin\") pod \"multus-additional-cni-plugins-l9qrh\" (UID: \"d5f7e8bb-6df3-4c30-84c7-4e30f116545a\") " pod="openshift-multus/multus-additional-cni-plugins-l9qrh" Jul 14 22:09:32.187721 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187283 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2cbd6276-0837-4e3e-b5d0-1770b73f48a2-etc-selinux\") pod \"aws-ebs-csi-driver-node-nsf6v\" (UID: \"2cbd6276-0837-4e3e-b5d0-1770b73f48a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsf6v" Jul 14 22:09:32.187721 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187306 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxdn4\" (UniqueName: \"kubernetes.io/projected/ca583798-970a-42ec-a335-edb5a50383ad-kube-api-access-sxdn4\") pod \"iptables-alerter-kws8r\" (UID: \"ca583798-970a-42ec-a335-edb5a50383ad\") " pod="openshift-network-operator/iptables-alerter-kws8r" Jul 14 22:09:32.187721 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187331 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fe40e4f0-af16-4743-b435-48a292cd5fc4-etc-sysconfig\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.187721 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187355 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fe40e4f0-af16-4743-b435-48a292cd5fc4-etc-sysctl-d\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.187721 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187401 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5f7e8bb-6df3-4c30-84c7-4e30f116545a-system-cni-dir\") pod \"multus-additional-cni-plugins-l9qrh\" (UID: \"d5f7e8bb-6df3-4c30-84c7-4e30f116545a\") " pod="openshift-multus/multus-additional-cni-plugins-l9qrh" Jul 14 22:09:32.187721 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187436 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3a228cf3-be18-4e92-bc1b-0045d1b2c823-konnectivity-ca\") pod \"konnectivity-agent-gmmnx\" (UID: \"3a228cf3-be18-4e92-bc1b-0045d1b2c823\") " pod="kube-system/konnectivity-agent-gmmnx" Jul 14 22:09:32.187721 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187466 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fe40e4f0-af16-4743-b435-48a292cd5fc4-run\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.187721 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187480 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fe40e4f0-af16-4743-b435-48a292cd5fc4-etc-sysctl-d\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.187721 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187492 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5f7e8bb-6df3-4c30-84c7-4e30f116545a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l9qrh\" (UID: \"d5f7e8bb-6df3-4c30-84c7-4e30f116545a\") " pod="openshift-multus/multus-additional-cni-plugins-l9qrh" Jul 14 22:09:32.187721 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187517 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3a228cf3-be18-4e92-bc1b-0045d1b2c823-agent-certs\") pod \"konnectivity-agent-gmmnx\" (UID: \"3a228cf3-be18-4e92-bc1b-0045d1b2c823\") " pod="kube-system/konnectivity-agent-gmmnx" Jul 14 22:09:32.187721 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187524 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5f7e8bb-6df3-4c30-84c7-4e30f116545a-cnibin\") pod \"multus-additional-cni-plugins-l9qrh\" (UID: \"d5f7e8bb-6df3-4c30-84c7-4e30f116545a\") " pod="openshift-multus/multus-additional-cni-plugins-l9qrh" Jul 14 22:09:32.187721 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187540 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2cbd6276-0837-4e3e-b5d0-1770b73f48a2-sys-fs\") pod \"aws-ebs-csi-driver-node-nsf6v\" (UID: \"2cbd6276-0837-4e3e-b5d0-1770b73f48a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsf6v" Jul 14 22:09:32.187721 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187579 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1159189b-384f-4107-807e-250f243409df-tmp-dir\") pod \"node-resolver-6tt4w\" (UID: \"1159189b-384f-4107-807e-250f243409df\") " pod="openshift-dns/node-resolver-6tt4w" Jul 14 22:09:32.187721 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187606 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-system-cni-dir\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.187721 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187628 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-host-run-netns\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.188510 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187627 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2cbd6276-0837-4e3e-b5d0-1770b73f48a2-etc-selinux\") pod \"aws-ebs-csi-driver-node-nsf6v\" (UID: \"2cbd6276-0837-4e3e-b5d0-1770b73f48a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsf6v" Jul 14 22:09:32.188510 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187643 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0147c9ad-b474-4fda-8ad1-0661515acfdb-multus-daemon-config\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.188510 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187654 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d5f7e8bb-6df3-4c30-84c7-4e30f116545a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l9qrh\" (UID: \"d5f7e8bb-6df3-4c30-84c7-4e30f116545a\") " pod="openshift-multus/multus-additional-cni-plugins-l9qrh" Jul 14 22:09:32.188510 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187679 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca583798-970a-42ec-a335-edb5a50383ad-host-slash\") pod \"iptables-alerter-kws8r\" (UID: \"ca583798-970a-42ec-a335-edb5a50383ad\") " pod="openshift-network-operator/iptables-alerter-kws8r" Jul 14 22:09:32.188510 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187751 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe40e4f0-af16-4743-b435-48a292cd5fc4-lib-modules\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.188510 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187848 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fe40e4f0-af16-4743-b435-48a292cd5fc4-etc-sysconfig\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.188510 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187882 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-hostroot\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.188510 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187913 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-multus-conf-dir\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.188510 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187943 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-hostroot\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.188510 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187946 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1159189b-384f-4107-807e-250f243409df-hosts-file\") pod \"node-resolver-6tt4w\" (UID: \"1159189b-384f-4107-807e-250f243409df\") " pod="openshift-dns/node-resolver-6tt4w" Jul 14 22:09:32.188510 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187963 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe40e4f0-af16-4743-b435-48a292cd5fc4-sys\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.188510 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187981 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fe40e4f0-af16-4743-b435-48a292cd5fc4-etc-modprobe-d\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.188510 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.187984 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-multus-conf-dir\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.188510 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188025 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fe40e4f0-af16-4743-b435-48a292cd5fc4-etc-sysctl-conf\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.188510 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188053 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe40e4f0-af16-4743-b435-48a292cd5fc4-host\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.188510 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188059 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fe40e4f0-af16-4743-b435-48a292cd5fc4-etc-modprobe-d\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.188510 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188077 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2cbd6276-0837-4e3e-b5d0-1770b73f48a2-socket-dir\") pod \"aws-ebs-csi-driver-node-nsf6v\" (UID: \"2cbd6276-0837-4e3e-b5d0-1770b73f48a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsf6v" Jul 14 22:09:32.188510 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188104 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1159189b-384f-4107-807e-250f243409df-hosts-file\") pod \"node-resolver-6tt4w\" (UID: \"1159189b-384f-4107-807e-250f243409df\") " pod="openshift-dns/node-resolver-6tt4w" Jul 14 22:09:32.189377 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188105 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5gxh\" (UniqueName: \"kubernetes.io/projected/1159189b-384f-4107-807e-250f243409df-kube-api-access-f5gxh\") pod \"node-resolver-6tt4w\" (UID: \"1159189b-384f-4107-807e-250f243409df\") " pod="openshift-dns/node-resolver-6tt4w" Jul 14 22:09:32.189377 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188140 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-etc-kubernetes\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.189377 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188166 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krq6v\" (UniqueName: \"kubernetes.io/projected/0147c9ad-b474-4fda-8ad1-0661515acfdb-kube-api-access-krq6v\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.189377 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188180 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3a228cf3-be18-4e92-bc1b-0045d1b2c823-konnectivity-ca\") pod \"konnectivity-agent-gmmnx\" (UID: \"3a228cf3-be18-4e92-bc1b-0045d1b2c823\") " pod="kube-system/konnectivity-agent-gmmnx" Jul 14 22:09:32.189377 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188189 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5f7e8bb-6df3-4c30-84c7-4e30f116545a-cni-binary-copy\") pod \"multus-additional-cni-plugins-l9qrh\" (UID: \"d5f7e8bb-6df3-4c30-84c7-4e30f116545a\") " pod="openshift-multus/multus-additional-cni-plugins-l9qrh" Jul 14 22:09:32.189377 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188216 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d5f7e8bb-6df3-4c30-84c7-4e30f116545a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l9qrh\" (UID: \"d5f7e8bb-6df3-4c30-84c7-4e30f116545a\") " pod="openshift-multus/multus-additional-cni-plugins-l9qrh" Jul 14 22:09:32.189377 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188242 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fe40e4f0-af16-4743-b435-48a292cd5fc4-etc-systemd\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.189377 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188264 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5f7e8bb-6df3-4c30-84c7-4e30f116545a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l9qrh\" (UID: \"d5f7e8bb-6df3-4c30-84c7-4e30f116545a\") " pod="openshift-multus/multus-additional-cni-plugins-l9qrh" Jul 14 22:09:32.189377 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188265 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d5f7e8bb-6df3-4c30-84c7-4e30f116545a-os-release\") pod \"multus-additional-cni-plugins-l9qrh\" (UID: \"d5f7e8bb-6df3-4c30-84c7-4e30f116545a\") " pod="openshift-multus/multus-additional-cni-plugins-l9qrh" Jul 14 22:09:32.189377 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188304 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2cbd6276-0837-4e3e-b5d0-1770b73f48a2-registration-dir\") pod \"aws-ebs-csi-driver-node-nsf6v\" (UID: \"2cbd6276-0837-4e3e-b5d0-1770b73f48a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsf6v" Jul 14 22:09:32.189377 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188321 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d5f7e8bb-6df3-4c30-84c7-4e30f116545a-os-release\") pod \"multus-additional-cni-plugins-l9qrh\" (UID: \"d5f7e8bb-6df3-4c30-84c7-4e30f116545a\") " pod="openshift-multus/multus-additional-cni-plugins-l9qrh" Jul 14 22:09:32.189377 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188111 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2cbd6276-0837-4e3e-b5d0-1770b73f48a2-sys-fs\") pod \"aws-ebs-csi-driver-node-nsf6v\" (UID: \"2cbd6276-0837-4e3e-b5d0-1770b73f48a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsf6v" Jul 14 22:09:32.189377 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188345 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-cnibin\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.189377 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188370 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-host-run-k8s-cni-cncf-io\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.189377 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188373 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-etc-kubernetes\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.189377 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188393 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-host-var-lib-cni-bin\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.189377 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188417 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-host-var-lib-cni-multus\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.190170 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188441 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1159189b-384f-4107-807e-250f243409df-tmp-dir\") pod \"node-resolver-6tt4w\" (UID: \"1159189b-384f-4107-807e-250f243409df\") " pod="openshift-dns/node-resolver-6tt4w" Jul 14 22:09:32.190170 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188451 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-multus-cni-dir\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.190170 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188479 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-os-release\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.190170 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188498 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-host-run-netns\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.190170 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188503 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-multus-socket-dir-parent\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.190170 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188528 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ca583798-970a-42ec-a335-edb5a50383ad-iptables-alerter-script\") pod \"iptables-alerter-kws8r\" (UID: \"ca583798-970a-42ec-a335-edb5a50383ad\") " pod="openshift-network-operator/iptables-alerter-kws8r" Jul 14 22:09:32.190170 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188537 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fe40e4f0-af16-4743-b435-48a292cd5fc4-etc-sysctl-conf\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.190170 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188571 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2cbd6276-0837-4e3e-b5d0-1770b73f48a2-registration-dir\") pod \"aws-ebs-csi-driver-node-nsf6v\" (UID: \"2cbd6276-0837-4e3e-b5d0-1770b73f48a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsf6v" Jul 14 22:09:32.190170 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188618 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-host-var-lib-cni-multus\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.190170 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188625 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe40e4f0-af16-4743-b435-48a292cd5fc4-host\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.190170 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188662 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fe40e4f0-af16-4743-b435-48a292cd5fc4-etc-systemd\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.190170 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188682 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-os-release\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.190170 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188628 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-system-cni-dir\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.190170 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188699 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-host-run-k8s-cni-cncf-io\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.190170 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188740 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fe40e4f0-af16-4743-b435-48a292cd5fc4-run\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.190170 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188739 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-multus-socket-dir-parent\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.190170 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188619 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-cnibin\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.190170 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188761 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrtk5\" (UniqueName: \"kubernetes.io/projected/a549c30a-9937-422b-9f9b-17d10c48ab25-kube-api-access-wrtk5\") pod \"network-check-target-6p45d\" (UID: \"a549c30a-9937-422b-9f9b-17d10c48ab25\") " pod="openshift-network-diagnostics/network-check-target-6p45d" Jul 14 22:09:32.191042 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188787 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0147c9ad-b474-4fda-8ad1-0661515acfdb-cni-binary-copy\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.191042 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188803 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-multus-cni-dir\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.191042 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188831 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-host-var-lib-cni-bin\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.191042 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.188936 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2cbd6276-0837-4e3e-b5d0-1770b73f48a2-socket-dir\") pod \"aws-ebs-csi-driver-node-nsf6v\" (UID: \"2cbd6276-0837-4e3e-b5d0-1770b73f48a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsf6v" Jul 14 22:09:32.191042 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.189065 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d5f7e8bb-6df3-4c30-84c7-4e30f116545a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l9qrh\" (UID: \"d5f7e8bb-6df3-4c30-84c7-4e30f116545a\") " pod="openshift-multus/multus-additional-cni-plugins-l9qrh" Jul 14 22:09:32.191042 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.189127 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fe40e4f0-af16-4743-b435-48a292cd5fc4-etc-tuned\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.191042 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.189155 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe40e4f0-af16-4743-b435-48a292cd5fc4-var-lib-kubelet\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.191042 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.189180 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-host-run-multus-certs\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.191042 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.189202 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2cbd6276-0837-4e3e-b5d0-1770b73f48a2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nsf6v\" (UID: \"2cbd6276-0837-4e3e-b5d0-1770b73f48a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsf6v" Jul 14 22:09:32.191042 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.189230 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2cbd6276-0837-4e3e-b5d0-1770b73f48a2-device-dir\") pod \"aws-ebs-csi-driver-node-nsf6v\" (UID: \"2cbd6276-0837-4e3e-b5d0-1770b73f48a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsf6v" Jul 14 22:09:32.191042 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.189258 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv6ds\" (UniqueName: \"kubernetes.io/projected/d5f7e8bb-6df3-4c30-84c7-4e30f116545a-kube-api-access-nv6ds\") pod \"multus-additional-cni-plugins-l9qrh\" (UID: \"d5f7e8bb-6df3-4c30-84c7-4e30f116545a\") " pod="openshift-multus/multus-additional-cni-plugins-l9qrh" Jul 14 22:09:32.191042 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.189262 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0147c9ad-b474-4fda-8ad1-0661515acfdb-cni-binary-copy\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.191042 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.189322 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5f7e8bb-6df3-4c30-84c7-4e30f116545a-cni-binary-copy\") pod \"multus-additional-cni-plugins-l9qrh\" (UID: \"d5f7e8bb-6df3-4c30-84c7-4e30f116545a\") " pod="openshift-multus/multus-additional-cni-plugins-l9qrh" Jul 14 22:09:32.191042 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.189330 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0147c9ad-b474-4fda-8ad1-0661515acfdb-host-run-multus-certs\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.191042 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.189391 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe40e4f0-af16-4743-b435-48a292cd5fc4-var-lib-kubelet\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.191042 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.189442 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2cbd6276-0837-4e3e-b5d0-1770b73f48a2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nsf6v\" (UID: \"2cbd6276-0837-4e3e-b5d0-1770b73f48a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsf6v" Jul 14 22:09:32.191042 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.189484 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2cbd6276-0837-4e3e-b5d0-1770b73f48a2-device-dir\") pod \"aws-ebs-csi-driver-node-nsf6v\" (UID: \"2cbd6276-0837-4e3e-b5d0-1770b73f48a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsf6v" Jul 14 22:09:32.191891 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.189567 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ca583798-970a-42ec-a335-edb5a50383ad-iptables-alerter-script\") pod \"iptables-alerter-kws8r\" (UID: \"ca583798-970a-42ec-a335-edb5a50383ad\") " pod="openshift-network-operator/iptables-alerter-kws8r" Jul 14 22:09:32.191891 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.190746 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe40e4f0-af16-4743-b435-48a292cd5fc4-tmp\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.191891 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.191786 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fe40e4f0-af16-4743-b435-48a292cd5fc4-etc-tuned\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.192042 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.191908 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3a228cf3-be18-4e92-bc1b-0045d1b2c823-agent-certs\") pod \"konnectivity-agent-gmmnx\" (UID: \"3a228cf3-be18-4e92-bc1b-0045d1b2c823\") " pod="kube-system/konnectivity-agent-gmmnx" Jul 14 22:09:32.196132 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:32.196022 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 22:09:32.196132 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:32.196043 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 22:09:32.196132 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:32.196056 2550 projected.go:196] Error preparing data for projected volume kube-api-access-wrtk5 for pod openshift-network-diagnostics/network-check-target-6p45d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 22:09:32.196132 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:32.196123 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a549c30a-9937-422b-9f9b-17d10c48ab25-kube-api-access-wrtk5 podName:a549c30a-9937-422b-9f9b-17d10c48ab25 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:32.696104366 +0000 UTC m=+3.055250801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wrtk5" (UniqueName: "kubernetes.io/projected/a549c30a-9937-422b-9f9b-17d10c48ab25-kube-api-access-wrtk5") pod "network-check-target-6p45d" (UID: "a549c30a-9937-422b-9f9b-17d10c48ab25") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 22:09:32.197829 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.197772 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal" event={"ID":"70253776f2fd132651fc1b4e46ac7c7d","Type":"ContainerStarted","Data":"7650875d57173e4a2a3684f5fdbdfa2f85190e6ab613230a5e64d603258894ff"} Jul 14 22:09:32.199753 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.199364 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kqrw\" (UniqueName: \"kubernetes.io/projected/2cbd6276-0837-4e3e-b5d0-1770b73f48a2-kube-api-access-5kqrw\") pod \"aws-ebs-csi-driver-node-nsf6v\" (UID: \"2cbd6276-0837-4e3e-b5d0-1770b73f48a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsf6v" Jul 14 22:09:32.199753 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.199489 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal" event={"ID":"da6abbb4b54c9a3e9a35bc6fff12e4c3","Type":"ContainerStarted","Data":"769a7b973ec1444f2f7483cd1c53f51ef9f994edf511cd1b07e15564fb84c801"} Jul 14 22:09:32.199913 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.199822 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnqhc\" (UniqueName: \"kubernetes.io/projected/fe40e4f0-af16-4743-b435-48a292cd5fc4-kube-api-access-xnqhc\") pod \"tuned-jdnlk\" (UID: \"fe40e4f0-af16-4743-b435-48a292cd5fc4\") " pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.200189 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.200168 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krq6v\" (UniqueName: \"kubernetes.io/projected/0147c9ad-b474-4fda-8ad1-0661515acfdb-kube-api-access-krq6v\") pod \"multus-wtrck\" (UID: \"0147c9ad-b474-4fda-8ad1-0661515acfdb\") " pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.200392 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.200367 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxdn4\" (UniqueName: \"kubernetes.io/projected/ca583798-970a-42ec-a335-edb5a50383ad-kube-api-access-sxdn4\") pod \"iptables-alerter-kws8r\" (UID: \"ca583798-970a-42ec-a335-edb5a50383ad\") " pod="openshift-network-operator/iptables-alerter-kws8r" Jul 14 22:09:32.201007 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.200990 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv6ds\" (UniqueName: \"kubernetes.io/projected/d5f7e8bb-6df3-4c30-84c7-4e30f116545a-kube-api-access-nv6ds\") pod \"multus-additional-cni-plugins-l9qrh\" (UID: \"d5f7e8bb-6df3-4c30-84c7-4e30f116545a\") " pod="openshift-multus/multus-additional-cni-plugins-l9qrh" Jul 14 22:09:32.201263 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.201248 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5gxh\" (UniqueName: \"kubernetes.io/projected/1159189b-384f-4107-807e-250f243409df-kube-api-access-f5gxh\") pod \"node-resolver-6tt4w\" (UID: \"1159189b-384f-4107-807e-250f243409df\") " pod="openshift-dns/node-resolver-6tt4w" Jul 14 22:09:32.278668 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.278584 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dj68k" Jul 14 22:09:32.286485 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.286463 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:32.296501 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.296479 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsf6v" Jul 14 22:09:32.305259 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.305240 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6tt4w" Jul 14 22:09:32.311965 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.311945 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" Jul 14 22:09:32.319742 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.319722 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wtrck" Jul 14 22:09:32.328384 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.328366 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kws8r" Jul 14 22:09:32.335108 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.335076 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l9qrh" Jul 14 22:09:32.340931 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.340912 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gmmnx" Jul 14 22:09:32.591641 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.591542 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f9358e0-96a8-473f-896b-907e7129d452-metrics-certs\") pod \"network-metrics-daemon-55gw6\" (UID: \"7f9358e0-96a8-473f-896b-907e7129d452\") " pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:09:32.591805 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.591645 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce5ffbb6-99ac-4817-acd9-f6cd4577f724-original-pull-secret\") pod \"global-pull-secret-syncer-6g6bp\" (UID: \"ce5ffbb6-99ac-4817-acd9-f6cd4577f724\") " pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:09:32.591805 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:32.591678 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 22:09:32.591805 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:32.591753 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f9358e0-96a8-473f-896b-907e7129d452-metrics-certs podName:7f9358e0-96a8-473f-896b-907e7129d452 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:33.591736271 +0000 UTC m=+3.950882707 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f9358e0-96a8-473f-896b-907e7129d452-metrics-certs") pod "network-metrics-daemon-55gw6" (UID: "7f9358e0-96a8-473f-896b-907e7129d452") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 22:09:32.591805 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:32.591754 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 22:09:32.591805 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:32.591808 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce5ffbb6-99ac-4817-acd9-f6cd4577f724-original-pull-secret podName:ce5ffbb6-99ac-4817-acd9-f6cd4577f724 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:33.591790296 +0000 UTC m=+3.950936715 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce5ffbb6-99ac-4817-acd9-f6cd4577f724-original-pull-secret") pod "global-pull-secret-syncer-6g6bp" (UID: "ce5ffbb6-99ac-4817-acd9-f6cd4577f724") : object "kube-system"/"original-pull-secret" not registered Jul 14 22:09:32.793222 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:32.793188 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrtk5\" (UniqueName: \"kubernetes.io/projected/a549c30a-9937-422b-9f9b-17d10c48ab25-kube-api-access-wrtk5\") pod \"network-check-target-6p45d\" (UID: \"a549c30a-9937-422b-9f9b-17d10c48ab25\") " pod="openshift-network-diagnostics/network-check-target-6p45d" Jul 14 22:09:32.793387 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:32.793344 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 22:09:32.793387 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:32.793366 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 22:09:32.793387 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:32.793377 2550 projected.go:196] Error preparing data for projected volume kube-api-access-wrtk5 for pod openshift-network-diagnostics/network-check-target-6p45d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 22:09:32.793510 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:32.793427 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a549c30a-9937-422b-9f9b-17d10c48ab25-kube-api-access-wrtk5 podName:a549c30a-9937-422b-9f9b-17d10c48ab25 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:33.793413333 +0000 UTC m=+4.152559747 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wrtk5" (UniqueName: "kubernetes.io/projected/a549c30a-9937-422b-9f9b-17d10c48ab25-kube-api-access-wrtk5") pod "network-check-target-6p45d" (UID: "a549c30a-9937-422b-9f9b-17d10c48ab25") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 22:09:33.021093 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:33.020995 2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 22:04:32 +0000 UTC" deadline="2028-02-18 18:57:08.184672478 +0000 UTC" Jul 14 22:09:33.021093 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:33.021051 2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14012h47m35.163623934s" Jul 14 22:09:33.050090 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:33.050066 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0884baa5_9206_4196_a7c9_b5686623311b.slice/crio-13067e28fb14200e2f1a005516f2dddff0ed5a3d908a05b53e67115001650253 WatchSource:0}: Error finding container 13067e28fb14200e2f1a005516f2dddff0ed5a3d908a05b53e67115001650253: Status 404 returned error can't find the container with id 13067e28fb14200e2f1a005516f2dddff0ed5a3d908a05b53e67115001650253 Jul 14 22:09:33.055716 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:33.055680 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a228cf3_be18_4e92_bc1b_0045d1b2c823.slice/crio-cc371e0682c8112d1d264ef080b87b3d852be4f908c515d4e23068b95d5c818c WatchSource:0}: Error finding container cc371e0682c8112d1d264ef080b87b3d852be4f908c515d4e23068b95d5c818c: Status 404 returned error can't find the container with id cc371e0682c8112d1d264ef080b87b3d852be4f908c515d4e23068b95d5c818c Jul 14 22:09:33.056269 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:33.056248 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe40e4f0_af16_4743_b435_48a292cd5fc4.slice/crio-632043765c4e3ab1385c025c8dd792dedef598352333bd7ecf5fc1a6485ef749 WatchSource:0}: Error finding container 632043765c4e3ab1385c025c8dd792dedef598352333bd7ecf5fc1a6485ef749: Status 404 returned error can't find the container with id 632043765c4e3ab1385c025c8dd792dedef598352333bd7ecf5fc1a6485ef749 Jul 14 22:09:33.074802 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:33.074780 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cbd6276_0837_4e3e_b5d0_1770b73f48a2.slice/crio-6fca6c40d8fcdbfc1c0813e46f977699bfcf225b3672aec6dad982474731ac26 WatchSource:0}: Error finding container 6fca6c40d8fcdbfc1c0813e46f977699bfcf225b3672aec6dad982474731ac26: Status 404 returned error can't find the container with id 6fca6c40d8fcdbfc1c0813e46f977699bfcf225b3672aec6dad982474731ac26 Jul 14 22:09:33.075297 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:33.075271 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1159189b_384f_4107_807e_250f243409df.slice/crio-b33bcd0aa566582fcf7ced90c9979062ad15fb4beecd46e94ca34757144bd1a2 WatchSource:0}: Error finding container b33bcd0aa566582fcf7ced90c9979062ad15fb4beecd46e94ca34757144bd1a2: Status 404 returned error can't find the container with id b33bcd0aa566582fcf7ced90c9979062ad15fb4beecd46e94ca34757144bd1a2 Jul 14 22:09:33.076995 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:33.076906 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c265d22_e8f9_4516_aaf7_b274420c1b23.slice/crio-20af33801db6d655575bc6fa3899a8bd9491e8582b68ffe91e819f0e8124c127 WatchSource:0}: Error finding container 20af33801db6d655575bc6fa3899a8bd9491e8582b68ffe91e819f0e8124c127: Status 404 returned error can't find the container with id 20af33801db6d655575bc6fa3899a8bd9491e8582b68ffe91e819f0e8124c127 Jul 14 22:09:33.077229 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:33.077189 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca583798_970a_42ec_a335_edb5a50383ad.slice/crio-e7aeaa8e92040e64900ea13f72f9504a065375ff82c72bdd7a1fe8aba7789294 WatchSource:0}: Error finding container e7aeaa8e92040e64900ea13f72f9504a065375ff82c72bdd7a1fe8aba7789294: Status 404 returned error can't find the container with id e7aeaa8e92040e64900ea13f72f9504a065375ff82c72bdd7a1fe8aba7789294 Jul 14 22:09:33.080735 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:33.080714 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5f7e8bb_6df3_4c30_84c7_4e30f116545a.slice/crio-35ad7902336907ebdd5292aa52db7d5be9eb9bf27ac5a2245eb3cff11feca3b1 WatchSource:0}: Error finding container 35ad7902336907ebdd5292aa52db7d5be9eb9bf27ac5a2245eb3cff11feca3b1: Status 404 returned error can't find the container with id 35ad7902336907ebdd5292aa52db7d5be9eb9bf27ac5a2245eb3cff11feca3b1 Jul 14 22:09:33.081145 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:09:33.081118 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0147c9ad_b474_4fda_8ad1_0661515acfdb.slice/crio-1c71e1f7e1dfe4c121c4e3fe35a7e2c5cbc27b2218b435885ff18599aec2a1f1 WatchSource:0}: Error finding container 1c71e1f7e1dfe4c121c4e3fe35a7e2c5cbc27b2218b435885ff18599aec2a1f1: Status 404 returned error can't find the container with id 1c71e1f7e1dfe4c121c4e3fe35a7e2c5cbc27b2218b435885ff18599aec2a1f1 Jul 14 22:09:33.190028 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:33.189875 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6p45d" Jul 14 22:09:33.190145 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:33.189907 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:09:33.190206 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:33.190181 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6p45d" podUID="a549c30a-9937-422b-9f9b-17d10c48ab25" Jul 14 22:09:33.190332 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:33.190310 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55gw6" podUID="7f9358e0-96a8-473f-896b-907e7129d452" Jul 14 22:09:33.202427 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:33.202404 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wtrck" event={"ID":"0147c9ad-b474-4fda-8ad1-0661515acfdb","Type":"ContainerStarted","Data":"1c71e1f7e1dfe4c121c4e3fe35a7e2c5cbc27b2218b435885ff18599aec2a1f1"} Jul 14 22:09:33.204904 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:33.204873 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kws8r" event={"ID":"ca583798-970a-42ec-a335-edb5a50383ad","Type":"ContainerStarted","Data":"e7aeaa8e92040e64900ea13f72f9504a065375ff82c72bdd7a1fe8aba7789294"} Jul 14 22:09:33.206452 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:33.206433 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dj68k" event={"ID":"8c265d22-e8f9-4516-aaf7-b274420c1b23","Type":"ContainerStarted","Data":"20af33801db6d655575bc6fa3899a8bd9491e8582b68ffe91e819f0e8124c127"} Jul 14 22:09:33.207908 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:33.207887 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6tt4w" event={"ID":"1159189b-384f-4107-807e-250f243409df","Type":"ContainerStarted","Data":"b33bcd0aa566582fcf7ced90c9979062ad15fb4beecd46e94ca34757144bd1a2"} Jul 14 22:09:33.208818 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:33.208794 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsf6v" event={"ID":"2cbd6276-0837-4e3e-b5d0-1770b73f48a2","Type":"ContainerStarted","Data":"6fca6c40d8fcdbfc1c0813e46f977699bfcf225b3672aec6dad982474731ac26"} Jul 14 22:09:33.210064 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:33.210045 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gmmnx" event={"ID":"3a228cf3-be18-4e92-bc1b-0045d1b2c823","Type":"ContainerStarted","Data":"cc371e0682c8112d1d264ef080b87b3d852be4f908c515d4e23068b95d5c818c"} Jul 14 22:09:33.211071 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:33.211040 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" event={"ID":"fe40e4f0-af16-4743-b435-48a292cd5fc4","Type":"ContainerStarted","Data":"632043765c4e3ab1385c025c8dd792dedef598352333bd7ecf5fc1a6485ef749"} Jul 14 22:09:33.212837 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:33.212815 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal" event={"ID":"da6abbb4b54c9a3e9a35bc6fff12e4c3","Type":"ContainerStarted","Data":"429f700c36663ae43331ed5bbd128378035d0934349bb15eeb1d3d4488c574e6"} Jul 14 22:09:33.213909 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:33.213891 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l9qrh" event={"ID":"d5f7e8bb-6df3-4c30-84c7-4e30f116545a","Type":"ContainerStarted","Data":"35ad7902336907ebdd5292aa52db7d5be9eb9bf27ac5a2245eb3cff11feca3b1"} Jul 14 22:09:33.215874 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:33.215852 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" event={"ID":"0884baa5-9206-4196-a7c9-b5686623311b","Type":"ContainerStarted","Data":"13067e28fb14200e2f1a005516f2dddff0ed5a3d908a05b53e67115001650253"} Jul 14 22:09:33.228397 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:33.228360 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-118.ec2.internal" podStartSLOduration=2.2283489 podStartE2EDuration="2.2283489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:09:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:09:33.228087795 +0000 UTC m=+3.587234243" watchObservedRunningTime="2026-07-14 22:09:33.2283489 +0000 UTC m=+3.587495336" Jul 14 22:09:33.600079 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:33.599986 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f9358e0-96a8-473f-896b-907e7129d452-metrics-certs\") pod \"network-metrics-daemon-55gw6\" (UID: \"7f9358e0-96a8-473f-896b-907e7129d452\") " pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:09:33.600234 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:33.600080 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce5ffbb6-99ac-4817-acd9-f6cd4577f724-original-pull-secret\") pod \"global-pull-secret-syncer-6g6bp\" (UID: \"ce5ffbb6-99ac-4817-acd9-f6cd4577f724\") " pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:09:33.600234 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:33.600210 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 22:09:33.600335 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:33.600272 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce5ffbb6-99ac-4817-acd9-f6cd4577f724-original-pull-secret podName:ce5ffbb6-99ac-4817-acd9-f6cd4577f724 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:35.600253181 +0000 UTC m=+5.959399598 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce5ffbb6-99ac-4817-acd9-f6cd4577f724-original-pull-secret") pod "global-pull-secret-syncer-6g6bp" (UID: "ce5ffbb6-99ac-4817-acd9-f6cd4577f724") : object "kube-system"/"original-pull-secret" not registered Jul 14 22:09:33.600761 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:33.600677 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 22:09:33.600761 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:33.600730 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f9358e0-96a8-473f-896b-907e7129d452-metrics-certs podName:7f9358e0-96a8-473f-896b-907e7129d452 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:35.600714425 +0000 UTC m=+5.959860841 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f9358e0-96a8-473f-896b-907e7129d452-metrics-certs") pod "network-metrics-daemon-55gw6" (UID: "7f9358e0-96a8-473f-896b-907e7129d452") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 22:09:33.801742 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:33.801704 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrtk5\" (UniqueName: \"kubernetes.io/projected/a549c30a-9937-422b-9f9b-17d10c48ab25-kube-api-access-wrtk5\") pod \"network-check-target-6p45d\" (UID: \"a549c30a-9937-422b-9f9b-17d10c48ab25\") " pod="openshift-network-diagnostics/network-check-target-6p45d" Jul 14 22:09:33.801933 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:33.801865 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 22:09:33.801933 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:33.801885 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 22:09:33.801933 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:33.801898 2550 projected.go:196] Error preparing data for projected volume kube-api-access-wrtk5 for pod openshift-network-diagnostics/network-check-target-6p45d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 22:09:33.802093 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:33.801955 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a549c30a-9937-422b-9f9b-17d10c48ab25-kube-api-access-wrtk5 podName:a549c30a-9937-422b-9f9b-17d10c48ab25 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:35.801936594 +0000 UTC m=+6.161083011 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wrtk5" (UniqueName: "kubernetes.io/projected/a549c30a-9937-422b-9f9b-17d10c48ab25-kube-api-access-wrtk5") pod "network-check-target-6p45d" (UID: "a549c30a-9937-422b-9f9b-17d10c48ab25") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 22:09:34.021971 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:34.021852 2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 22:04:32 +0000 UTC" deadline="2028-01-21 01:23:28.401554719 +0000 UTC" Jul 14 22:09:34.021971 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:34.021889 2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13323h13m54.379670687s" Jul 14 22:09:34.189781 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:34.189729 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:09:34.189941 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:34.189877 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6bp" podUID="ce5ffbb6-99ac-4817-acd9-f6cd4577f724" Jul 14 22:09:34.225578 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:34.225525 2550 generic.go:356] "Generic (PLEG): container finished" podID="70253776f2fd132651fc1b4e46ac7c7d" containerID="730d914a6594259454e77baa476b86c056dcc038ee909238d4cd85db22cd29b2" exitCode=0 Jul 14 22:09:34.226759 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:34.226241 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal" event={"ID":"70253776f2fd132651fc1b4e46ac7c7d","Type":"ContainerDied","Data":"730d914a6594259454e77baa476b86c056dcc038ee909238d4cd85db22cd29b2"} Jul 14 22:09:35.189372 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:35.189339 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6p45d" Jul 14 22:09:35.189854 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:35.189472 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6p45d" podUID="a549c30a-9937-422b-9f9b-17d10c48ab25" Jul 14 22:09:35.189929 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:35.189890 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:09:35.190060 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:35.189992 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55gw6" podUID="7f9358e0-96a8-473f-896b-907e7129d452" Jul 14 22:09:35.257681 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:35.257646 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal" event={"ID":"70253776f2fd132651fc1b4e46ac7c7d","Type":"ContainerStarted","Data":"437d24d1f239218057bda7b670d31d2c00c19a020bb42c0d6c15eea2dfc23d56"} Jul 14 22:09:35.276227 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:35.275144 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-118.ec2.internal" podStartSLOduration=4.275125505 podStartE2EDuration="4.275125505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:09:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:09:35.274912975 +0000 UTC m=+5.634059414" watchObservedRunningTime="2026-07-14 22:09:35.275125505 +0000 UTC m=+5.634271943" Jul 14 22:09:35.618574 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:35.618483 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f9358e0-96a8-473f-896b-907e7129d452-metrics-certs\") pod \"network-metrics-daemon-55gw6\" (UID: \"7f9358e0-96a8-473f-896b-907e7129d452\") " pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:09:35.618737 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:35.618589 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce5ffbb6-99ac-4817-acd9-f6cd4577f724-original-pull-secret\") pod \"global-pull-secret-syncer-6g6bp\" (UID: \"ce5ffbb6-99ac-4817-acd9-f6cd4577f724\") " pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:09:35.618737 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:35.618658 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 22:09:35.618737 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:35.618728 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f9358e0-96a8-473f-896b-907e7129d452-metrics-certs podName:7f9358e0-96a8-473f-896b-907e7129d452 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:39.61870724 +0000 UTC m=+9.977853669 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f9358e0-96a8-473f-896b-907e7129d452-metrics-certs") pod "network-metrics-daemon-55gw6" (UID: "7f9358e0-96a8-473f-896b-907e7129d452") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 22:09:35.618737 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:35.618733 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 22:09:35.618938 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:35.618785 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce5ffbb6-99ac-4817-acd9-f6cd4577f724-original-pull-secret podName:ce5ffbb6-99ac-4817-acd9-f6cd4577f724 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:39.618769221 +0000 UTC m=+9.977915638 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce5ffbb6-99ac-4817-acd9-f6cd4577f724-original-pull-secret") pod "global-pull-secret-syncer-6g6bp" (UID: "ce5ffbb6-99ac-4817-acd9-f6cd4577f724") : object "kube-system"/"original-pull-secret" not registered Jul 14 22:09:35.820904 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:35.820845 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrtk5\" (UniqueName: \"kubernetes.io/projected/a549c30a-9937-422b-9f9b-17d10c48ab25-kube-api-access-wrtk5\") pod \"network-check-target-6p45d\" (UID: \"a549c30a-9937-422b-9f9b-17d10c48ab25\") " pod="openshift-network-diagnostics/network-check-target-6p45d" Jul 14 22:09:35.821063 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:35.821032 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 22:09:35.821063 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:35.821054 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 22:09:35.821169 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:35.821066 2550 projected.go:196] Error preparing data for projected volume kube-api-access-wrtk5 for pod openshift-network-diagnostics/network-check-target-6p45d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 22:09:35.821169 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:35.821126 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a549c30a-9937-422b-9f9b-17d10c48ab25-kube-api-access-wrtk5 podName:a549c30a-9937-422b-9f9b-17d10c48ab25 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:39.821107382 +0000 UTC m=+10.180253801 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wrtk5" (UniqueName: "kubernetes.io/projected/a549c30a-9937-422b-9f9b-17d10c48ab25-kube-api-access-wrtk5") pod "network-check-target-6p45d" (UID: "a549c30a-9937-422b-9f9b-17d10c48ab25") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 22:09:36.190983 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:36.190506 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:09:36.190983 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:36.190660 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6bp" podUID="ce5ffbb6-99ac-4817-acd9-f6cd4577f724" Jul 14 22:09:37.189982 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:37.189949 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6p45d" Jul 14 22:09:37.190171 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:37.190087 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6p45d" podUID="a549c30a-9937-422b-9f9b-17d10c48ab25" Jul 14 22:09:37.190431 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:37.189957 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:09:37.190567 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:37.190529 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55gw6" podUID="7f9358e0-96a8-473f-896b-907e7129d452" Jul 14 22:09:38.189634 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:38.189597 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:09:38.190068 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:38.189744 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6bp" podUID="ce5ffbb6-99ac-4817-acd9-f6cd4577f724" Jul 14 22:09:39.189980 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:39.189460 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6p45d" Jul 14 22:09:39.189980 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:39.189515 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:09:39.189980 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:39.189615 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6p45d" podUID="a549c30a-9937-422b-9f9b-17d10c48ab25" Jul 14 22:09:39.189980 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:39.189761 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55gw6" podUID="7f9358e0-96a8-473f-896b-907e7129d452" Jul 14 22:09:39.652094 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:39.651227 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce5ffbb6-99ac-4817-acd9-f6cd4577f724-original-pull-secret\") pod \"global-pull-secret-syncer-6g6bp\" (UID: \"ce5ffbb6-99ac-4817-acd9-f6cd4577f724\") " pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:09:39.652094 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:39.651298 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f9358e0-96a8-473f-896b-907e7129d452-metrics-certs\") pod \"network-metrics-daemon-55gw6\" (UID: \"7f9358e0-96a8-473f-896b-907e7129d452\") " pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:09:39.652094 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:39.651451 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 22:09:39.652094 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:39.651520 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f9358e0-96a8-473f-896b-907e7129d452-metrics-certs podName:7f9358e0-96a8-473f-896b-907e7129d452 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:47.651497334 +0000 UTC m=+18.010643771 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f9358e0-96a8-473f-896b-907e7129d452-metrics-certs") pod "network-metrics-daemon-55gw6" (UID: "7f9358e0-96a8-473f-896b-907e7129d452") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 22:09:39.652094 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:39.651949 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 22:09:39.652094 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:39.652002 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce5ffbb6-99ac-4817-acd9-f6cd4577f724-original-pull-secret podName:ce5ffbb6-99ac-4817-acd9-f6cd4577f724 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:47.651986492 +0000 UTC m=+18.011132910 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce5ffbb6-99ac-4817-acd9-f6cd4577f724-original-pull-secret") pod "global-pull-secret-syncer-6g6bp" (UID: "ce5ffbb6-99ac-4817-acd9-f6cd4577f724") : object "kube-system"/"original-pull-secret" not registered Jul 14 22:09:39.853118 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:39.853082 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrtk5\" (UniqueName: \"kubernetes.io/projected/a549c30a-9937-422b-9f9b-17d10c48ab25-kube-api-access-wrtk5\") pod \"network-check-target-6p45d\" (UID: \"a549c30a-9937-422b-9f9b-17d10c48ab25\") " pod="openshift-network-diagnostics/network-check-target-6p45d" Jul 14 22:09:39.853313 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:39.853237 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 22:09:39.853313 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:39.853255 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 22:09:39.853313 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:39.853268 2550 projected.go:196] Error preparing data for projected volume kube-api-access-wrtk5 for pod openshift-network-diagnostics/network-check-target-6p45d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 22:09:39.853470 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:39.853339 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a549c30a-9937-422b-9f9b-17d10c48ab25-kube-api-access-wrtk5 podName:a549c30a-9937-422b-9f9b-17d10c48ab25 nodeName:}" failed. No retries permitted until 2026-07-14 22:09:47.853321181 +0000 UTC m=+18.212467611 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wrtk5" (UniqueName: "kubernetes.io/projected/a549c30a-9937-422b-9f9b-17d10c48ab25-kube-api-access-wrtk5") pod "network-check-target-6p45d" (UID: "a549c30a-9937-422b-9f9b-17d10c48ab25") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 22:09:40.190506 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:40.190365 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:09:40.190506 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:40.190487 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6bp" podUID="ce5ffbb6-99ac-4817-acd9-f6cd4577f724" Jul 14 22:09:41.190025 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:41.189644 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:09:41.190025 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:41.189787 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55gw6" podUID="7f9358e0-96a8-473f-896b-907e7129d452" Jul 14 22:09:41.190256 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:41.190064 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6p45d" Jul 14 22:09:41.190256 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:41.190160 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6p45d" podUID="a549c30a-9937-422b-9f9b-17d10c48ab25" Jul 14 22:09:42.190301 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:42.190271 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:09:42.190758 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:42.190422 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6bp" podUID="ce5ffbb6-99ac-4817-acd9-f6cd4577f724" Jul 14 22:09:43.189844 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:43.189791 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6p45d" Jul 14 22:09:43.189844 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:43.189844 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:09:43.190067 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:43.189929 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6p45d" podUID="a549c30a-9937-422b-9f9b-17d10c48ab25" Jul 14 22:09:43.190125 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:43.190070 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55gw6" podUID="7f9358e0-96a8-473f-896b-907e7129d452" Jul 14 22:09:44.189165 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:44.189130 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:09:44.189641 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:44.189261 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6bp" podUID="ce5ffbb6-99ac-4817-acd9-f6cd4577f724" Jul 14 22:09:45.189562 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:45.189506 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6p45d" Jul 14 22:09:45.190024 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:45.189576 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:09:45.190024 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:45.189660 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6p45d" podUID="a549c30a-9937-422b-9f9b-17d10c48ab25" Jul 14 22:09:45.190024 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:45.189813 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55gw6" podUID="7f9358e0-96a8-473f-896b-907e7129d452" Jul 14 22:09:46.190013 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:46.189981 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:09:46.190462 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:46.190111 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6bp" podUID="ce5ffbb6-99ac-4817-acd9-f6cd4577f724" Jul 14 22:09:47.189247 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:47.189215 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6p45d" Jul 14 22:09:47.189480 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:47.189348 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6p45d" podUID="a549c30a-9937-422b-9f9b-17d10c48ab25" Jul 14 22:09:47.189480 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:47.189387 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:09:47.189480 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:47.189474 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55gw6" podUID="7f9358e0-96a8-473f-896b-907e7129d452" Jul 14 22:09:47.717939 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:47.717904 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce5ffbb6-99ac-4817-acd9-f6cd4577f724-original-pull-secret\") pod \"global-pull-secret-syncer-6g6bp\" (UID: \"ce5ffbb6-99ac-4817-acd9-f6cd4577f724\") " pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:09:47.718378 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:47.717952 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f9358e0-96a8-473f-896b-907e7129d452-metrics-certs\") pod \"network-metrics-daemon-55gw6\" (UID: \"7f9358e0-96a8-473f-896b-907e7129d452\") " pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:09:47.718378 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:47.718061 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 22:09:47.718378 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:47.718128 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce5ffbb6-99ac-4817-acd9-f6cd4577f724-original-pull-secret podName:ce5ffbb6-99ac-4817-acd9-f6cd4577f724 nodeName:}" failed. No retries permitted until 2026-07-14 22:10:03.718108794 +0000 UTC m=+34.077255222 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce5ffbb6-99ac-4817-acd9-f6cd4577f724-original-pull-secret") pod "global-pull-secret-syncer-6g6bp" (UID: "ce5ffbb6-99ac-4817-acd9-f6cd4577f724") : object "kube-system"/"original-pull-secret" not registered Jul 14 22:09:47.718378 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:47.718133 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 22:09:47.718378 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:47.718173 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f9358e0-96a8-473f-896b-907e7129d452-metrics-certs podName:7f9358e0-96a8-473f-896b-907e7129d452 nodeName:}" failed. No retries permitted until 2026-07-14 22:10:03.718158884 +0000 UTC m=+34.077305299 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f9358e0-96a8-473f-896b-907e7129d452-metrics-certs") pod "network-metrics-daemon-55gw6" (UID: "7f9358e0-96a8-473f-896b-907e7129d452") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 22:09:47.919668 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:47.919573 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrtk5\" (UniqueName: \"kubernetes.io/projected/a549c30a-9937-422b-9f9b-17d10c48ab25-kube-api-access-wrtk5\") pod \"network-check-target-6p45d\" (UID: \"a549c30a-9937-422b-9f9b-17d10c48ab25\") " pod="openshift-network-diagnostics/network-check-target-6p45d" Jul 14 22:09:47.919830 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:47.919743 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 22:09:47.919830 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:47.919773 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 22:09:47.919830 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:47.919786 2550 projected.go:196] Error preparing data for projected volume kube-api-access-wrtk5 for pod openshift-network-diagnostics/network-check-target-6p45d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 22:09:47.919992 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:47.919836 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a549c30a-9937-422b-9f9b-17d10c48ab25-kube-api-access-wrtk5 podName:a549c30a-9937-422b-9f9b-17d10c48ab25 nodeName:}" failed. No retries permitted until 2026-07-14 22:10:03.919822321 +0000 UTC m=+34.278968741 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wrtk5" (UniqueName: "kubernetes.io/projected/a549c30a-9937-422b-9f9b-17d10c48ab25-kube-api-access-wrtk5") pod "network-check-target-6p45d" (UID: "a549c30a-9937-422b-9f9b-17d10c48ab25") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 22:09:48.189219 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:48.189148 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:09:48.189367 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:48.189257 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6bp" podUID="ce5ffbb6-99ac-4817-acd9-f6cd4577f724" Jul 14 22:09:49.190101 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:49.190058 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6p45d" Jul 14 22:09:49.190540 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:49.190068 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:09:49.190540 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:49.190214 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6p45d" podUID="a549c30a-9937-422b-9f9b-17d10c48ab25" Jul 14 22:09:49.190540 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:49.190313 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55gw6" podUID="7f9358e0-96a8-473f-896b-907e7129d452" Jul 14 22:09:50.189985 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:50.189946 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:09:50.190147 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:50.190047 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6bp" podUID="ce5ffbb6-99ac-4817-acd9-f6cd4577f724" Jul 14 22:09:51.189901 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:51.189871 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:09:51.190056 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:51.189983 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55gw6" podUID="7f9358e0-96a8-473f-896b-907e7129d452" Jul 14 22:09:51.190115 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:51.190043 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6p45d" Jul 14 22:09:51.190453 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:51.190165 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6p45d" podUID="a549c30a-9937-422b-9f9b-17d10c48ab25" Jul 14 22:09:52.193790 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:52.193519 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:09:52.194391 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:52.193872 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6bp" podUID="ce5ffbb6-99ac-4817-acd9-f6cd4577f724" Jul 14 22:09:52.292448 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:52.292420 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gmmnx" event={"ID":"3a228cf3-be18-4e92-bc1b-0045d1b2c823","Type":"ContainerStarted","Data":"3bb8751cba5f517a7092c5d8b5c79b0354309516542fdbd88d12dd988c18f53e"} Jul 14 22:09:52.293635 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:52.293613 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" event={"ID":"fe40e4f0-af16-4743-b435-48a292cd5fc4","Type":"ContainerStarted","Data":"c8963bc96325350d1c490e9b86e60fcefcde83796ce6941938762e46fd1bce8e"} Jul 14 22:09:52.294864 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:52.294838 2550 generic.go:356] "Generic (PLEG): container finished" podID="d5f7e8bb-6df3-4c30-84c7-4e30f116545a" containerID="238403e2058cdb0a9a0b2896d350a92b2c554666e0eaa841476ef0fc181e364d" exitCode=0 Jul 14 22:09:52.294951 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:52.294922 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l9qrh" event={"ID":"d5f7e8bb-6df3-4c30-84c7-4e30f116545a","Type":"ContainerDied","Data":"238403e2058cdb0a9a0b2896d350a92b2c554666e0eaa841476ef0fc181e364d"} Jul 14 22:09:52.297103 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:52.297086 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zj4sb_0884baa5-9206-4196-a7c9-b5686623311b/ovn-acl-logging/0.log" Jul 14 22:09:52.297452 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:52.297392 2550 generic.go:356] "Generic (PLEG): container finished" podID="0884baa5-9206-4196-a7c9-b5686623311b" containerID="780fea4fd7c9217abb973ef2af03d0ffd0949172b41959208ea567a4f8e30db9" exitCode=1 Jul 14 22:09:52.297514 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:52.297448 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" event={"ID":"0884baa5-9206-4196-a7c9-b5686623311b","Type":"ContainerStarted","Data":"4cb619c5e9ab765429e8012f8d250d3a6522241a14458bdb4686d11f9fcb3731"} Jul 14 22:09:52.297514 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:52.297509 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" event={"ID":"0884baa5-9206-4196-a7c9-b5686623311b","Type":"ContainerStarted","Data":"0f58afe11d9f637723457d46c20be1e63ecdd836067479582ae82c642df3a6ff"} Jul 14 22:09:52.297632 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:52.297571 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" event={"ID":"0884baa5-9206-4196-a7c9-b5686623311b","Type":"ContainerStarted","Data":"f415192c77d1c5b4d440fb14c265740f02bc83c7d6ae9be52395769ceaf578a2"} Jul 14 22:09:52.297632 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:52.297601 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" event={"ID":"0884baa5-9206-4196-a7c9-b5686623311b","Type":"ContainerDied","Data":"780fea4fd7c9217abb973ef2af03d0ffd0949172b41959208ea567a4f8e30db9"} Jul 14 22:09:52.297632 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:52.297629 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" event={"ID":"0884baa5-9206-4196-a7c9-b5686623311b","Type":"ContainerStarted","Data":"b74d09df80ac4310f03dfa60f9ef513acb6e893957fb7bdfffb04a684dc2b876"} Jul 14 22:09:52.298835 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:52.298818 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wtrck" event={"ID":"0147c9ad-b474-4fda-8ad1-0661515acfdb","Type":"ContainerStarted","Data":"41a5197b249d011bf2083fadf26f14e869b1af336c3d87fdb2017b5d1499f74e"} Jul 14 22:09:52.300135 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:52.300113 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dj68k" event={"ID":"8c265d22-e8f9-4516-aaf7-b274420c1b23","Type":"ContainerStarted","Data":"b05898c048b3c4de17077719bc93462e37bbe7b287a8e246d46095f2278088f1"} Jul 14 22:09:52.302090 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:52.301868 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6tt4w" event={"ID":"1159189b-384f-4107-807e-250f243409df","Type":"ContainerStarted","Data":"445ee5597472936607be6a9ab6f827033da6afb34ac92ab3d1183031cb93cd41"} Jul 14 22:09:52.303119 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:52.303099 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsf6v" event={"ID":"2cbd6276-0837-4e3e-b5d0-1770b73f48a2","Type":"ContainerStarted","Data":"f5637a9ee4982f403d71aec5b6a7d8c3c3d4404fc72328095e82300711ce5020"} Jul 14 22:09:52.334329 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:52.334279 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6tt4w" podStartSLOduration=3.768901905 podStartE2EDuration="22.334257359s" totalImagesPullingTime="18.565355454s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:09:52.334126963 +0000 UTC m=+22.693273402" watchObservedRunningTime="2026-07-14 22:09:52.334257359 +0000 UTC m=+22.693403796" Jul 14 22:09:52.371695 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:52.371645 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wtrck" podStartSLOduration=3.767934685 podStartE2EDuration="22.371624125s" totalImagesPullingTime="18.60368944s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:09:52.355842222 +0000 UTC m=+22.714988686" watchObservedRunningTime="2026-07-14 22:09:52.371624125 +0000 UTC m=+22.730770563" Jul 14 22:09:52.371954 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:52.371917 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dj68k" podStartSLOduration=3.809627955 podStartE2EDuration="22.371908159s" totalImagesPullingTime="18.562280204s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:09:52.371336629 +0000 UTC m=+22.730483057" watchObservedRunningTime="2026-07-14 22:09:52.371908159 +0000 UTC m=+22.731054596" Jul 14 22:09:52.389497 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:52.389450 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jdnlk" podStartSLOduration=3.8171453360000003 podStartE2EDuration="22.38943128s" totalImagesPullingTime="18.572285944s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:09:52.388858976 +0000 UTC m=+22.748005412" watchObservedRunningTime="2026-07-14 22:09:52.38943128 +0000 UTC m=+22.748577720" Jul 14 22:09:52.405907 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:52.405780 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gmmnx" podStartSLOduration=7.922500043 podStartE2EDuration="22.405766263s" totalImagesPullingTime="14.48326622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:09:52.405437462 +0000 UTC m=+22.764583903" watchObservedRunningTime="2026-07-14 22:09:52.405766263 +0000 UTC m=+22.764912700" Jul 14 22:09:52.500303 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:52.500268 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:52.703819 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:52.703769 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:53.046389 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:53.046347 2550 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 14 22:09:53.061349 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:53.061246 2550 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-14T22:09:53.046368488Z","UUID":"579327b0-7c3a-4b49-a9df-f070c7f15c6a","Handler":null,"Name":"","Endpoint":""} Jul 14 22:09:53.062852 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:53.062833 2550 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 14 22:09:53.062957 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:53.062872 2550 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 14 22:09:53.123609 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:53.123576 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:53.189833 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:53.189806 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6p45d" Jul 14 22:09:53.190037 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:53.189818 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:09:53.190037 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:53.189958 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6p45d" podUID="a549c30a-9937-422b-9f9b-17d10c48ab25" Jul 14 22:09:53.190193 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:53.190095 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55gw6" podUID="7f9358e0-96a8-473f-896b-907e7129d452" Jul 14 22:09:53.306739 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:53.306655 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kws8r" event={"ID":"ca583798-970a-42ec-a335-edb5a50383ad","Type":"ContainerStarted","Data":"cf151e6b630cfb48b8624bd8d9eec1371875f77816ee37c9706c947050b2394c"} Jul 14 22:09:53.308786 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:53.308758 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsf6v" event={"ID":"2cbd6276-0837-4e3e-b5d0-1770b73f48a2","Type":"ContainerStarted","Data":"99161b5511434bcf24792a51218c219dd9080e5de232973063e50647e7e00cd1"} Jul 14 22:09:53.311883 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:53.311852 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zj4sb_0884baa5-9206-4196-a7c9-b5686623311b/ovn-acl-logging/0.log" Jul 14 22:09:53.312239 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:53.312212 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" event={"ID":"0884baa5-9206-4196-a7c9-b5686623311b","Type":"ContainerStarted","Data":"e23fc526950987c9582788d4bb5b9957dafa589f41d006917192bae019699cfc"} Jul 14 22:09:53.323386 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:53.323345 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kws8r" podStartSLOduration=4.761550375 podStartE2EDuration="23.323330666s" totalImagesPullingTime="18.561780291s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:09:53.322805809 +0000 UTC m=+23.681952246" watchObservedRunningTime="2026-07-14 22:09:53.323330666 +0000 UTC m=+23.682477103" Jul 14 22:09:53.461565 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:53.461526 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gmmnx" Jul 14 22:09:53.462122 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:53.462102 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gmmnx" Jul 14 22:09:53.698844 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:53.698768 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:53.907238 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:53.907197 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:54.189887 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:54.189817 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:09:54.190040 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:54.189944 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6bp" podUID="ce5ffbb6-99ac-4817-acd9-f6cd4577f724" Jul 14 22:09:54.310954 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:54.310924 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:54.702037 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:54.701798 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:54.898811 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:54.898731 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:55.090568 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:55.090523 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:55.189893 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:55.189807 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:09:55.190053 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:55.189805 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6p45d" Jul 14 22:09:55.190053 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:55.189946 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55gw6" podUID="7f9358e0-96a8-473f-896b-907e7129d452" Jul 14 22:09:55.190053 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:55.190020 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6p45d" podUID="a549c30a-9937-422b-9f9b-17d10c48ab25" Jul 14 22:09:55.311394 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:55.311354 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:55.320307 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:55.320290 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zj4sb_0884baa5-9206-4196-a7c9-b5686623311b/ovn-acl-logging/0.log" Jul 14 22:09:55.320826 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:55.320800 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" event={"ID":"0884baa5-9206-4196-a7c9-b5686623311b","Type":"ContainerStarted","Data":"1945da895d5ac6329cd44d3bdded125d619fdeb62f32b29da542202428e08555"} Jul 14 22:09:55.324073 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:55.324047 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsf6v" event={"ID":"2cbd6276-0837-4e3e-b5d0-1770b73f48a2","Type":"ContainerStarted","Data":"c23f5170e742e64dda1954cabb73405656ae18079c5c03b0dfa9bf44439522c7"} Jul 14 22:09:55.324161 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:55.324084 2550 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 14 22:09:55.347826 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:55.347789 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsf6v" podStartSLOduration=4.445485505 podStartE2EDuration="25.34777245s" totalImagesPullingTime="20.902286945s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:09:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:09:55.347132544 +0000 UTC m=+25.706278983" watchObservedRunningTime="2026-07-14 22:09:55.34777245 +0000 UTC m=+25.706918896" Jul 14 22:09:55.699801 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:55.699763 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:55.905081 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:55.905047 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:09:56.189731 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:56.189660 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:09:56.189880 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:56.189796 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6bp" podUID="ce5ffbb6-99ac-4817-acd9-f6cd4577f724" Jul 14 22:09:57.189972 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:57.189744 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6p45d" Jul 14 22:09:57.190597 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:57.189775 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:09:57.190597 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:57.189981 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6p45d" podUID="a549c30a-9937-422b-9f9b-17d10c48ab25" Jul 14 22:09:57.190597 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:57.190081 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55gw6" podUID="7f9358e0-96a8-473f-896b-907e7129d452" Jul 14 22:09:57.329967 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:57.329933 2550 generic.go:356] "Generic (PLEG): container finished" podID="d5f7e8bb-6df3-4c30-84c7-4e30f116545a" containerID="293d61103750f458f5142a55b741797884414f46eb0e0c21608456aaa0d9348e" exitCode=0 Jul 14 22:09:57.330110 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:57.330024 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l9qrh" event={"ID":"d5f7e8bb-6df3-4c30-84c7-4e30f116545a","Type":"ContainerDied","Data":"293d61103750f458f5142a55b741797884414f46eb0e0c21608456aaa0d9348e"} Jul 14 22:09:57.335771 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:57.335751 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zj4sb_0884baa5-9206-4196-a7c9-b5686623311b/ovn-acl-logging/0.log" Jul 14 22:09:57.336116 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:57.336089 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" event={"ID":"0884baa5-9206-4196-a7c9-b5686623311b","Type":"ContainerStarted","Data":"3f63b54af6c8496acc8a5d00bece06941381e48a9bd009b0b5982532572b50aa"} Jul 14 22:09:57.336360 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:57.336343 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:57.336451 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:57.336390 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:57.336537 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:57.336523 2550 scope.go:117] "RemoveContainer" containerID="780fea4fd7c9217abb973ef2af03d0ffd0949172b41959208ea567a4f8e30db9" Jul 14 22:09:57.352221 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:57.352199 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:58.136674 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:58.136647 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gmmnx" Jul 14 22:09:58.136824 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:58.136812 2550 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 14 22:09:58.137242 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:58.137219 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gmmnx" Jul 14 22:09:58.192314 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:58.192297 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:09:58.192639 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:58.192370 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6bp" podUID="ce5ffbb6-99ac-4817-acd9-f6cd4577f724" Jul 14 22:09:58.340437 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:58.340352 2550 generic.go:356] "Generic (PLEG): container finished" podID="d5f7e8bb-6df3-4c30-84c7-4e30f116545a" containerID="3617262784a6f27c073935c2cd732e21a0d9d364995d8d34b76ad19c7e37674c" exitCode=0 Jul 14 22:09:58.340592 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:58.340437 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l9qrh" event={"ID":"d5f7e8bb-6df3-4c30-84c7-4e30f116545a","Type":"ContainerDied","Data":"3617262784a6f27c073935c2cd732e21a0d9d364995d8d34b76ad19c7e37674c"} Jul 14 22:09:58.344098 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:58.344082 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zj4sb_0884baa5-9206-4196-a7c9-b5686623311b/ovn-acl-logging/0.log" Jul 14 22:09:58.344439 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:58.344413 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" event={"ID":"0884baa5-9206-4196-a7c9-b5686623311b","Type":"ContainerStarted","Data":"a0b3174b64aff9d3c45930a9cb7296efced26166d2dc24cd8344998998bf2426"} Jul 14 22:09:58.344722 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:58.344699 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:58.359565 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:58.359537 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:09:58.397911 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:58.397871 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" podStartSLOduration=9.782372119 podStartE2EDuration="28.397856917s" totalImagesPullingTime="18.615484798s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:09:58.391867501 +0000 UTC m=+28.751013938" watchObservedRunningTime="2026-07-14 22:09:58.397856917 +0000 UTC m=+28.757003354" Jul 14 22:09:59.189489 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:59.189416 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6p45d" Jul 14 22:09:59.189635 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:59.189512 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6p45d" podUID="a549c30a-9937-422b-9f9b-17d10c48ab25" Jul 14 22:09:59.189635 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:59.189566 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:09:59.189705 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:09:59.189660 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55gw6" podUID="7f9358e0-96a8-473f-896b-907e7129d452" Jul 14 22:09:59.348585 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:59.348534 2550 generic.go:356] "Generic (PLEG): container finished" podID="d5f7e8bb-6df3-4c30-84c7-4e30f116545a" containerID="7b01d25abb4e9615e68c87dbdf83ee55124b7df4ce42476c757946d554b52f8e" exitCode=0 Jul 14 22:09:59.349012 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:09:59.348620 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l9qrh" event={"ID":"d5f7e8bb-6df3-4c30-84c7-4e30f116545a","Type":"ContainerDied","Data":"7b01d25abb4e9615e68c87dbdf83ee55124b7df4ce42476c757946d554b52f8e"} Jul 14 22:10:00.190669 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:00.190637 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:10:00.190843 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:10:00.190731 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6bp" podUID="ce5ffbb6-99ac-4817-acd9-f6cd4577f724" Jul 14 22:10:00.584616 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:00.584526 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:01.189849 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:01.189809 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6p45d" Jul 14 22:10:01.190034 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:10:01.189928 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6p45d" podUID="a549c30a-9937-422b-9f9b-17d10c48ab25" Jul 14 22:10:01.190034 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:01.189987 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:10:01.190138 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:10:01.190115 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55gw6" podUID="7f9358e0-96a8-473f-896b-907e7129d452" Jul 14 22:10:02.189881 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:02.189843 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:10:02.190361 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:10:02.189983 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6bp" podUID="ce5ffbb6-99ac-4817-acd9-f6cd4577f724" Jul 14 22:10:02.859592 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:02.859538 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:02.985653 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:02.985615 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:03.156164 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:03.156017 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:03.189923 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:03.189893 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6p45d" Jul 14 22:10:03.190331 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:03.189897 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:10:03.190331 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:10:03.190017 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6p45d" podUID="a549c30a-9937-422b-9f9b-17d10c48ab25" Jul 14 22:10:03.190331 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:10:03.190130 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55gw6" podUID="7f9358e0-96a8-473f-896b-907e7129d452" Jul 14 22:10:03.355581 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:03.355478 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:03.450719 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:03.450644 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:03.624251 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:03.623945 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-55gw6"] Jul 14 22:10:03.624251 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:03.624072 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:10:03.624251 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:10:03.624191 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55gw6" podUID="7f9358e0-96a8-473f-896b-907e7129d452" Jul 14 22:10:03.624802 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:03.624770 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6p45d"] Jul 14 22:10:03.624902 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:03.624847 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6p45d" Jul 14 22:10:03.624962 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:10:03.624927 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6p45d" podUID="a549c30a-9937-422b-9f9b-17d10c48ab25" Jul 14 22:10:03.626789 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:03.626728 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6g6bp"] Jul 14 22:10:03.626900 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:03.626810 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:10:03.626960 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:10:03.626902 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6bp" podUID="ce5ffbb6-99ac-4817-acd9-f6cd4577f724" Jul 14 22:10:03.743503 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:03.743475 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce5ffbb6-99ac-4817-acd9-f6cd4577f724-original-pull-secret\") pod \"global-pull-secret-syncer-6g6bp\" (UID: \"ce5ffbb6-99ac-4817-acd9-f6cd4577f724\") " pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:10:03.743659 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:03.743517 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f9358e0-96a8-473f-896b-907e7129d452-metrics-certs\") pod \"network-metrics-daemon-55gw6\" (UID: \"7f9358e0-96a8-473f-896b-907e7129d452\") " pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:10:03.743659 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:10:03.743640 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 22:10:03.743659 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:10:03.743645 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 22:10:03.743797 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:10:03.743710 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f9358e0-96a8-473f-896b-907e7129d452-metrics-certs podName:7f9358e0-96a8-473f-896b-907e7129d452 nodeName:}" failed. No retries permitted until 2026-07-14 22:10:35.743690172 +0000 UTC m=+66.102836607 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f9358e0-96a8-473f-896b-907e7129d452-metrics-certs") pod "network-metrics-daemon-55gw6" (UID: "7f9358e0-96a8-473f-896b-907e7129d452") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 22:10:03.743797 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:10:03.743729 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce5ffbb6-99ac-4817-acd9-f6cd4577f724-original-pull-secret podName:ce5ffbb6-99ac-4817-acd9-f6cd4577f724 nodeName:}" failed. No retries permitted until 2026-07-14 22:10:35.743720025 +0000 UTC m=+66.102866442 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce5ffbb6-99ac-4817-acd9-f6cd4577f724-original-pull-secret") pod "global-pull-secret-syncer-6g6bp" (UID: "ce5ffbb6-99ac-4817-acd9-f6cd4577f724") : object "kube-system"/"original-pull-secret" not registered Jul 14 22:10:03.840007 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:03.839972 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:03.945253 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:03.945221 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrtk5\" (UniqueName: \"kubernetes.io/projected/a549c30a-9937-422b-9f9b-17d10c48ab25-kube-api-access-wrtk5\") pod \"network-check-target-6p45d\" (UID: \"a549c30a-9937-422b-9f9b-17d10c48ab25\") " pod="openshift-network-diagnostics/network-check-target-6p45d" Jul 14 22:10:03.945434 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:10:03.945359 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 22:10:03.945434 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:10:03.945379 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 22:10:03.945434 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:10:03.945393 2550 projected.go:196] Error preparing data for projected volume kube-api-access-wrtk5 for pod openshift-network-diagnostics/network-check-target-6p45d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 22:10:03.945621 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:10:03.945451 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a549c30a-9937-422b-9f9b-17d10c48ab25-kube-api-access-wrtk5 podName:a549c30a-9937-422b-9f9b-17d10c48ab25 nodeName:}" failed. No retries permitted until 2026-07-14 22:10:35.945432399 +0000 UTC m=+66.304578828 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wrtk5" (UniqueName: "kubernetes.io/projected/a549c30a-9937-422b-9f9b-17d10c48ab25-kube-api-access-wrtk5") pod "network-check-target-6p45d" (UID: "a549c30a-9937-422b-9f9b-17d10c48ab25") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 22:10:04.225950 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:04.225911 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:04.326305 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:04.326276 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:04.362205 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:04.362178 2550 generic.go:356] "Generic (PLEG): container finished" podID="d5f7e8bb-6df3-4c30-84c7-4e30f116545a" containerID="c352b8ec3fa3d2f362a7398f18955fb59d2a1fb1ba71944c00c0043c778a5af3" exitCode=0 Jul 14 22:10:04.362354 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:04.362232 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l9qrh" event={"ID":"d5f7e8bb-6df3-4c30-84c7-4e30f116545a","Type":"ContainerDied","Data":"c352b8ec3fa3d2f362a7398f18955fb59d2a1fb1ba71944c00c0043c778a5af3"} Jul 14 22:10:04.954388 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:04.954307 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:05.051799 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:05.051765 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:05.190048 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:05.190019 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6p45d" Jul 14 22:10:05.190216 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:05.190114 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:10:05.190216 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:10:05.190125 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6p45d" podUID="a549c30a-9937-422b-9f9b-17d10c48ab25" Jul 14 22:10:05.190216 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:05.190139 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:10:05.190312 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:10:05.190220 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6bp" podUID="ce5ffbb6-99ac-4817-acd9-f6cd4577f724" Jul 14 22:10:05.190312 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:10:05.190285 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55gw6" podUID="7f9358e0-96a8-473f-896b-907e7129d452" Jul 14 22:10:05.366805 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:05.366734 2550 generic.go:356] "Generic (PLEG): container finished" podID="d5f7e8bb-6df3-4c30-84c7-4e30f116545a" containerID="06a732f955eea53731fecf1f0c13cd09f79aa1bbee1191e95ae2d92811296b99" exitCode=0 Jul 14 22:10:05.366805 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:05.366780 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l9qrh" event={"ID":"d5f7e8bb-6df3-4c30-84c7-4e30f116545a","Type":"ContainerDied","Data":"06a732f955eea53731fecf1f0c13cd09f79aa1bbee1191e95ae2d92811296b99"} Jul 14 22:10:06.372071 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:06.372041 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l9qrh" event={"ID":"d5f7e8bb-6df3-4c30-84c7-4e30f116545a","Type":"ContainerStarted","Data":"519bfc79e0273bbd42c502dc4c75b62929250b9eeefb2a7f7c0ee9f4e18b9a19"} Jul 14 22:10:06.397941 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:06.397900 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-l9qrh" podStartSLOduration=8.411719075 podStartE2EDuration="36.397885064s" totalImagesPullingTime="27.986165989s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:09:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:10:06.397590261 +0000 UTC m=+36.756736699" watchObservedRunningTime="2026-07-14 22:10:06.397885064 +0000 UTC m=+36.757031502" Jul 14 22:10:06.526291 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:06.526258 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:06.627672 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:06.627601 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:07.190074 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.190045 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:10:07.190236 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:10:07.190135 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6g6bp" podUID="ce5ffbb6-99ac-4817-acd9-f6cd4577f724" Jul 14 22:10:07.190236 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.190211 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:10:07.190322 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:10:07.190301 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55gw6" podUID="7f9358e0-96a8-473f-896b-907e7129d452" Jul 14 22:10:07.190395 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.190381 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6p45d" Jul 14 22:10:07.190484 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:10:07.190466 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6p45d" podUID="a549c30a-9937-422b-9f9b-17d10c48ab25" Jul 14 22:10:07.485837 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.485600 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-118.ec2.internal" event="NodeReady" Jul 14 22:10:07.486258 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.485877 2550 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 14 22:10:07.523129 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.523105 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-794ffd4775-6nwdp"] Jul 14 22:10:07.530585 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.530565 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-794ffd4775-6nwdp" Jul 14 22:10:07.535028 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.535006 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 14 22:10:07.535131 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.535027 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 14 22:10:07.535131 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.535064 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gv7p9\"" Jul 14 22:10:07.535367 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.535352 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 14 22:10:07.540158 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.540141 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 14 22:10:07.554640 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.554613 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-794ffd4775-6nwdp"] Jul 14 22:10:07.557308 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.557267 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-b4h5c"] Jul 14 22:10:07.564542 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.564525 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dktnm"] Jul 14 22:10:07.564676 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.564661 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b4h5c" Jul 14 22:10:07.568857 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.568836 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 14 22:10:07.568857 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.568850 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 14 22:10:07.569007 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.568869 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 14 22:10:07.569007 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.568954 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-z7z4x\"" Jul 14 22:10:07.569697 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.569677 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 14 22:10:07.572931 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.572908 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b4h5c"] Jul 14 22:10:07.573016 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.573002 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dktnm" Jul 14 22:10:07.578390 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.578370 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 14 22:10:07.578390 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.578382 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 14 22:10:07.578588 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.578572 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-29nlw\"" Jul 14 22:10:07.579208 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.579191 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dktnm"] Jul 14 22:10:07.645122 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.645091 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2jnrx"] Jul 14 22:10:07.662704 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.662680 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2jnrx"] Jul 14 22:10:07.662840 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.662768 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2jnrx" Jul 14 22:10:07.666317 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.666294 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 14 22:10:07.666483 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.666467 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 14 22:10:07.666578 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.666512 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jl5db\"" Jul 14 22:10:07.666810 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.666795 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 14 22:10:07.672599 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.672577 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9aa0f4fd-943b-4367-9e82-1190d2d4e1c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b4h5c\" (UID: \"9aa0f4fd-943b-4367-9e82-1190d2d4e1c8\") " pod="openshift-insights/insights-runtime-extractor-b4h5c" Jul 14 22:10:07.672707 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.672610 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5e9f87-1d85-41a5-af29-5e7586fe23f9-config-volume\") pod \"dns-default-dktnm\" (UID: \"7d5e9f87-1d85-41a5-af29-5e7586fe23f9\") " pod="openshift-dns/dns-default-dktnm" Jul 14 22:10:07.672707 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.672627 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3-registry-tls\") pod \"image-registry-794ffd4775-6nwdp\" (UID: \"f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3\") " pod="openshift-image-registry/image-registry-794ffd4775-6nwdp" Jul 14 22:10:07.672707 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.672643 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3-ca-trust-extracted\") pod \"image-registry-794ffd4775-6nwdp\" (UID: \"f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3\") " pod="openshift-image-registry/image-registry-794ffd4775-6nwdp" Jul 14 22:10:07.672707 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.672658 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3-installation-pull-secrets\") pod \"image-registry-794ffd4775-6nwdp\" (UID: \"f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3\") " pod="openshift-image-registry/image-registry-794ffd4775-6nwdp" Jul 14 22:10:07.673123 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.672736 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9aa0f4fd-943b-4367-9e82-1190d2d4e1c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b4h5c\" (UID: \"9aa0f4fd-943b-4367-9e82-1190d2d4e1c8\") " pod="openshift-insights/insights-runtime-extractor-b4h5c" Jul 14 22:10:07.673232 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.673134 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3-registry-certificates\") pod \"image-registry-794ffd4775-6nwdp\" (UID: \"f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3\") " pod="openshift-image-registry/image-registry-794ffd4775-6nwdp" Jul 14 22:10:07.673232 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.673175 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl4wl\" (UniqueName: \"kubernetes.io/projected/9aa0f4fd-943b-4367-9e82-1190d2d4e1c8-kube-api-access-rl4wl\") pod \"insights-runtime-extractor-b4h5c\" (UID: \"9aa0f4fd-943b-4367-9e82-1190d2d4e1c8\") " pod="openshift-insights/insights-runtime-extractor-b4h5c" Jul 14 22:10:07.673357 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.673258 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3-trusted-ca\") pod \"image-registry-794ffd4775-6nwdp\" (UID: \"f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3\") " pod="openshift-image-registry/image-registry-794ffd4775-6nwdp" Jul 14 22:10:07.673411 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.673352 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9aa0f4fd-943b-4367-9e82-1190d2d4e1c8-crio-socket\") pod \"insights-runtime-extractor-b4h5c\" (UID: \"9aa0f4fd-943b-4367-9e82-1190d2d4e1c8\") " pod="openshift-insights/insights-runtime-extractor-b4h5c" Jul 14 22:10:07.673467 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.673431 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3-bound-sa-token\") pod \"image-registry-794ffd4775-6nwdp\" (UID: \"f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3\") " pod="openshift-image-registry/image-registry-794ffd4775-6nwdp" Jul 14 22:10:07.673588 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.673473 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d5e9f87-1d85-41a5-af29-5e7586fe23f9-metrics-tls\") pod \"dns-default-dktnm\" (UID: \"7d5e9f87-1d85-41a5-af29-5e7586fe23f9\") " pod="openshift-dns/dns-default-dktnm" Jul 14 22:10:07.673588 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.673504 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9aa0f4fd-943b-4367-9e82-1190d2d4e1c8-data-volume\") pod \"insights-runtime-extractor-b4h5c\" (UID: \"9aa0f4fd-943b-4367-9e82-1190d2d4e1c8\") " pod="openshift-insights/insights-runtime-extractor-b4h5c" Jul 14 22:10:07.673588 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.673537 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d5e9f87-1d85-41a5-af29-5e7586fe23f9-tmp-dir\") pod \"dns-default-dktnm\" (UID: \"7d5e9f87-1d85-41a5-af29-5e7586fe23f9\") " pod="openshift-dns/dns-default-dktnm" Jul 14 22:10:07.673804 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.673584 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6n6l\" (UniqueName: \"kubernetes.io/projected/f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3-kube-api-access-l6n6l\") pod \"image-registry-794ffd4775-6nwdp\" (UID: \"f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3\") " pod="openshift-image-registry/image-registry-794ffd4775-6nwdp" Jul 14 22:10:07.673804 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.673663 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvglj\" (UniqueName: \"kubernetes.io/projected/7d5e9f87-1d85-41a5-af29-5e7586fe23f9-kube-api-access-mvglj\") pod \"dns-default-dktnm\" (UID: \"7d5e9f87-1d85-41a5-af29-5e7586fe23f9\") " pod="openshift-dns/dns-default-dktnm" Jul 14 22:10:07.673804 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.673693 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3-image-registry-private-configuration\") pod \"image-registry-794ffd4775-6nwdp\" (UID: \"f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3\") " pod="openshift-image-registry/image-registry-794ffd4775-6nwdp" Jul 14 22:10:07.724301 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.724271 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:07.774342 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.774284 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d5e9f87-1d85-41a5-af29-5e7586fe23f9-tmp-dir\") pod \"dns-default-dktnm\" (UID: \"7d5e9f87-1d85-41a5-af29-5e7586fe23f9\") " pod="openshift-dns/dns-default-dktnm" Jul 14 22:10:07.774342 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.774319 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6n6l\" (UniqueName: \"kubernetes.io/projected/f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3-kube-api-access-l6n6l\") pod \"image-registry-794ffd4775-6nwdp\" (UID: \"f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3\") " pod="openshift-image-registry/image-registry-794ffd4775-6nwdp" Jul 14 22:10:07.774532 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.774367 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvglj\" (UniqueName: \"kubernetes.io/projected/7d5e9f87-1d85-41a5-af29-5e7586fe23f9-kube-api-access-mvglj\") pod \"dns-default-dktnm\" (UID: \"7d5e9f87-1d85-41a5-af29-5e7586fe23f9\") " pod="openshift-dns/dns-default-dktnm" Jul 14 22:10:07.774532 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.774395 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3-image-registry-private-configuration\") pod \"image-registry-794ffd4775-6nwdp\" (UID: \"f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3\") " pod="openshift-image-registry/image-registry-794ffd4775-6nwdp" Jul 14 22:10:07.774532 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.774424 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9aa0f4fd-943b-4367-9e82-1190d2d4e1c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b4h5c\" (UID: \"9aa0f4fd-943b-4367-9e82-1190d2d4e1c8\") " pod="openshift-insights/insights-runtime-extractor-b4h5c" Jul 14 22:10:07.774720 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.774680 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d5e9f87-1d85-41a5-af29-5e7586fe23f9-tmp-dir\") pod \"dns-default-dktnm\" (UID: \"7d5e9f87-1d85-41a5-af29-5e7586fe23f9\") " pod="openshift-dns/dns-default-dktnm" Jul 14 22:10:07.774720 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.774693 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5e9f87-1d85-41a5-af29-5e7586fe23f9-config-volume\") pod \"dns-default-dktnm\" (UID: \"7d5e9f87-1d85-41a5-af29-5e7586fe23f9\") " pod="openshift-dns/dns-default-dktnm" Jul 14 22:10:07.774814 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.774725 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3-registry-tls\") pod \"image-registry-794ffd4775-6nwdp\" (UID: \"f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3\") " pod="openshift-image-registry/image-registry-794ffd4775-6nwdp" Jul 14 22:10:07.774814 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.774745 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3-ca-trust-extracted\") pod \"image-registry-794ffd4775-6nwdp\" (UID: \"f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3\") " pod="openshift-image-registry/image-registry-794ffd4775-6nwdp" Jul 14 22:10:07.774814 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.774770 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3-installation-pull-secrets\") pod \"image-registry-794ffd4775-6nwdp\" (UID: \"f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3\") " pod="openshift-image-registry/image-registry-794ffd4775-6nwdp" Jul 14 22:10:07.774937 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.774815 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9aa0f4fd-943b-4367-9e82-1190d2d4e1c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b4h5c\" (UID: \"9aa0f4fd-943b-4367-9e82-1190d2d4e1c8\") " pod="openshift-insights/insights-runtime-extractor-b4h5c" Jul 14 22:10:07.774937 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.774844 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkkxf\" (UniqueName: \"kubernetes.io/projected/ae93f9b7-e76d-4f57-b149-b726ed58218a-kube-api-access-qkkxf\") pod \"ingress-canary-2jnrx\" (UID: \"ae93f9b7-e76d-4f57-b149-b726ed58218a\") " pod="openshift-ingress-canary/ingress-canary-2jnrx" Jul 14 22:10:07.774937 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.774868 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3-registry-certificates\") pod \"image-registry-794ffd4775-6nwdp\" (UID: \"f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3\") " pod="openshift-image-registry/image-registry-794ffd4775-6nwdp" Jul 14 22:10:07.774937 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.774891 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl4wl\" (UniqueName: \"kubernetes.io/projected/9aa0f4fd-943b-4367-9e82-1190d2d4e1c8-kube-api-access-rl4wl\") pod \"insights-runtime-extractor-b4h5c\" (UID: \"9aa0f4fd-943b-4367-9e82-1190d2d4e1c8\") " pod="openshift-insights/insights-runtime-extractor-b4h5c" Jul 14 22:10:07.775120 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.774960 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3-trusted-ca\") pod \"image-registry-794ffd4775-6nwdp\" (UID: \"f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3\") " pod="openshift-image-registry/image-registry-794ffd4775-6nwdp" Jul 14 22:10:07.775120 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.774986 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9aa0f4fd-943b-4367-9e82-1190d2d4e1c8-crio-socket\") pod \"insights-runtime-extractor-b4h5c\" (UID: \"9aa0f4fd-943b-4367-9e82-1190d2d4e1c8\") " pod="openshift-insights/insights-runtime-extractor-b4h5c" Jul 14 22:10:07.775120 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.775015 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3-bound-sa-token\") pod \"image-registry-794ffd4775-6nwdp\" (UID: \"f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3\") " pod="openshift-image-registry/image-registry-794ffd4775-6nwdp" Jul 14 22:10:07.775120 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.775049 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d5e9f87-1d85-41a5-af29-5e7586fe23f9-metrics-tls\") pod \"dns-default-dktnm\" (UID: \"7d5e9f87-1d85-41a5-af29-5e7586fe23f9\") " pod="openshift-dns/dns-default-dktnm" Jul 14 22:10:07.775120 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.775077 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae93f9b7-e76d-4f57-b149-b726ed58218a-cert\") pod \"ingress-canary-2jnrx\" (UID: \"ae93f9b7-e76d-4f57-b149-b726ed58218a\") " pod="openshift-ingress-canary/ingress-canary-2jnrx" Jul 14 22:10:07.775120 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.775104 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9aa0f4fd-943b-4367-9e82-1190d2d4e1c8-data-volume\") pod \"insights-runtime-extractor-b4h5c\" (UID: \"9aa0f4fd-943b-4367-9e82-1190d2d4e1c8\") " pod="openshift-insights/insights-runtime-extractor-b4h5c" Jul 14 22:10:07.775120 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.775106 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3-ca-trust-extracted\") pod \"image-registry-794ffd4775-6nwdp\" (UID: \"f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3\") " pod="openshift-image-registry/image-registry-794ffd4775-6nwdp" Jul 14 22:10:07.775405 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.775320 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9aa0f4fd-943b-4367-9e82-1190d2d4e1c8-crio-socket\") pod \"insights-runtime-extractor-b4h5c\" (UID: \"9aa0f4fd-943b-4367-9e82-1190d2d4e1c8\") " pod="openshift-insights/insights-runtime-extractor-b4h5c" Jul 14 22:10:07.775405 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.775329 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5e9f87-1d85-41a5-af29-5e7586fe23f9-config-volume\") pod \"dns-default-dktnm\" (UID: \"7d5e9f87-1d85-41a5-af29-5e7586fe23f9\") " pod="openshift-dns/dns-default-dktnm" Jul 14 22:10:07.776278 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.775655 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9aa0f4fd-943b-4367-9e82-1190d2d4e1c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b4h5c\" (UID: \"9aa0f4fd-943b-4367-9e82-1190d2d4e1c8\") " pod="openshift-insights/insights-runtime-extractor-b4h5c" Jul 14 22:10:07.776278 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.776100 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9aa0f4fd-943b-4367-9e82-1190d2d4e1c8-data-volume\") pod \"insights-runtime-extractor-b4h5c\" (UID: \"9aa0f4fd-943b-4367-9e82-1190d2d4e1c8\") " pod="openshift-insights/insights-runtime-extractor-b4h5c" Jul 14 22:10:07.776278 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.776239 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3-registry-certificates\") pod \"image-registry-794ffd4775-6nwdp\" (UID: \"f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3\") " pod="openshift-image-registry/image-registry-794ffd4775-6nwdp" Jul 14 22:10:07.776529 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.776513 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3-trusted-ca\") pod \"image-registry-794ffd4775-6nwdp\" (UID: \"f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3\") " pod="openshift-image-registry/image-registry-794ffd4775-6nwdp" Jul 14 22:10:07.778750 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.778727 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d5e9f87-1d85-41a5-af29-5e7586fe23f9-metrics-tls\") pod \"dns-default-dktnm\" (UID: \"7d5e9f87-1d85-41a5-af29-5e7586fe23f9\") " pod="openshift-dns/dns-default-dktnm" Jul 14 22:10:07.778840 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.778799 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9aa0f4fd-943b-4367-9e82-1190d2d4e1c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b4h5c\" (UID: \"9aa0f4fd-943b-4367-9e82-1190d2d4e1c8\") " pod="openshift-insights/insights-runtime-extractor-b4h5c" Jul 14 22:10:07.778840 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.778819 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3-registry-tls\") pod \"image-registry-794ffd4775-6nwdp\" (UID: \"f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3\") " pod="openshift-image-registry/image-registry-794ffd4775-6nwdp" Jul 14 22:10:07.778908 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.778849 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3-image-registry-private-configuration\") pod \"image-registry-794ffd4775-6nwdp\" (UID: \"f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3\") " pod="openshift-image-registry/image-registry-794ffd4775-6nwdp" Jul 14 22:10:07.778941 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.778912 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3-installation-pull-secrets\") pod \"image-registry-794ffd4775-6nwdp\" (UID: \"f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3\") " pod="openshift-image-registry/image-registry-794ffd4775-6nwdp" Jul 14 22:10:07.783685 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.783659 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvglj\" (UniqueName: \"kubernetes.io/projected/7d5e9f87-1d85-41a5-af29-5e7586fe23f9-kube-api-access-mvglj\") pod \"dns-default-dktnm\" (UID: \"7d5e9f87-1d85-41a5-af29-5e7586fe23f9\") " pod="openshift-dns/dns-default-dktnm" Jul 14 22:10:07.784007 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.783990 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl4wl\" (UniqueName: \"kubernetes.io/projected/9aa0f4fd-943b-4367-9e82-1190d2d4e1c8-kube-api-access-rl4wl\") pod \"insights-runtime-extractor-b4h5c\" (UID: \"9aa0f4fd-943b-4367-9e82-1190d2d4e1c8\") " pod="openshift-insights/insights-runtime-extractor-b4h5c" Jul 14 22:10:07.784094 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.784077 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3-bound-sa-token\") pod \"image-registry-794ffd4775-6nwdp\" (UID: \"f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3\") " pod="openshift-image-registry/image-registry-794ffd4775-6nwdp" Jul 14 22:10:07.784191 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.784173 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6n6l\" (UniqueName: \"kubernetes.io/projected/f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3-kube-api-access-l6n6l\") pod \"image-registry-794ffd4775-6nwdp\" (UID: \"f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3\") " pod="openshift-image-registry/image-registry-794ffd4775-6nwdp" Jul 14 22:10:07.792087 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.792063 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:07.839981 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.839957 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-794ffd4775-6nwdp" Jul 14 22:10:07.872665 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.872638 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b4h5c" Jul 14 22:10:07.875837 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.875816 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkkxf\" (UniqueName: \"kubernetes.io/projected/ae93f9b7-e76d-4f57-b149-b726ed58218a-kube-api-access-qkkxf\") pod \"ingress-canary-2jnrx\" (UID: \"ae93f9b7-e76d-4f57-b149-b726ed58218a\") " pod="openshift-ingress-canary/ingress-canary-2jnrx" Jul 14 22:10:07.875938 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.875878 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae93f9b7-e76d-4f57-b149-b726ed58218a-cert\") pod \"ingress-canary-2jnrx\" (UID: \"ae93f9b7-e76d-4f57-b149-b726ed58218a\") " pod="openshift-ingress-canary/ingress-canary-2jnrx" Jul 14 22:10:07.878180 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.878159 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae93f9b7-e76d-4f57-b149-b726ed58218a-cert\") pod \"ingress-canary-2jnrx\" (UID: \"ae93f9b7-e76d-4f57-b149-b726ed58218a\") " pod="openshift-ingress-canary/ingress-canary-2jnrx" Jul 14 22:10:07.881976 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.881955 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dktnm" Jul 14 22:10:07.885257 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.885236 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkkxf\" (UniqueName: \"kubernetes.io/projected/ae93f9b7-e76d-4f57-b149-b726ed58218a-kube-api-access-qkkxf\") pod \"ingress-canary-2jnrx\" (UID: \"ae93f9b7-e76d-4f57-b149-b726ed58218a\") " pod="openshift-ingress-canary/ingress-canary-2jnrx" Jul 14 22:10:07.970989 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:07.970957 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2jnrx" Jul 14 22:10:08.008794 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:08.008764 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-794ffd4775-6nwdp"] Jul 14 22:10:08.014701 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:10:08.014667 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1f23e43_cf64_40e8_9dd3_4ac6ad774dc3.slice/crio-a740f53420765e7dee25b136726a51e12d599042047e2e6d666f059ee3eecb08 WatchSource:0}: Error finding container a740f53420765e7dee25b136726a51e12d599042047e2e6d666f059ee3eecb08: Status 404 returned error can't find the container with id a740f53420765e7dee25b136726a51e12d599042047e2e6d666f059ee3eecb08 Jul 14 22:10:08.022188 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:08.022100 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b4h5c"] Jul 14 22:10:08.026098 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:10:08.026077 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aa0f4fd_943b_4367_9e82_1190d2d4e1c8.slice/crio-807b59df150f656f53e5179b02269a512a113488d97f9024cde154d4827d5268 WatchSource:0}: Error finding container 807b59df150f656f53e5179b02269a512a113488d97f9024cde154d4827d5268: Status 404 returned error can't find the container with id 807b59df150f656f53e5179b02269a512a113488d97f9024cde154d4827d5268 Jul 14 22:10:08.036185 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:08.036004 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dktnm"] Jul 14 22:10:08.039231 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:10:08.039205 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5e9f87_1d85_41a5_af29_5e7586fe23f9.slice/crio-42519b207389a2774b5535bbf4c26896ac1e63165edffd5d534d50351ddcdbf1 WatchSource:0}: Error finding container 42519b207389a2774b5535bbf4c26896ac1e63165edffd5d534d50351ddcdbf1: Status 404 returned error can't find the container with id 42519b207389a2774b5535bbf4c26896ac1e63165edffd5d534d50351ddcdbf1 Jul 14 22:10:08.109129 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:08.109099 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2jnrx"] Jul 14 22:10:08.111610 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:10:08.111576 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae93f9b7_e76d_4f57_b149_b726ed58218a.slice/crio-df87a81eb36667d7d1fbbbf06ba15010886686ce50dd523b797bb2980226da58 WatchSource:0}: Error finding container df87a81eb36667d7d1fbbbf06ba15010886686ce50dd523b797bb2980226da58: Status 404 returned error can't find the container with id df87a81eb36667d7d1fbbbf06ba15010886686ce50dd523b797bb2980226da58 Jul 14 22:10:08.378395 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:08.378308 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dktnm" event={"ID":"7d5e9f87-1d85-41a5-af29-5e7586fe23f9","Type":"ContainerStarted","Data":"42519b207389a2774b5535bbf4c26896ac1e63165edffd5d534d50351ddcdbf1"} Jul 14 22:10:08.379469 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:08.379434 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2jnrx" event={"ID":"ae93f9b7-e76d-4f57-b149-b726ed58218a","Type":"ContainerStarted","Data":"df87a81eb36667d7d1fbbbf06ba15010886686ce50dd523b797bb2980226da58"} Jul 14 22:10:08.380644 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:08.380621 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b4h5c" event={"ID":"9aa0f4fd-943b-4367-9e82-1190d2d4e1c8","Type":"ContainerStarted","Data":"807b59df150f656f53e5179b02269a512a113488d97f9024cde154d4827d5268"} Jul 14 22:10:08.382105 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:08.382086 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-794ffd4775-6nwdp" event={"ID":"f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3","Type":"ContainerStarted","Data":"23a78d5d0cfc81de61a74f9ea95d55c7868f0f86b1bed91926ecfb608dbcf692"} Jul 14 22:10:08.382189 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:08.382121 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-794ffd4775-6nwdp" event={"ID":"f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3","Type":"ContainerStarted","Data":"a740f53420765e7dee25b136726a51e12d599042047e2e6d666f059ee3eecb08"} Jul 14 22:10:08.382246 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:08.382213 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-794ffd4775-6nwdp" Jul 14 22:10:08.405274 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:08.405228 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-794ffd4775-6nwdp" podStartSLOduration=6.405211044 podStartE2EDuration="6.405211044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:10:08.403846782 +0000 UTC m=+38.762993219" watchObservedRunningTime="2026-07-14 22:10:08.405211044 +0000 UTC m=+38.764357471" Jul 14 22:10:08.872457 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:08.872348 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:08.964215 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:08.964160 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:09.116238 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:09.116181 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:09.190621 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:09.189660 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:10:09.190621 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:09.190087 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6p45d" Jul 14 22:10:09.190621 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:09.190344 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:10:09.195725 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:09.195071 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 22:10:09.197356 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:09.196792 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nwz7d\"" Jul 14 22:10:09.197356 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:09.196819 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vcczq\"" Jul 14 22:10:09.197356 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:09.197094 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 14 22:10:09.197356 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:09.197127 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 22:10:09.197628 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:09.197365 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 14 22:10:09.209234 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:09.209129 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:09.562924 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:09.562841 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:10.350101 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:10.350073 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:10.419497 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:10.419472 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:14.401739 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:14.398764 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dktnm" event={"ID":"7d5e9f87-1d85-41a5-af29-5e7586fe23f9","Type":"ContainerStarted","Data":"2097a9535475623740fac47fadbdeb14295fdca549d72ea3d2d42c90117a3b1c"} Jul 14 22:10:14.403171 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:14.403147 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2jnrx" event={"ID":"ae93f9b7-e76d-4f57-b149-b726ed58218a","Type":"ContainerStarted","Data":"e1cbd8c8be9cb19247b171cb48a104356eed4bde4ccd30a2ee9f6a48c23c2dfe"} Jul 14 22:10:14.690026 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:14.689975 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2jnrx" podStartSLOduration=1.742407485 podStartE2EDuration="7.689957273s" totalImagesPullingTime="5.947549788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:10:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:10:14.41974663 +0000 UTC m=+44.778893097" watchObservedRunningTime="2026-07-14 22:10:14.689957273 +0000 UTC m=+45.049103721" Jul 14 22:10:14.690581 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:14.690567 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-jghjl"] Jul 14 22:10:14.706433 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:14.706411 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-jghjl"] Jul 14 22:10:14.706591 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:14.706502 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-jghjl" Jul 14 22:10:14.709463 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:14.709441 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 14 22:10:14.709463 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:14.709461 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 14 22:10:14.709644 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:14.709620 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-4xqbm\"" Jul 14 22:10:14.727753 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:14.727730 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:14.731951 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:14.731932 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twt84\" (UniqueName: \"kubernetes.io/projected/1337dea6-26b7-4707-a955-1fc73f898523-kube-api-access-twt84\") pod \"downloads-68c9cbb497-jghjl\" (UID: \"1337dea6-26b7-4707-a955-1fc73f898523\") " pod="openshift-console/downloads-68c9cbb497-jghjl" Jul 14 22:10:14.828459 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:14.828425 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:14.832268 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:14.832246 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twt84\" (UniqueName: \"kubernetes.io/projected/1337dea6-26b7-4707-a955-1fc73f898523-kube-api-access-twt84\") pod \"downloads-68c9cbb497-jghjl\" (UID: \"1337dea6-26b7-4707-a955-1fc73f898523\") " pod="openshift-console/downloads-68c9cbb497-jghjl" Jul 14 22:10:14.842903 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:14.842875 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twt84\" (UniqueName: \"kubernetes.io/projected/1337dea6-26b7-4707-a955-1fc73f898523-kube-api-access-twt84\") pod \"downloads-68c9cbb497-jghjl\" (UID: \"1337dea6-26b7-4707-a955-1fc73f898523\") " pod="openshift-console/downloads-68c9cbb497-jghjl" Jul 14 22:10:15.002723 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:15.002653 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:15.015577 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:15.015545 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-jghjl" Jul 14 22:10:15.174597 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:15.174570 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:15.276356 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:15.276331 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:15.301734 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:15.301706 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-jghjl"] Jul 14 22:10:15.304400 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:10:15.304371 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1337dea6_26b7_4707_a955_1fc73f898523.slice/crio-d7e27354d596fe379ddbb4589b989a613e7e1277ed3d54d23fef78102b1817b3 WatchSource:0}: Error finding container d7e27354d596fe379ddbb4589b989a613e7e1277ed3d54d23fef78102b1817b3: Status 404 returned error can't find the container with id d7e27354d596fe379ddbb4589b989a613e7e1277ed3d54d23fef78102b1817b3 Jul 14 22:10:15.408204 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:15.408177 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b4h5c" event={"ID":"9aa0f4fd-943b-4367-9e82-1190d2d4e1c8","Type":"ContainerStarted","Data":"f6562c5f4b7500a498b64f3bdd5e539dbc63c474e5d30b123cbd441241b563ce"} Jul 14 22:10:15.409234 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:15.409214 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-jghjl" event={"ID":"1337dea6-26b7-4707-a955-1fc73f898523","Type":"ContainerStarted","Data":"d7e27354d596fe379ddbb4589b989a613e7e1277ed3d54d23fef78102b1817b3"} Jul 14 22:10:15.410870 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:15.410849 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dktnm" event={"ID":"7d5e9f87-1d85-41a5-af29-5e7586fe23f9","Type":"ContainerStarted","Data":"57cefa0ed93bb57a83aa99242562f8e040706226c2f8242796baf9d2b6f19be4"} Jul 14 22:10:15.451852 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:15.451829 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:15.475994 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:15.475950 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dktnm" podStartSLOduration=2.467697242 podStartE2EDuration="8.475933242s" totalImagesPullingTime="6.008236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:10:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:10:15.475314569 +0000 UTC m=+45.834461006" watchObservedRunningTime="2026-07-14 22:10:15.475933242 +0000 UTC m=+45.835079678" Jul 14 22:10:15.556279 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:15.556208 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:15.657156 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:15.657138 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:16.003207 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:16.003174 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:16.074221 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:16.074188 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:16.193038 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:16.192961 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:16.278239 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:16.278152 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:16.398187 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:16.398148 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:16.416830 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:16.416801 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b4h5c" event={"ID":"9aa0f4fd-943b-4367-9e82-1190d2d4e1c8","Type":"ContainerStarted","Data":"adcba07a1f907933845ebc0cf36482425c24abe4ec20d717d84dab542ec39817"} Jul 14 22:10:16.417256 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:16.416974 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dktnm" Jul 14 22:10:16.580009 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:16.579928 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:16.680309 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:16.680280 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:17.410688 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:17.410622 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:17.483222 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:17.483153 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:18.954749 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:18.954710 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:19.430411 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:19.430378 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b4h5c" event={"ID":"9aa0f4fd-943b-4367-9e82-1190d2d4e1c8","Type":"ContainerStarted","Data":"dede4138136245c0d64df2db70a7824b14ec12ef593bebbd00b5e4c5c552bb3e"} Jul 14 22:10:19.452795 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:19.452740 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-b4h5c" podStartSLOduration=1.743053938 podStartE2EDuration="12.45272024s" totalImagesPullingTime="10.709666302s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:10:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:10:19.451260632 +0000 UTC m=+49.810407070" watchObservedRunningTime="2026-07-14 22:10:19.45272024 +0000 UTC m=+49.811866678" Jul 14 22:10:20.398683 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.398640 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:20.589686 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.589649 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:20.637209 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.637178 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-t66mm"] Jul 14 22:10:20.649890 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.649828 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t66mm" Jul 14 22:10:20.652251 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.652232 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 14 22:10:20.652251 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.652241 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rz6kq\"" Jul 14 22:10:20.652509 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.652491 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 14 22:10:20.652607 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.652521 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 14 22:10:20.653626 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.653612 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 14 22:10:20.653685 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.653641 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 14 22:10:20.659455 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.659437 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-462d8"] Jul 14 22:10:20.674362 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.674339 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-t66mm"] Jul 14 22:10:20.674477 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.674388 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9bglr"] Jul 14 22:10:20.674572 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.674475 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-462d8" Jul 14 22:10:20.679101 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.679080 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 14 22:10:20.679295 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.679267 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4tl8j\"" Jul 14 22:10:20.679531 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.679512 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 14 22:10:20.679663 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.679516 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 14 22:10:20.684150 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.684134 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c770b0db-7fa6-4090-826a-412b9d542753-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-t66mm\" (UID: \"c770b0db-7fa6-4090-826a-412b9d542753\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t66mm" Jul 14 22:10:20.684226 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.684160 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c770b0db-7fa6-4090-826a-412b9d542753-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-t66mm\" (UID: \"c770b0db-7fa6-4090-826a-412b9d542753\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t66mm" Jul 14 22:10:20.684226 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.684177 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c770b0db-7fa6-4090-826a-412b9d542753-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-t66mm\" (UID: \"c770b0db-7fa6-4090-826a-412b9d542753\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t66mm" Jul 14 22:10:20.684298 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.684274 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdbxd\" (UniqueName: \"kubernetes.io/projected/c770b0db-7fa6-4090-826a-412b9d542753-kube-api-access-fdbxd\") pod \"openshift-state-metrics-84bd68b896-t66mm\" (UID: \"c770b0db-7fa6-4090-826a-412b9d542753\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t66mm" Jul 14 22:10:20.688475 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.688462 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-462d8"] Jul 14 22:10:20.688581 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.688545 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9bglr" Jul 14 22:10:20.690935 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.690919 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 14 22:10:20.691009 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.690970 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 14 22:10:20.691211 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.691199 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 14 22:10:20.691597 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.691583 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xrbwq\"" Jul 14 22:10:20.692590 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.692573 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:20.785400 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.785366 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/af11146f-0b62-492b-b73a-440e2c69a95b-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-462d8\" (UID: \"af11146f-0b62-492b-b73a-440e2c69a95b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-462d8" Jul 14 22:10:20.785400 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.785413 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1d845a84-9484-41d0-9373-27ede38c8f5f-root\") pod \"node-exporter-9bglr\" (UID: \"1d845a84-9484-41d0-9373-27ede38c8f5f\") " pod="openshift-monitoring/node-exporter-9bglr" Jul 14 22:10:20.785660 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.785445 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d845a84-9484-41d0-9373-27ede38c8f5f-node-exporter-tls\") pod \"node-exporter-9bglr\" (UID: \"1d845a84-9484-41d0-9373-27ede38c8f5f\") " pod="openshift-monitoring/node-exporter-9bglr" Jul 14 22:10:20.785660 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.785548 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1d845a84-9484-41d0-9373-27ede38c8f5f-node-exporter-wtmp\") pod \"node-exporter-9bglr\" (UID: \"1d845a84-9484-41d0-9373-27ede38c8f5f\") " pod="openshift-monitoring/node-exporter-9bglr" Jul 14 22:10:20.785660 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.785597 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/af11146f-0b62-492b-b73a-440e2c69a95b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-462d8\" (UID: \"af11146f-0b62-492b-b73a-440e2c69a95b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-462d8" Jul 14 22:10:20.785660 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.785628 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm6qr\" (UniqueName: \"kubernetes.io/projected/af11146f-0b62-492b-b73a-440e2c69a95b-kube-api-access-nm6qr\") pod \"kube-state-metrics-785778ff74-462d8\" (UID: \"af11146f-0b62-492b-b73a-440e2c69a95b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-462d8" Jul 14 22:10:20.785660 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.785656 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/af11146f-0b62-492b-b73a-440e2c69a95b-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-462d8\" (UID: \"af11146f-0b62-492b-b73a-440e2c69a95b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-462d8" Jul 14 22:10:20.785904 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.785680 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d845a84-9484-41d0-9373-27ede38c8f5f-metrics-client-ca\") pod \"node-exporter-9bglr\" (UID: \"1d845a84-9484-41d0-9373-27ede38c8f5f\") " pod="openshift-monitoring/node-exporter-9bglr" Jul 14 22:10:20.785904 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.785747 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdbxd\" (UniqueName: \"kubernetes.io/projected/c770b0db-7fa6-4090-826a-412b9d542753-kube-api-access-fdbxd\") pod \"openshift-state-metrics-84bd68b896-t66mm\" (UID: \"c770b0db-7fa6-4090-826a-412b9d542753\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t66mm" Jul 14 22:10:20.785904 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.785804 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1d845a84-9484-41d0-9373-27ede38c8f5f-node-exporter-textfile\") pod \"node-exporter-9bglr\" (UID: \"1d845a84-9484-41d0-9373-27ede38c8f5f\") " pod="openshift-monitoring/node-exporter-9bglr" Jul 14 22:10:20.785904 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.785835 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/af11146f-0b62-492b-b73a-440e2c69a95b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-462d8\" (UID: \"af11146f-0b62-492b-b73a-440e2c69a95b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-462d8" Jul 14 22:10:20.785904 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.785867 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d845a84-9484-41d0-9373-27ede38c8f5f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9bglr\" (UID: \"1d845a84-9484-41d0-9373-27ede38c8f5f\") " pod="openshift-monitoring/node-exporter-9bglr" Jul 14 22:10:20.785904 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.785894 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x9z6\" (UniqueName: \"kubernetes.io/projected/1d845a84-9484-41d0-9373-27ede38c8f5f-kube-api-access-2x9z6\") pod \"node-exporter-9bglr\" (UID: \"1d845a84-9484-41d0-9373-27ede38c8f5f\") " pod="openshift-monitoring/node-exporter-9bglr" Jul 14 22:10:20.786140 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.785931 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af11146f-0b62-492b-b73a-440e2c69a95b-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-462d8\" (UID: \"af11146f-0b62-492b-b73a-440e2c69a95b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-462d8" Jul 14 22:10:20.786140 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.785976 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1d845a84-9484-41d0-9373-27ede38c8f5f-node-exporter-accelerators-collector-config\") pod \"node-exporter-9bglr\" (UID: \"1d845a84-9484-41d0-9373-27ede38c8f5f\") " pod="openshift-monitoring/node-exporter-9bglr" Jul 14 22:10:20.786140 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.786001 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d845a84-9484-41d0-9373-27ede38c8f5f-sys\") pod \"node-exporter-9bglr\" (UID: \"1d845a84-9484-41d0-9373-27ede38c8f5f\") " pod="openshift-monitoring/node-exporter-9bglr" Jul 14 22:10:20.786140 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.786031 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c770b0db-7fa6-4090-826a-412b9d542753-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-t66mm\" (UID: \"c770b0db-7fa6-4090-826a-412b9d542753\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t66mm" Jul 14 22:10:20.786140 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.786068 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c770b0db-7fa6-4090-826a-412b9d542753-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-t66mm\" (UID: \"c770b0db-7fa6-4090-826a-412b9d542753\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t66mm" Jul 14 22:10:20.786140 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.786095 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c770b0db-7fa6-4090-826a-412b9d542753-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-t66mm\" (UID: \"c770b0db-7fa6-4090-826a-412b9d542753\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t66mm" Jul 14 22:10:20.786461 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:10:20.786203 2550 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 14 22:10:20.786461 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:10:20.786289 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c770b0db-7fa6-4090-826a-412b9d542753-openshift-state-metrics-tls podName:c770b0db-7fa6-4090-826a-412b9d542753 nodeName:}" failed. No retries permitted until 2026-07-14 22:10:21.286266705 +0000 UTC m=+51.645413138 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/c770b0db-7fa6-4090-826a-412b9d542753-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-t66mm" (UID: "c770b0db-7fa6-4090-826a-412b9d542753") : secret "openshift-state-metrics-tls" not found Jul 14 22:10:20.786779 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.786763 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c770b0db-7fa6-4090-826a-412b9d542753-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-t66mm\" (UID: \"c770b0db-7fa6-4090-826a-412b9d542753\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t66mm" Jul 14 22:10:20.788978 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.788953 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c770b0db-7fa6-4090-826a-412b9d542753-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-t66mm\" (UID: \"c770b0db-7fa6-4090-826a-412b9d542753\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t66mm" Jul 14 22:10:20.798332 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.798309 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdbxd\" (UniqueName: \"kubernetes.io/projected/c770b0db-7fa6-4090-826a-412b9d542753-kube-api-access-fdbxd\") pod \"openshift-state-metrics-84bd68b896-t66mm\" (UID: \"c770b0db-7fa6-4090-826a-412b9d542753\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t66mm" Jul 14 22:10:20.836465 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.836436 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:20.887365 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.887332 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d845a84-9484-41d0-9373-27ede38c8f5f-node-exporter-tls\") pod \"node-exporter-9bglr\" (UID: \"1d845a84-9484-41d0-9373-27ede38c8f5f\") " pod="openshift-monitoring/node-exporter-9bglr" Jul 14 22:10:20.887505 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.887379 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1d845a84-9484-41d0-9373-27ede38c8f5f-node-exporter-wtmp\") pod \"node-exporter-9bglr\" (UID: \"1d845a84-9484-41d0-9373-27ede38c8f5f\") " pod="openshift-monitoring/node-exporter-9bglr" Jul 14 22:10:20.887505 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.887407 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/af11146f-0b62-492b-b73a-440e2c69a95b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-462d8\" (UID: \"af11146f-0b62-492b-b73a-440e2c69a95b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-462d8" Jul 14 22:10:20.887505 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.887430 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm6qr\" (UniqueName: \"kubernetes.io/projected/af11146f-0b62-492b-b73a-440e2c69a95b-kube-api-access-nm6qr\") pod \"kube-state-metrics-785778ff74-462d8\" (UID: \"af11146f-0b62-492b-b73a-440e2c69a95b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-462d8" Jul 14 22:10:20.887505 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.887457 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/af11146f-0b62-492b-b73a-440e2c69a95b-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-462d8\" (UID: \"af11146f-0b62-492b-b73a-440e2c69a95b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-462d8" Jul 14 22:10:20.887505 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.887483 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d845a84-9484-41d0-9373-27ede38c8f5f-metrics-client-ca\") pod \"node-exporter-9bglr\" (UID: \"1d845a84-9484-41d0-9373-27ede38c8f5f\") " pod="openshift-monitoring/node-exporter-9bglr" Jul 14 22:10:20.887788 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.887535 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1d845a84-9484-41d0-9373-27ede38c8f5f-node-exporter-wtmp\") pod \"node-exporter-9bglr\" (UID: \"1d845a84-9484-41d0-9373-27ede38c8f5f\") " pod="openshift-monitoring/node-exporter-9bglr" Jul 14 22:10:20.887788 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.887542 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1d845a84-9484-41d0-9373-27ede38c8f5f-node-exporter-textfile\") pod \"node-exporter-9bglr\" (UID: \"1d845a84-9484-41d0-9373-27ede38c8f5f\") " pod="openshift-monitoring/node-exporter-9bglr" Jul 14 22:10:20.887788 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.887614 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/af11146f-0b62-492b-b73a-440e2c69a95b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-462d8\" (UID: \"af11146f-0b62-492b-b73a-440e2c69a95b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-462d8" Jul 14 22:10:20.887788 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.887650 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d845a84-9484-41d0-9373-27ede38c8f5f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9bglr\" (UID: \"1d845a84-9484-41d0-9373-27ede38c8f5f\") " pod="openshift-monitoring/node-exporter-9bglr" Jul 14 22:10:20.887788 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.887678 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x9z6\" (UniqueName: \"kubernetes.io/projected/1d845a84-9484-41d0-9373-27ede38c8f5f-kube-api-access-2x9z6\") pod \"node-exporter-9bglr\" (UID: \"1d845a84-9484-41d0-9373-27ede38c8f5f\") " pod="openshift-monitoring/node-exporter-9bglr" Jul 14 22:10:20.887788 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.887703 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af11146f-0b62-492b-b73a-440e2c69a95b-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-462d8\" (UID: \"af11146f-0b62-492b-b73a-440e2c69a95b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-462d8" Jul 14 22:10:20.887788 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.887731 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1d845a84-9484-41d0-9373-27ede38c8f5f-node-exporter-accelerators-collector-config\") pod \"node-exporter-9bglr\" (UID: \"1d845a84-9484-41d0-9373-27ede38c8f5f\") " pod="openshift-monitoring/node-exporter-9bglr" Jul 14 22:10:20.887788 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.887755 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d845a84-9484-41d0-9373-27ede38c8f5f-sys\") pod \"node-exporter-9bglr\" (UID: \"1d845a84-9484-41d0-9373-27ede38c8f5f\") " pod="openshift-monitoring/node-exporter-9bglr" Jul 14 22:10:20.888144 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.887839 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/af11146f-0b62-492b-b73a-440e2c69a95b-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-462d8\" (UID: \"af11146f-0b62-492b-b73a-440e2c69a95b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-462d8" Jul 14 22:10:20.888144 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.887865 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1d845a84-9484-41d0-9373-27ede38c8f5f-root\") pod \"node-exporter-9bglr\" (UID: \"1d845a84-9484-41d0-9373-27ede38c8f5f\") " pod="openshift-monitoring/node-exporter-9bglr" Jul 14 22:10:20.888144 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.887937 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1d845a84-9484-41d0-9373-27ede38c8f5f-root\") pod \"node-exporter-9bglr\" (UID: \"1d845a84-9484-41d0-9373-27ede38c8f5f\") " pod="openshift-monitoring/node-exporter-9bglr" Jul 14 22:10:20.888144 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.887954 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d845a84-9484-41d0-9373-27ede38c8f5f-sys\") pod \"node-exporter-9bglr\" (UID: \"1d845a84-9484-41d0-9373-27ede38c8f5f\") " pod="openshift-monitoring/node-exporter-9bglr" Jul 14 22:10:20.888332 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.888159 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/af11146f-0b62-492b-b73a-440e2c69a95b-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-462d8\" (UID: \"af11146f-0b62-492b-b73a-440e2c69a95b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-462d8" Jul 14 22:10:20.888332 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.888207 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1d845a84-9484-41d0-9373-27ede38c8f5f-node-exporter-textfile\") pod \"node-exporter-9bglr\" (UID: \"1d845a84-9484-41d0-9373-27ede38c8f5f\") " pod="openshift-monitoring/node-exporter-9bglr" Jul 14 22:10:20.888509 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.888478 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d845a84-9484-41d0-9373-27ede38c8f5f-metrics-client-ca\") pod \"node-exporter-9bglr\" (UID: \"1d845a84-9484-41d0-9373-27ede38c8f5f\") " pod="openshift-monitoring/node-exporter-9bglr" Jul 14 22:10:20.888693 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.888648 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1d845a84-9484-41d0-9373-27ede38c8f5f-node-exporter-accelerators-collector-config\") pod \"node-exporter-9bglr\" (UID: \"1d845a84-9484-41d0-9373-27ede38c8f5f\") " pod="openshift-monitoring/node-exporter-9bglr" Jul 14 22:10:20.888693 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.888665 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/af11146f-0b62-492b-b73a-440e2c69a95b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-462d8\" (UID: \"af11146f-0b62-492b-b73a-440e2c69a95b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-462d8" Jul 14 22:10:20.888853 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.888833 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af11146f-0b62-492b-b73a-440e2c69a95b-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-462d8\" (UID: \"af11146f-0b62-492b-b73a-440e2c69a95b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-462d8" Jul 14 22:10:20.890452 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.890426 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d845a84-9484-41d0-9373-27ede38c8f5f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9bglr\" (UID: \"1d845a84-9484-41d0-9373-27ede38c8f5f\") " pod="openshift-monitoring/node-exporter-9bglr" Jul 14 22:10:20.890538 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.890507 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d845a84-9484-41d0-9373-27ede38c8f5f-node-exporter-tls\") pod \"node-exporter-9bglr\" (UID: \"1d845a84-9484-41d0-9373-27ede38c8f5f\") " pod="openshift-monitoring/node-exporter-9bglr" Jul 14 22:10:20.890963 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.890768 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/af11146f-0b62-492b-b73a-440e2c69a95b-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-462d8\" (UID: \"af11146f-0b62-492b-b73a-440e2c69a95b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-462d8" Jul 14 22:10:20.890963 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.890878 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/af11146f-0b62-492b-b73a-440e2c69a95b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-462d8\" (UID: \"af11146f-0b62-492b-b73a-440e2c69a95b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-462d8" Jul 14 22:10:20.911853 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.911810 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm6qr\" (UniqueName: \"kubernetes.io/projected/af11146f-0b62-492b-b73a-440e2c69a95b-kube-api-access-nm6qr\") pod \"kube-state-metrics-785778ff74-462d8\" (UID: \"af11146f-0b62-492b-b73a-440e2c69a95b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-462d8" Jul 14 22:10:20.911964 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.911941 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x9z6\" (UniqueName: \"kubernetes.io/projected/1d845a84-9484-41d0-9373-27ede38c8f5f-kube-api-access-2x9z6\") pod \"node-exporter-9bglr\" (UID: \"1d845a84-9484-41d0-9373-27ede38c8f5f\") " pod="openshift-monitoring/node-exporter-9bglr" Jul 14 22:10:20.983679 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.983647 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-462d8" Jul 14 22:10:20.996941 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:20.996920 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9bglr" Jul 14 22:10:21.023521 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:21.023493 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:21.043543 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:10:21.043509 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d845a84_9484_41d0_9373_27ede38c8f5f.slice/crio-18a4f07e1a4ba856df6fffb5136ee96615668ef1523336812eefa5325fcf36cb WatchSource:0}: Error finding container 18a4f07e1a4ba856df6fffb5136ee96615668ef1523336812eefa5325fcf36cb: Status 404 returned error can't find the container with id 18a4f07e1a4ba856df6fffb5136ee96615668ef1523336812eefa5325fcf36cb Jul 14 22:10:21.163601 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:21.163503 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-462d8"] Jul 14 22:10:21.167141 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:10:21.167115 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf11146f_0b62_492b_b73a_440e2c69a95b.slice/crio-cba1f280f8482fc66e9c3323e3885e22e68b994eca788e6dec1a0bcbe9fb9d6c WatchSource:0}: Error finding container cba1f280f8482fc66e9c3323e3885e22e68b994eca788e6dec1a0bcbe9fb9d6c: Status 404 returned error can't find the container with id cba1f280f8482fc66e9c3323e3885e22e68b994eca788e6dec1a0bcbe9fb9d6c Jul 14 22:10:21.204214 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:21.204185 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:21.290653 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:21.290619 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c770b0db-7fa6-4090-826a-412b9d542753-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-t66mm\" (UID: \"c770b0db-7fa6-4090-826a-412b9d542753\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t66mm" Jul 14 22:10:21.293275 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:21.293246 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c770b0db-7fa6-4090-826a-412b9d542753-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-t66mm\" (UID: \"c770b0db-7fa6-4090-826a-412b9d542753\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t66mm" Jul 14 22:10:21.318675 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:21.318645 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:21.437776 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:21.437710 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-462d8" event={"ID":"af11146f-0b62-492b-b73a-440e2c69a95b","Type":"ContainerStarted","Data":"cba1f280f8482fc66e9c3323e3885e22e68b994eca788e6dec1a0bcbe9fb9d6c"} Jul 14 22:10:21.439030 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:21.439001 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9bglr" event={"ID":"1d845a84-9484-41d0-9373-27ede38c8f5f","Type":"ContainerStarted","Data":"18a4f07e1a4ba856df6fffb5136ee96615668ef1523336812eefa5325fcf36cb"} Jul 14 22:10:21.444065 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:21.444042 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:21.558433 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:21.558403 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t66mm" Jul 14 22:10:21.667932 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:21.667901 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:21.738811 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:21.738724 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-t66mm"] Jul 14 22:10:21.742613 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:10:21.742583 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc770b0db_7fa6_4090_826a_412b9d542753.slice/crio-f8730485056934cb22dde1b9afa579bbd90e7de5410a9ba540cb12645017663b WatchSource:0}: Error finding container f8730485056934cb22dde1b9afa579bbd90e7de5410a9ba540cb12645017663b: Status 404 returned error can't find the container with id f8730485056934cb22dde1b9afa579bbd90e7de5410a9ba540cb12645017663b Jul 14 22:10:21.771948 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:21.771916 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:21.990021 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:21.989942 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:22.099431 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:22.099382 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:22.292930 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:22.292858 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:22.445826 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:22.445783 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t66mm" event={"ID":"c770b0db-7fa6-4090-826a-412b9d542753","Type":"ContainerStarted","Data":"6ac78888ebd64ae9f5710202860a4bc2a6b956f0ebc881676e007f4985b8be39"} Jul 14 22:10:22.446268 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:22.445848 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t66mm" event={"ID":"c770b0db-7fa6-4090-826a-412b9d542753","Type":"ContainerStarted","Data":"6ccd537927791d1b038bdd0586ce6b415a26cbf8314aabaf2ff06a04bf61b594"} Jul 14 22:10:22.446268 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:22.445881 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t66mm" event={"ID":"c770b0db-7fa6-4090-826a-412b9d542753","Type":"ContainerStarted","Data":"f8730485056934cb22dde1b9afa579bbd90e7de5410a9ba540cb12645017663b"} Jul 14 22:10:23.453420 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:23.452375 2550 generic.go:356] "Generic (PLEG): container finished" podID="1d845a84-9484-41d0-9373-27ede38c8f5f" containerID="3600313a3c822d930a358d282ab280dfef2498dfab8dea896571f1e3ef72e52a" exitCode=0 Jul 14 22:10:23.453420 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:23.452516 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9bglr" event={"ID":"1d845a84-9484-41d0-9373-27ede38c8f5f","Type":"ContainerDied","Data":"3600313a3c822d930a358d282ab280dfef2498dfab8dea896571f1e3ef72e52a"} Jul 14 22:10:23.456242 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:23.456060 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-462d8" event={"ID":"af11146f-0b62-492b-b73a-440e2c69a95b","Type":"ContainerStarted","Data":"4a9e8698c9da8e54ceb2677bf6ee00fd0a0e2329160458e0c4ed7cff2b959aac"} Jul 14 22:10:23.456242 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:23.456111 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-462d8" event={"ID":"af11146f-0b62-492b-b73a-440e2c69a95b","Type":"ContainerStarted","Data":"401a35fdf34c7c33c50f50e0b5d8773e5581f23cd225b2c97dc5eed8dce422d1"} Jul 14 22:10:23.456242 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:23.456139 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-462d8" event={"ID":"af11146f-0b62-492b-b73a-440e2c69a95b","Type":"ContainerStarted","Data":"7c28b27a47744b6459ac77dcf04ea75451a9d44fd16099712641deaa0902b708"} Jul 14 22:10:23.499483 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:23.499429 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-462d8" podStartSLOduration=1.6094923300000001 podStartE2EDuration="3.49941185s" totalImagesPullingTime="1.88991952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:10:23.499018234 +0000 UTC m=+53.858164672" watchObservedRunningTime="2026-07-14 22:10:23.49941185 +0000 UTC m=+53.858558288" Jul 14 22:10:23.609355 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:23.609267 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:23.711121 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:23.711094 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:23.917302 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:23.917263 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:24.153011 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:24.152983 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:24.462007 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:24.461968 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9bglr" event={"ID":"1d845a84-9484-41d0-9373-27ede38c8f5f","Type":"ContainerStarted","Data":"04955625acc774b20ade0ad7102df02671ec5289c8c55a41d851346c2a7813c3"} Jul 14 22:10:24.462469 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:24.462029 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9bglr" event={"ID":"1d845a84-9484-41d0-9373-27ede38c8f5f","Type":"ContainerStarted","Data":"b74cc7bb8dba08fb89ccdc0bf22bea66be68322817fa8f5b9e2f502fe20b3d17"} Jul 14 22:10:24.464198 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:24.464169 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t66mm" event={"ID":"c770b0db-7fa6-4090-826a-412b9d542753","Type":"ContainerStarted","Data":"3502fd87f9e2c26f96a61f4c3d934fa612ffd62cf300d58d665b4c56fb4b728a"} Jul 14 22:10:24.484986 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:24.484942 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9bglr" podStartSLOduration=2.343222002 podStartE2EDuration="4.484924959s" totalImagesPullingTime="2.010410677s" totalInitContainerRuntime="131.29228ms" isStatefulPod=true podCreationTimestamp="2026-07-14 22:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:10:24.483213106 +0000 UTC m=+54.842359547" watchObservedRunningTime="2026-07-14 22:10:24.484924959 +0000 UTC m=+54.844071398" Jul 14 22:10:24.505402 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:24.505362 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-t66mm" podStartSLOduration=2.857233072 podStartE2EDuration="4.505346633s" totalImagesPullingTime="1.648113561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:10:24.50465998 +0000 UTC m=+54.863806429" watchObservedRunningTime="2026-07-14 22:10:24.505346633 +0000 UTC m=+54.864493069" Jul 14 22:10:25.240787 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.240754 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6977c64d5f-44v55"] Jul 14 22:10:25.243008 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.242987 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6977c64d5f-44v55" Jul 14 22:10:25.247107 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.247085 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 14 22:10:25.247207 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.247114 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 14 22:10:25.247207 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.247123 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 14 22:10:25.247324 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.247216 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 14 22:10:25.247324 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.247267 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-306delc9oovh0\"" Jul 14 22:10:25.247500 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.247486 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-7tz5f\"" Jul 14 22:10:25.254982 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.254961 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6977c64d5f-44v55"] Jul 14 22:10:25.331974 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.331944 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/079986c7-2809-4036-9fe3-3587d5902d62-client-ca-bundle\") pod \"metrics-server-6977c64d5f-44v55\" (UID: \"079986c7-2809-4036-9fe3-3587d5902d62\") " pod="openshift-monitoring/metrics-server-6977c64d5f-44v55" Jul 14 22:10:25.331974 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.331977 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/079986c7-2809-4036-9fe3-3587d5902d62-metrics-server-audit-profiles\") pod \"metrics-server-6977c64d5f-44v55\" (UID: \"079986c7-2809-4036-9fe3-3587d5902d62\") " pod="openshift-monitoring/metrics-server-6977c64d5f-44v55" Jul 14 22:10:25.332145 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.332005 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/079986c7-2809-4036-9fe3-3587d5902d62-audit-log\") pod \"metrics-server-6977c64d5f-44v55\" (UID: \"079986c7-2809-4036-9fe3-3587d5902d62\") " pod="openshift-monitoring/metrics-server-6977c64d5f-44v55" Jul 14 22:10:25.332145 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.332093 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/079986c7-2809-4036-9fe3-3587d5902d62-secret-metrics-server-client-certs\") pod \"metrics-server-6977c64d5f-44v55\" (UID: \"079986c7-2809-4036-9fe3-3587d5902d62\") " pod="openshift-monitoring/metrics-server-6977c64d5f-44v55" Jul 14 22:10:25.332145 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.332128 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/079986c7-2809-4036-9fe3-3587d5902d62-secret-metrics-server-tls\") pod \"metrics-server-6977c64d5f-44v55\" (UID: \"079986c7-2809-4036-9fe3-3587d5902d62\") " pod="openshift-monitoring/metrics-server-6977c64d5f-44v55" Jul 14 22:10:25.332257 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.332149 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/079986c7-2809-4036-9fe3-3587d5902d62-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6977c64d5f-44v55\" (UID: \"079986c7-2809-4036-9fe3-3587d5902d62\") " pod="openshift-monitoring/metrics-server-6977c64d5f-44v55" Jul 14 22:10:25.332257 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.332189 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w965\" (UniqueName: \"kubernetes.io/projected/079986c7-2809-4036-9fe3-3587d5902d62-kube-api-access-9w965\") pod \"metrics-server-6977c64d5f-44v55\" (UID: \"079986c7-2809-4036-9fe3-3587d5902d62\") " pod="openshift-monitoring/metrics-server-6977c64d5f-44v55" Jul 14 22:10:25.391501 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.391465 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:25.394732 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.394707 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-r856d"] Jul 14 22:10:25.397003 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.396984 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r856d" Jul 14 22:10:25.399675 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.399651 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 14 22:10:25.399782 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.399700 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-c5d24\"" Jul 14 22:10:25.408054 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.408032 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-r856d"] Jul 14 22:10:25.432794 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.432761 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/079986c7-2809-4036-9fe3-3587d5902d62-client-ca-bundle\") pod \"metrics-server-6977c64d5f-44v55\" (UID: \"079986c7-2809-4036-9fe3-3587d5902d62\") " pod="openshift-monitoring/metrics-server-6977c64d5f-44v55" Jul 14 22:10:25.432794 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.432810 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/079986c7-2809-4036-9fe3-3587d5902d62-metrics-server-audit-profiles\") pod \"metrics-server-6977c64d5f-44v55\" (UID: \"079986c7-2809-4036-9fe3-3587d5902d62\") " pod="openshift-monitoring/metrics-server-6977c64d5f-44v55" Jul 14 22:10:25.432794 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.432839 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hgcl\" (UniqueName: \"kubernetes.io/projected/5cc10e6c-47f8-4b27-9dd4-0554cc6ee905-kube-api-access-8hgcl\") pod \"monitoring-plugin-5f68df7f58-r856d\" (UID: \"5cc10e6c-47f8-4b27-9dd4-0554cc6ee905\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r856d" Jul 14 22:10:25.432794 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.432879 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/079986c7-2809-4036-9fe3-3587d5902d62-audit-log\") pod \"metrics-server-6977c64d5f-44v55\" (UID: \"079986c7-2809-4036-9fe3-3587d5902d62\") " pod="openshift-monitoring/metrics-server-6977c64d5f-44v55" Jul 14 22:10:25.432794 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.432905 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5cc10e6c-47f8-4b27-9dd4-0554cc6ee905-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-r856d\" (UID: \"5cc10e6c-47f8-4b27-9dd4-0554cc6ee905\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r856d" Jul 14 22:10:25.432794 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.432948 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/079986c7-2809-4036-9fe3-3587d5902d62-secret-metrics-server-client-certs\") pod \"metrics-server-6977c64d5f-44v55\" (UID: \"079986c7-2809-4036-9fe3-3587d5902d62\") " pod="openshift-monitoring/metrics-server-6977c64d5f-44v55" Jul 14 22:10:25.432794 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.432976 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/079986c7-2809-4036-9fe3-3587d5902d62-secret-metrics-server-tls\") pod \"metrics-server-6977c64d5f-44v55\" (UID: \"079986c7-2809-4036-9fe3-3587d5902d62\") " pod="openshift-monitoring/metrics-server-6977c64d5f-44v55" Jul 14 22:10:25.432794 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.433002 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/079986c7-2809-4036-9fe3-3587d5902d62-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6977c64d5f-44v55\" (UID: \"079986c7-2809-4036-9fe3-3587d5902d62\") " pod="openshift-monitoring/metrics-server-6977c64d5f-44v55" Jul 14 22:10:25.432794 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.433027 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w965\" (UniqueName: \"kubernetes.io/projected/079986c7-2809-4036-9fe3-3587d5902d62-kube-api-access-9w965\") pod \"metrics-server-6977c64d5f-44v55\" (UID: \"079986c7-2809-4036-9fe3-3587d5902d62\") " pod="openshift-monitoring/metrics-server-6977c64d5f-44v55" Jul 14 22:10:25.433925 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.433776 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/079986c7-2809-4036-9fe3-3587d5902d62-audit-log\") pod \"metrics-server-6977c64d5f-44v55\" (UID: \"079986c7-2809-4036-9fe3-3587d5902d62\") " pod="openshift-monitoring/metrics-server-6977c64d5f-44v55" Jul 14 22:10:25.434154 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.434126 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/079986c7-2809-4036-9fe3-3587d5902d62-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6977c64d5f-44v55\" (UID: \"079986c7-2809-4036-9fe3-3587d5902d62\") " pod="openshift-monitoring/metrics-server-6977c64d5f-44v55" Jul 14 22:10:25.434514 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.434469 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/079986c7-2809-4036-9fe3-3587d5902d62-metrics-server-audit-profiles\") pod \"metrics-server-6977c64d5f-44v55\" (UID: \"079986c7-2809-4036-9fe3-3587d5902d62\") " pod="openshift-monitoring/metrics-server-6977c64d5f-44v55" Jul 14 22:10:25.436493 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.436457 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/079986c7-2809-4036-9fe3-3587d5902d62-client-ca-bundle\") pod \"metrics-server-6977c64d5f-44v55\" (UID: \"079986c7-2809-4036-9fe3-3587d5902d62\") " pod="openshift-monitoring/metrics-server-6977c64d5f-44v55" Jul 14 22:10:25.436493 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.436464 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/079986c7-2809-4036-9fe3-3587d5902d62-secret-metrics-server-client-certs\") pod \"metrics-server-6977c64d5f-44v55\" (UID: \"079986c7-2809-4036-9fe3-3587d5902d62\") " pod="openshift-monitoring/metrics-server-6977c64d5f-44v55" Jul 14 22:10:25.436664 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.436625 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/079986c7-2809-4036-9fe3-3587d5902d62-secret-metrics-server-tls\") pod \"metrics-server-6977c64d5f-44v55\" (UID: \"079986c7-2809-4036-9fe3-3587d5902d62\") " pod="openshift-monitoring/metrics-server-6977c64d5f-44v55" Jul 14 22:10:25.441623 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.441585 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w965\" (UniqueName: \"kubernetes.io/projected/079986c7-2809-4036-9fe3-3587d5902d62-kube-api-access-9w965\") pod \"metrics-server-6977c64d5f-44v55\" (UID: \"079986c7-2809-4036-9fe3-3587d5902d62\") " pod="openshift-monitoring/metrics-server-6977c64d5f-44v55" Jul 14 22:10:25.488949 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.488916 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:25.534278 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.534188 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5cc10e6c-47f8-4b27-9dd4-0554cc6ee905-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-r856d\" (UID: \"5cc10e6c-47f8-4b27-9dd4-0554cc6ee905\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r856d" Jul 14 22:10:25.534921 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.534572 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hgcl\" (UniqueName: \"kubernetes.io/projected/5cc10e6c-47f8-4b27-9dd4-0554cc6ee905-kube-api-access-8hgcl\") pod \"monitoring-plugin-5f68df7f58-r856d\" (UID: \"5cc10e6c-47f8-4b27-9dd4-0554cc6ee905\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r856d" Jul 14 22:10:25.536823 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.536801 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5cc10e6c-47f8-4b27-9dd4-0554cc6ee905-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-r856d\" (UID: \"5cc10e6c-47f8-4b27-9dd4-0554cc6ee905\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r856d" Jul 14 22:10:25.543438 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.543414 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hgcl\" (UniqueName: \"kubernetes.io/projected/5cc10e6c-47f8-4b27-9dd4-0554cc6ee905-kube-api-access-8hgcl\") pod \"monitoring-plugin-5f68df7f58-r856d\" (UID: \"5cc10e6c-47f8-4b27-9dd4-0554cc6ee905\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r856d" Jul 14 22:10:25.553968 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.553943 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6977c64d5f-44v55" Jul 14 22:10:25.588196 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.588164 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:25.690307 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.690275 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6977c64d5f-44v55"] Jul 14 22:10:25.693403 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:10:25.693377 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod079986c7_2809_4036_9fe3_3587d5902d62.slice/crio-07a08337a6aa692d96ecc469981f872b3bdf41aadf23c07c68edf5d2373d56b3 WatchSource:0}: Error finding container 07a08337a6aa692d96ecc469981f872b3bdf41aadf23c07c68edf5d2373d56b3: Status 404 returned error can't find the container with id 07a08337a6aa692d96ecc469981f872b3bdf41aadf23c07c68edf5d2373d56b3 Jul 14 22:10:25.694084 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.694062 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:25.708804 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.708783 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r856d" Jul 14 22:10:25.847291 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:25.847218 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-r856d"] Jul 14 22:10:25.851027 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:10:25.850999 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc10e6c_47f8_4b27_9dd4_0554cc6ee905.slice/crio-0b051f4b81101a161cf95e0683af8e9e3b14233eae5db8b524dce684e8692e62 WatchSource:0}: Error finding container 0b051f4b81101a161cf95e0683af8e9e3b14233eae5db8b524dce684e8692e62: Status 404 returned error can't find the container with id 0b051f4b81101a161cf95e0683af8e9e3b14233eae5db8b524dce684e8692e62 Jul 14 22:10:26.422685 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:26.422653 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dktnm" Jul 14 22:10:26.473615 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:26.473580 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6977c64d5f-44v55" event={"ID":"079986c7-2809-4036-9fe3-3587d5902d62","Type":"ContainerStarted","Data":"07a08337a6aa692d96ecc469981f872b3bdf41aadf23c07c68edf5d2373d56b3"} Jul 14 22:10:26.474981 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:26.474950 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r856d" event={"ID":"5cc10e6c-47f8-4b27-9dd4-0554cc6ee905","Type":"ContainerStarted","Data":"0b051f4b81101a161cf95e0683af8e9e3b14233eae5db8b524dce684e8692e62"} Jul 14 22:10:26.950021 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:26.949986 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:27.421815 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:27.421790 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:27.517199 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:27.517168 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:27.575748 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:27.575720 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:27.844764 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:27.844678 2550 patch_prober.go:28] interesting pod/image-registry-794ffd4775-6nwdp container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 14 22:10:27.844764 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:27.844733 2550 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-794ffd4775-6nwdp" podUID="f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 22:10:28.421438 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:28.421403 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:28.488007 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:28.487978 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:29.008102 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.008072 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-865959987d-tqnzd"] Jul 14 22:10:29.044217 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.044186 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-865959987d-tqnzd"] Jul 14 22:10:29.044369 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.044290 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-865959987d-tqnzd" Jul 14 22:10:29.047469 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.047437 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 14 22:10:29.047607 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.047532 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 14 22:10:29.047678 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.047650 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 14 22:10:29.047769 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.047754 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bgx8h\"" Jul 14 22:10:29.047827 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.047807 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 14 22:10:29.048862 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.048831 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 14 22:10:29.066308 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.066288 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 14 22:10:29.168196 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.168164 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/90746d41-fcbd-4201-aa27-5db8b1e722fe-console-serving-cert\") pod \"console-865959987d-tqnzd\" (UID: \"90746d41-fcbd-4201-aa27-5db8b1e722fe\") " pod="openshift-console/console-865959987d-tqnzd" Jul 14 22:10:29.168347 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.168225 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/90746d41-fcbd-4201-aa27-5db8b1e722fe-service-ca\") pod \"console-865959987d-tqnzd\" (UID: \"90746d41-fcbd-4201-aa27-5db8b1e722fe\") " pod="openshift-console/console-865959987d-tqnzd" Jul 14 22:10:29.168347 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.168296 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/90746d41-fcbd-4201-aa27-5db8b1e722fe-console-oauth-config\") pod \"console-865959987d-tqnzd\" (UID: \"90746d41-fcbd-4201-aa27-5db8b1e722fe\") " pod="openshift-console/console-865959987d-tqnzd" Jul 14 22:10:29.168347 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.168326 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfr6d\" (UniqueName: \"kubernetes.io/projected/90746d41-fcbd-4201-aa27-5db8b1e722fe-kube-api-access-gfr6d\") pod \"console-865959987d-tqnzd\" (UID: \"90746d41-fcbd-4201-aa27-5db8b1e722fe\") " pod="openshift-console/console-865959987d-tqnzd" Jul 14 22:10:29.168518 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.168374 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/90746d41-fcbd-4201-aa27-5db8b1e722fe-console-config\") pod \"console-865959987d-tqnzd\" (UID: \"90746d41-fcbd-4201-aa27-5db8b1e722fe\") " pod="openshift-console/console-865959987d-tqnzd" Jul 14 22:10:29.168518 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.168446 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/90746d41-fcbd-4201-aa27-5db8b1e722fe-oauth-serving-cert\") pod \"console-865959987d-tqnzd\" (UID: \"90746d41-fcbd-4201-aa27-5db8b1e722fe\") " pod="openshift-console/console-865959987d-tqnzd" Jul 14 22:10:29.168518 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.168499 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90746d41-fcbd-4201-aa27-5db8b1e722fe-trusted-ca-bundle\") pod \"console-865959987d-tqnzd\" (UID: \"90746d41-fcbd-4201-aa27-5db8b1e722fe\") " pod="openshift-console/console-865959987d-tqnzd" Jul 14 22:10:29.269153 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.269076 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/90746d41-fcbd-4201-aa27-5db8b1e722fe-console-serving-cert\") pod \"console-865959987d-tqnzd\" (UID: \"90746d41-fcbd-4201-aa27-5db8b1e722fe\") " pod="openshift-console/console-865959987d-tqnzd" Jul 14 22:10:29.269153 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.269138 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/90746d41-fcbd-4201-aa27-5db8b1e722fe-service-ca\") pod \"console-865959987d-tqnzd\" (UID: \"90746d41-fcbd-4201-aa27-5db8b1e722fe\") " pod="openshift-console/console-865959987d-tqnzd" Jul 14 22:10:29.269365 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.269190 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/90746d41-fcbd-4201-aa27-5db8b1e722fe-console-oauth-config\") pod \"console-865959987d-tqnzd\" (UID: \"90746d41-fcbd-4201-aa27-5db8b1e722fe\") " pod="openshift-console/console-865959987d-tqnzd" Jul 14 22:10:29.269365 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.269217 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfr6d\" (UniqueName: \"kubernetes.io/projected/90746d41-fcbd-4201-aa27-5db8b1e722fe-kube-api-access-gfr6d\") pod \"console-865959987d-tqnzd\" (UID: \"90746d41-fcbd-4201-aa27-5db8b1e722fe\") " pod="openshift-console/console-865959987d-tqnzd" Jul 14 22:10:29.269365 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.269273 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/90746d41-fcbd-4201-aa27-5db8b1e722fe-console-config\") pod \"console-865959987d-tqnzd\" (UID: \"90746d41-fcbd-4201-aa27-5db8b1e722fe\") " pod="openshift-console/console-865959987d-tqnzd" Jul 14 22:10:29.269365 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.269303 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/90746d41-fcbd-4201-aa27-5db8b1e722fe-oauth-serving-cert\") pod \"console-865959987d-tqnzd\" (UID: \"90746d41-fcbd-4201-aa27-5db8b1e722fe\") " pod="openshift-console/console-865959987d-tqnzd" Jul 14 22:10:29.269365 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.269336 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90746d41-fcbd-4201-aa27-5db8b1e722fe-trusted-ca-bundle\") pod \"console-865959987d-tqnzd\" (UID: \"90746d41-fcbd-4201-aa27-5db8b1e722fe\") " pod="openshift-console/console-865959987d-tqnzd" Jul 14 22:10:29.270171 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.270057 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/90746d41-fcbd-4201-aa27-5db8b1e722fe-service-ca\") pod \"console-865959987d-tqnzd\" (UID: \"90746d41-fcbd-4201-aa27-5db8b1e722fe\") " pod="openshift-console/console-865959987d-tqnzd" Jul 14 22:10:29.270171 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.270128 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/90746d41-fcbd-4201-aa27-5db8b1e722fe-oauth-serving-cert\") pod \"console-865959987d-tqnzd\" (UID: \"90746d41-fcbd-4201-aa27-5db8b1e722fe\") " pod="openshift-console/console-865959987d-tqnzd" Jul 14 22:10:29.270338 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.270188 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/90746d41-fcbd-4201-aa27-5db8b1e722fe-console-config\") pod \"console-865959987d-tqnzd\" (UID: \"90746d41-fcbd-4201-aa27-5db8b1e722fe\") " pod="openshift-console/console-865959987d-tqnzd" Jul 14 22:10:29.270589 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.270572 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90746d41-fcbd-4201-aa27-5db8b1e722fe-trusted-ca-bundle\") pod \"console-865959987d-tqnzd\" (UID: \"90746d41-fcbd-4201-aa27-5db8b1e722fe\") " pod="openshift-console/console-865959987d-tqnzd" Jul 14 22:10:29.271877 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.271855 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/90746d41-fcbd-4201-aa27-5db8b1e722fe-console-oauth-config\") pod \"console-865959987d-tqnzd\" (UID: \"90746d41-fcbd-4201-aa27-5db8b1e722fe\") " pod="openshift-console/console-865959987d-tqnzd" Jul 14 22:10:29.271969 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.271894 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/90746d41-fcbd-4201-aa27-5db8b1e722fe-console-serving-cert\") pod \"console-865959987d-tqnzd\" (UID: \"90746d41-fcbd-4201-aa27-5db8b1e722fe\") " pod="openshift-console/console-865959987d-tqnzd" Jul 14 22:10:29.278961 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.278938 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfr6d\" (UniqueName: \"kubernetes.io/projected/90746d41-fcbd-4201-aa27-5db8b1e722fe-kube-api-access-gfr6d\") pod \"console-865959987d-tqnzd\" (UID: \"90746d41-fcbd-4201-aa27-5db8b1e722fe\") " pod="openshift-console/console-865959987d-tqnzd" Jul 14 22:10:29.359298 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.359261 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-865959987d-tqnzd" Jul 14 22:10:29.391589 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.391540 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-794ffd4775-6nwdp" Jul 14 22:10:29.538594 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.538495 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:29.637691 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:29.637654 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:30.363461 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:30.363424 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zj4sb" Jul 14 22:10:30.507952 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:30.507914 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:30.606599 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:30.606534 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:31.811662 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:31.811628 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:31.885279 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:31.885245 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:32.217987 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:32.217957 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:32.319282 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:32.319247 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:33.043804 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:33.043773 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-865959987d-tqnzd"] Jul 14 22:10:33.132147 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:10:33.132120 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90746d41_fcbd_4201_aa27_5db8b1e722fe.slice/crio-ca8c77405d69c0346a0e47b480e195a70b03656521a5b6a45a3fe54097db2af6 WatchSource:0}: Error finding container ca8c77405d69c0346a0e47b480e195a70b03656521a5b6a45a3fe54097db2af6: Status 404 returned error can't find the container with id ca8c77405d69c0346a0e47b480e195a70b03656521a5b6a45a3fe54097db2af6 Jul 14 22:10:33.502835 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:33.502755 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-865959987d-tqnzd" event={"ID":"90746d41-fcbd-4201-aa27-5db8b1e722fe","Type":"ContainerStarted","Data":"ca8c77405d69c0346a0e47b480e195a70b03656521a5b6a45a3fe54097db2af6"} Jul 14 22:10:33.504625 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:33.504494 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-jghjl" event={"ID":"1337dea6-26b7-4707-a955-1fc73f898523","Type":"ContainerStarted","Data":"46f4e8953dcfa2eb8374704cf0a85fe69f8397b306330f58f75cbcd0e0d34e05"} Jul 14 22:10:33.505100 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:33.505082 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-jghjl" Jul 14 22:10:33.506885 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:33.506839 2550 patch_prober.go:28] interesting pod/downloads-68c9cbb497-jghjl container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.10:8080/\": dial tcp 10.134.0.10:8080: connect: connection refused" start-of-body= Jul 14 22:10:33.507001 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:33.506890 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-68c9cbb497-jghjl" podUID="1337dea6-26b7-4707-a955-1fc73f898523" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.10:8080/\": dial tcp 10.134.0.10:8080: connect: connection refused" Jul 14 22:10:33.528566 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:33.527939 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-jghjl" podStartSLOduration=1.646162553 podStartE2EDuration="19.527919819s" totalImagesPullingTime="17.881757266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:10:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:10:33.52594486 +0000 UTC m=+63.885091297" watchObservedRunningTime="2026-07-14 22:10:33.527919819 +0000 UTC m=+63.887066258" Jul 14 22:10:34.154390 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:34.154318 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:34.227793 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:34.227731 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:34.525666 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:34.525587 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-jghjl" Jul 14 22:10:34.946747 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:34.946633 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:35.047841 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:35.047709 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:35.209313 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:35.209215 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:35.309714 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:35.309677 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:35.413819 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:35.413757 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:35.517630 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:35.517244 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r856d" event={"ID":"5cc10e6c-47f8-4b27-9dd4-0554cc6ee905","Type":"ContainerStarted","Data":"f515eafca12092eea07027ba2b7d93779bcba01bc38065c5e3ec5bbe778bb10c"} Jul 14 22:10:35.518253 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:35.518212 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r856d" Jul 14 22:10:35.520529 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:35.520474 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6977c64d5f-44v55" event={"ID":"079986c7-2809-4036-9fe3-3587d5902d62","Type":"ContainerStarted","Data":"6dd292d9a5b520e6d8057363294a47b8a57186b75a3f5851776a0c47c5e1bd0a"} Jul 14 22:10:35.525219 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:35.525190 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r856d" Jul 14 22:10:35.562574 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:35.562129 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r856d" podStartSLOduration=1.187207142 podStartE2EDuration="10.562106197s" totalImagesPullingTime="9.374899055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:10:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:10:35.540219358 +0000 UTC m=+65.899365800" watchObservedRunningTime="2026-07-14 22:10:35.562106197 +0000 UTC m=+65.921252636" Jul 14 22:10:35.585765 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:35.584669 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6977c64d5f-44v55" podStartSLOduration=1.047827825 podStartE2EDuration="10.584650497s" totalImagesPullingTime="9.536822672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:10:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:10:35.583818389 +0000 UTC m=+65.942964837" watchObservedRunningTime="2026-07-14 22:10:35.584650497 +0000 UTC m=+65.943796934" Jul 14 22:10:35.835244 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:35.835152 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce5ffbb6-99ac-4817-acd9-f6cd4577f724-original-pull-secret\") pod \"global-pull-secret-syncer-6g6bp\" (UID: \"ce5ffbb6-99ac-4817-acd9-f6cd4577f724\") " pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:10:35.835244 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:35.835217 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f9358e0-96a8-473f-896b-907e7129d452-metrics-certs\") pod \"network-metrics-daemon-55gw6\" (UID: \"7f9358e0-96a8-473f-896b-907e7129d452\") " pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:10:35.838505 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:35.838481 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 22:10:35.838779 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:35.838761 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 22:10:35.849326 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:35.849052 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce5ffbb6-99ac-4817-acd9-f6cd4577f724-original-pull-secret\") pod \"global-pull-secret-syncer-6g6bp\" (UID: \"ce5ffbb6-99ac-4817-acd9-f6cd4577f724\") " pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:10:35.849326 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:35.849269 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f9358e0-96a8-473f-896b-907e7129d452-metrics-certs\") pod \"network-metrics-daemon-55gw6\" (UID: \"7f9358e0-96a8-473f-896b-907e7129d452\") " pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:10:35.909390 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:35.909342 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nwz7d\"" Jul 14 22:10:35.916740 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:35.916714 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55gw6" Jul 14 22:10:35.933601 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:35.933572 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6g6bp" Jul 14 22:10:36.036885 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:36.036849 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrtk5\" (UniqueName: \"kubernetes.io/projected/a549c30a-9937-422b-9f9b-17d10c48ab25-kube-api-access-wrtk5\") pod \"network-check-target-6p45d\" (UID: \"a549c30a-9937-422b-9f9b-17d10c48ab25\") " pod="openshift-network-diagnostics/network-check-target-6p45d" Jul 14 22:10:36.040017 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:36.039979 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 14 22:10:36.050634 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:36.050608 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 14 22:10:36.062013 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:36.061954 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrtk5\" (UniqueName: \"kubernetes.io/projected/a549c30a-9937-422b-9f9b-17d10c48ab25-kube-api-access-wrtk5\") pod \"network-check-target-6p45d\" (UID: \"a549c30a-9937-422b-9f9b-17d10c48ab25\") " pod="openshift-network-diagnostics/network-check-target-6p45d" Jul 14 22:10:36.218643 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:36.218391 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vcczq\"" Jul 14 22:10:36.226529 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:36.226499 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6p45d" Jul 14 22:10:36.611950 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:36.611862 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:36.680527 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:36.680458 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:37.067289 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:37.067065 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6g6bp"] Jul 14 22:10:37.070587 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:10:37.070535 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce5ffbb6_99ac_4817_acd9_f6cd4577f724.slice/crio-9e60dcb50a0c00f678964fd24f5164916c1b637fc549af07df92415db9cd12a5 WatchSource:0}: Error finding container 9e60dcb50a0c00f678964fd24f5164916c1b637fc549af07df92415db9cd12a5: Status 404 returned error can't find the container with id 9e60dcb50a0c00f678964fd24f5164916c1b637fc549af07df92415db9cd12a5 Jul 14 22:10:37.095984 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:37.095959 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6p45d"] Jul 14 22:10:37.112141 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:37.112117 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-55gw6"] Jul 14 22:10:37.240395 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:10:37.240356 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda549c30a_9937_422b_9f9b_17d10c48ab25.slice/crio-339ac8074da42477dd2b4e0ab60835d0ec875700107cfcccc7257c939603e004 WatchSource:0}: Error finding container 339ac8074da42477dd2b4e0ab60835d0ec875700107cfcccc7257c939603e004: Status 404 returned error can't find the container with id 339ac8074da42477dd2b4e0ab60835d0ec875700107cfcccc7257c939603e004 Jul 14 22:10:37.240864 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:10:37.240637 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f9358e0_96a8_473f_896b_907e7129d452.slice/crio-c24077ec86b17839af72448ca3c142f4505f1166ddc33d29ac00bea53e39c17d WatchSource:0}: Error finding container c24077ec86b17839af72448ca3c142f4505f1166ddc33d29ac00bea53e39c17d: Status 404 returned error can't find the container with id c24077ec86b17839af72448ca3c142f4505f1166ddc33d29ac00bea53e39c17d Jul 14 22:10:37.475213 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:37.475177 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:37.533159 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:37.533073 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-55gw6" event={"ID":"7f9358e0-96a8-473f-896b-907e7129d452","Type":"ContainerStarted","Data":"c24077ec86b17839af72448ca3c142f4505f1166ddc33d29ac00bea53e39c17d"} Jul 14 22:10:37.534476 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:37.534439 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6p45d" event={"ID":"a549c30a-9937-422b-9f9b-17d10c48ab25","Type":"ContainerStarted","Data":"339ac8074da42477dd2b4e0ab60835d0ec875700107cfcccc7257c939603e004"} Jul 14 22:10:37.536290 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:37.536265 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-865959987d-tqnzd" event={"ID":"90746d41-fcbd-4201-aa27-5db8b1e722fe","Type":"ContainerStarted","Data":"83799fda9ee1e6c69fdba000b08b022858f45e762cc17fd0a82b7e21db5e2667"} Jul 14 22:10:37.537823 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:37.537761 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6g6bp" event={"ID":"ce5ffbb6-99ac-4817-acd9-f6cd4577f724","Type":"ContainerStarted","Data":"9e60dcb50a0c00f678964fd24f5164916c1b637fc549af07df92415db9cd12a5"} Jul 14 22:10:37.538388 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:37.538364 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:37.558961 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:37.558916 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-865959987d-tqnzd" podStartSLOduration=5.434126358 podStartE2EDuration="9.558898369s" totalImagesPullingTime="4.124772011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:10:37.557536484 +0000 UTC m=+67.916682957" watchObservedRunningTime="2026-07-14 22:10:37.558898369 +0000 UTC m=+67.918044805" Jul 14 22:10:39.359809 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:39.359639 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-865959987d-tqnzd" Jul 14 22:10:39.359809 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:39.359723 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-865959987d-tqnzd" Jul 14 22:10:39.368643 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:39.368408 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-865959987d-tqnzd" Jul 14 22:10:39.550272 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:39.550221 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-55gw6" event={"ID":"7f9358e0-96a8-473f-896b-907e7129d452","Type":"ContainerStarted","Data":"d7e859b46f09b3929f32e49de45cd108e16b53f740d6d40a3b99dc6752e235b6"} Jul 14 22:10:39.550272 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:39.550270 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-55gw6" event={"ID":"7f9358e0-96a8-473f-896b-907e7129d452","Type":"ContainerStarted","Data":"76b7c0a36acf4c823ebf9fbc2627ba9431168e7a0a50b381c108e846bfbd6948"} Jul 14 22:10:39.556272 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:39.556249 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-865959987d-tqnzd" Jul 14 22:10:39.571479 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:39.571207 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-55gw6" podStartSLOduration=68.144578398 podStartE2EDuration="1m9.571187195s" totalImagesPullingTime="1.426608797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:10:39.569315379 +0000 UTC m=+69.928461807" watchObservedRunningTime="2026-07-14 22:10:39.571187195 +0000 UTC m=+69.930333633" Jul 14 22:10:39.670653 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:39.670616 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:39.742808 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:39.742771 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:39.844660 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:39.844494 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:40.459018 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:40.458980 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:40.568806 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:40.568693 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:41.033191 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:41.033113 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:41.133801 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:41.133710 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:44.430541 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:44.430513 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:44.485253 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:44.485223 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:44.574295 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:44.574261 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6g6bp" event={"ID":"ce5ffbb6-99ac-4817-acd9-f6cd4577f724","Type":"ContainerStarted","Data":"a07814d5118f000c32f706c8cf20f3226b67268f13def6749d99f1aeb5ab79a4"} Jul 14 22:10:44.575652 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:44.575630 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6p45d" event={"ID":"a549c30a-9937-422b-9f9b-17d10c48ab25","Type":"ContainerStarted","Data":"8276b0dcb87603510166b85d8e4b8dc62fcc74a5ebb5ed77718b1085e4c1e784"} Jul 14 22:10:44.575796 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:44.575769 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6p45d" Jul 14 22:10:44.590891 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:44.590847 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6g6bp" podStartSLOduration=67.626201459 podStartE2EDuration="1m14.590835015s" totalImagesPullingTime="6.964633556s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:10:44.590321391 +0000 UTC m=+74.949467829" watchObservedRunningTime="2026-07-14 22:10:44.590835015 +0000 UTC m=+74.949981451" Jul 14 22:10:44.605940 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:44.605900 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6p45d" podStartSLOduration=67.835376061 podStartE2EDuration="1m14.605887644s" totalImagesPullingTime="6.770511583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:10:44.60492148 +0000 UTC m=+74.964067917" watchObservedRunningTime="2026-07-14 22:10:44.605887644 +0000 UTC m=+74.965034081" Jul 14 22:10:45.554876 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:45.554844 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6977c64d5f-44v55" Jul 14 22:10:45.555413 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:45.554939 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6977c64d5f-44v55" Jul 14 22:10:49.998353 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:49.998320 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:10:50.085159 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:10:50.085129 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:11:05.560517 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:05.560413 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6977c64d5f-44v55" Jul 14 22:11:05.564369 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:05.564345 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6977c64d5f-44v55" Jul 14 22:11:15.581217 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:15.581189 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6p45d" Jul 14 22:11:40.097352 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:40.097313 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:11:40.401056 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:40.400962 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:11:49.100966 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:49.100934 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-654ff486f-vsmql"] Jul 14 22:11:49.104114 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:49.104096 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-654ff486f-vsmql" Jul 14 22:11:49.119101 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:49.119079 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-654ff486f-vsmql"] Jul 14 22:11:49.186067 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:49.186042 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e382f7b4-6d31-482e-ac9d-392e614a2e49-console-oauth-config\") pod \"console-654ff486f-vsmql\" (UID: \"e382f7b4-6d31-482e-ac9d-392e614a2e49\") " pod="openshift-console/console-654ff486f-vsmql" Jul 14 22:11:49.186193 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:49.186084 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e382f7b4-6d31-482e-ac9d-392e614a2e49-console-config\") pod \"console-654ff486f-vsmql\" (UID: \"e382f7b4-6d31-482e-ac9d-392e614a2e49\") " pod="openshift-console/console-654ff486f-vsmql" Jul 14 22:11:49.186193 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:49.186109 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4j25\" (UniqueName: \"kubernetes.io/projected/e382f7b4-6d31-482e-ac9d-392e614a2e49-kube-api-access-w4j25\") pod \"console-654ff486f-vsmql\" (UID: \"e382f7b4-6d31-482e-ac9d-392e614a2e49\") " pod="openshift-console/console-654ff486f-vsmql" Jul 14 22:11:49.186274 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:49.186182 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e382f7b4-6d31-482e-ac9d-392e614a2e49-service-ca\") pod \"console-654ff486f-vsmql\" (UID: \"e382f7b4-6d31-482e-ac9d-392e614a2e49\") " pod="openshift-console/console-654ff486f-vsmql" Jul 14 22:11:49.186274 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:49.186230 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e382f7b4-6d31-482e-ac9d-392e614a2e49-console-serving-cert\") pod \"console-654ff486f-vsmql\" (UID: \"e382f7b4-6d31-482e-ac9d-392e614a2e49\") " pod="openshift-console/console-654ff486f-vsmql" Jul 14 22:11:49.186274 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:49.186255 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e382f7b4-6d31-482e-ac9d-392e614a2e49-oauth-serving-cert\") pod \"console-654ff486f-vsmql\" (UID: \"e382f7b4-6d31-482e-ac9d-392e614a2e49\") " pod="openshift-console/console-654ff486f-vsmql" Jul 14 22:11:49.186376 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:49.186277 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e382f7b4-6d31-482e-ac9d-392e614a2e49-trusted-ca-bundle\") pod \"console-654ff486f-vsmql\" (UID: \"e382f7b4-6d31-482e-ac9d-392e614a2e49\") " pod="openshift-console/console-654ff486f-vsmql" Jul 14 22:11:49.196812 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:49.196775 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:11:49.287340 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:49.287312 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e382f7b4-6d31-482e-ac9d-392e614a2e49-console-oauth-config\") pod \"console-654ff486f-vsmql\" (UID: \"e382f7b4-6d31-482e-ac9d-392e614a2e49\") " pod="openshift-console/console-654ff486f-vsmql" Jul 14 22:11:49.287463 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:49.287359 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e382f7b4-6d31-482e-ac9d-392e614a2e49-console-config\") pod \"console-654ff486f-vsmql\" (UID: \"e382f7b4-6d31-482e-ac9d-392e614a2e49\") " pod="openshift-console/console-654ff486f-vsmql" Jul 14 22:11:49.287463 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:49.287381 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4j25\" (UniqueName: \"kubernetes.io/projected/e382f7b4-6d31-482e-ac9d-392e614a2e49-kube-api-access-w4j25\") pod \"console-654ff486f-vsmql\" (UID: \"e382f7b4-6d31-482e-ac9d-392e614a2e49\") " pod="openshift-console/console-654ff486f-vsmql" Jul 14 22:11:49.287463 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:49.287430 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e382f7b4-6d31-482e-ac9d-392e614a2e49-service-ca\") pod \"console-654ff486f-vsmql\" (UID: \"e382f7b4-6d31-482e-ac9d-392e614a2e49\") " pod="openshift-console/console-654ff486f-vsmql" Jul 14 22:11:49.287463 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:49.287455 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e382f7b4-6d31-482e-ac9d-392e614a2e49-console-serving-cert\") pod \"console-654ff486f-vsmql\" (UID: \"e382f7b4-6d31-482e-ac9d-392e614a2e49\") " pod="openshift-console/console-654ff486f-vsmql" Jul 14 22:11:49.287723 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:49.287485 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e382f7b4-6d31-482e-ac9d-392e614a2e49-oauth-serving-cert\") pod \"console-654ff486f-vsmql\" (UID: \"e382f7b4-6d31-482e-ac9d-392e614a2e49\") " pod="openshift-console/console-654ff486f-vsmql" Jul 14 22:11:49.287723 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:49.287521 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e382f7b4-6d31-482e-ac9d-392e614a2e49-trusted-ca-bundle\") pod \"console-654ff486f-vsmql\" (UID: \"e382f7b4-6d31-482e-ac9d-392e614a2e49\") " pod="openshift-console/console-654ff486f-vsmql" Jul 14 22:11:49.288193 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:49.288167 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e382f7b4-6d31-482e-ac9d-392e614a2e49-service-ca\") pod \"console-654ff486f-vsmql\" (UID: \"e382f7b4-6d31-482e-ac9d-392e614a2e49\") " pod="openshift-console/console-654ff486f-vsmql" Jul 14 22:11:49.288277 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:49.288173 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e382f7b4-6d31-482e-ac9d-392e614a2e49-console-config\") pod \"console-654ff486f-vsmql\" (UID: \"e382f7b4-6d31-482e-ac9d-392e614a2e49\") " pod="openshift-console/console-654ff486f-vsmql" Jul 14 22:11:49.288404 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:49.288381 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e382f7b4-6d31-482e-ac9d-392e614a2e49-trusted-ca-bundle\") pod \"console-654ff486f-vsmql\" (UID: \"e382f7b4-6d31-482e-ac9d-392e614a2e49\") " pod="openshift-console/console-654ff486f-vsmql" Jul 14 22:11:49.288441 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:49.288404 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e382f7b4-6d31-482e-ac9d-392e614a2e49-oauth-serving-cert\") pod \"console-654ff486f-vsmql\" (UID: \"e382f7b4-6d31-482e-ac9d-392e614a2e49\") " pod="openshift-console/console-654ff486f-vsmql" Jul 14 22:11:49.289978 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:49.289950 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e382f7b4-6d31-482e-ac9d-392e614a2e49-console-oauth-config\") pod \"console-654ff486f-vsmql\" (UID: \"e382f7b4-6d31-482e-ac9d-392e614a2e49\") " pod="openshift-console/console-654ff486f-vsmql" Jul 14 22:11:49.290053 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:49.289985 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e382f7b4-6d31-482e-ac9d-392e614a2e49-console-serving-cert\") pod \"console-654ff486f-vsmql\" (UID: \"e382f7b4-6d31-482e-ac9d-392e614a2e49\") " pod="openshift-console/console-654ff486f-vsmql" Jul 14 22:11:49.295168 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:49.295139 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4j25\" (UniqueName: \"kubernetes.io/projected/e382f7b4-6d31-482e-ac9d-392e614a2e49-kube-api-access-w4j25\") pod \"console-654ff486f-vsmql\" (UID: \"e382f7b4-6d31-482e-ac9d-392e614a2e49\") " pod="openshift-console/console-654ff486f-vsmql" Jul 14 22:11:49.412901 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:49.412870 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-654ff486f-vsmql" Jul 14 22:11:49.539716 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:49.539686 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-654ff486f-vsmql"] Jul 14 22:11:49.544290 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:11:49.544266 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode382f7b4_6d31_482e_ac9d_392e614a2e49.slice/crio-1a6fb2cc991b78af0066c35de960e2ba3dec482e1eb7e226c0f97602eb935c1d WatchSource:0}: Error finding container 1a6fb2cc991b78af0066c35de960e2ba3dec482e1eb7e226c0f97602eb935c1d: Status 404 returned error can't find the container with id 1a6fb2cc991b78af0066c35de960e2ba3dec482e1eb7e226c0f97602eb935c1d Jul 14 22:11:49.808930 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:49.808834 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-654ff486f-vsmql" event={"ID":"e382f7b4-6d31-482e-ac9d-392e614a2e49","Type":"ContainerStarted","Data":"9bbb76b48907ac393a5d9550753256a73bd7420c24de8c922425de3fa1e07834"} Jul 14 22:11:49.808930 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:49.808890 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-654ff486f-vsmql" event={"ID":"e382f7b4-6d31-482e-ac9d-392e614a2e49","Type":"ContainerStarted","Data":"1a6fb2cc991b78af0066c35de960e2ba3dec482e1eb7e226c0f97602eb935c1d"} Jul 14 22:11:49.847418 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:49.847371 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-654ff486f-vsmql" podStartSLOduration=0.847354913 podStartE2EDuration="847.354913ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:11:49.846232798 +0000 UTC m=+140.205379261" watchObservedRunningTime="2026-07-14 22:11:49.847354913 +0000 UTC m=+140.206501350" Jul 14 22:11:59.413129 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:59.413094 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-654ff486f-vsmql" Jul 14 22:11:59.413701 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:59.413199 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-654ff486f-vsmql" Jul 14 22:11:59.417766 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:59.417747 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-654ff486f-vsmql" Jul 14 22:11:59.846317 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:59.846242 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-654ff486f-vsmql" Jul 14 22:11:59.898715 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:11:59.898688 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-865959987d-tqnzd"] Jul 14 22:12:00.042967 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:00.042940 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:12:00.146118 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:00.146036 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:12:00.246512 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:00.246481 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:12:02.132419 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:02.132387 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:12:21.663967 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:21.663892 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:12:21.736708 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:21.736676 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:12:24.919032 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:24.918990 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-865959987d-tqnzd" podUID="90746d41-fcbd-4201-aa27-5db8b1e722fe" containerName="console" containerID="cri-o://83799fda9ee1e6c69fdba000b08b022858f45e762cc17fd0a82b7e21db5e2667" gracePeriod=15 Jul 14 22:12:25.156350 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:25.156328 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-865959987d-tqnzd" Jul 14 22:12:25.346675 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:25.346591 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/90746d41-fcbd-4201-aa27-5db8b1e722fe-service-ca\") pod \"90746d41-fcbd-4201-aa27-5db8b1e722fe\" (UID: \"90746d41-fcbd-4201-aa27-5db8b1e722fe\") " Jul 14 22:12:25.346675 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:25.346634 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90746d41-fcbd-4201-aa27-5db8b1e722fe-trusted-ca-bundle\") pod \"90746d41-fcbd-4201-aa27-5db8b1e722fe\" (UID: \"90746d41-fcbd-4201-aa27-5db8b1e722fe\") " Jul 14 22:12:25.346675 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:25.346670 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/90746d41-fcbd-4201-aa27-5db8b1e722fe-console-oauth-config\") pod \"90746d41-fcbd-4201-aa27-5db8b1e722fe\" (UID: \"90746d41-fcbd-4201-aa27-5db8b1e722fe\") " Jul 14 22:12:25.346933 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:25.346688 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/90746d41-fcbd-4201-aa27-5db8b1e722fe-console-serving-cert\") pod \"90746d41-fcbd-4201-aa27-5db8b1e722fe\" (UID: \"90746d41-fcbd-4201-aa27-5db8b1e722fe\") " Jul 14 22:12:25.346933 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:25.346723 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/90746d41-fcbd-4201-aa27-5db8b1e722fe-oauth-serving-cert\") pod \"90746d41-fcbd-4201-aa27-5db8b1e722fe\" (UID: \"90746d41-fcbd-4201-aa27-5db8b1e722fe\") " Jul 14 22:12:25.346933 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:25.346762 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfr6d\" (UniqueName: \"kubernetes.io/projected/90746d41-fcbd-4201-aa27-5db8b1e722fe-kube-api-access-gfr6d\") pod \"90746d41-fcbd-4201-aa27-5db8b1e722fe\" (UID: \"90746d41-fcbd-4201-aa27-5db8b1e722fe\") " Jul 14 22:12:25.346933 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:25.346792 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/90746d41-fcbd-4201-aa27-5db8b1e722fe-console-config\") pod \"90746d41-fcbd-4201-aa27-5db8b1e722fe\" (UID: \"90746d41-fcbd-4201-aa27-5db8b1e722fe\") " Jul 14 22:12:25.347197 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:25.347158 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90746d41-fcbd-4201-aa27-5db8b1e722fe-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "90746d41-fcbd-4201-aa27-5db8b1e722fe" (UID: "90746d41-fcbd-4201-aa27-5db8b1e722fe"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:12:25.347288 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:25.347199 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90746d41-fcbd-4201-aa27-5db8b1e722fe-service-ca" (OuterVolumeSpecName: "service-ca") pod "90746d41-fcbd-4201-aa27-5db8b1e722fe" (UID: "90746d41-fcbd-4201-aa27-5db8b1e722fe"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:12:25.347365 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:25.347336 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90746d41-fcbd-4201-aa27-5db8b1e722fe-console-config" (OuterVolumeSpecName: "console-config") pod "90746d41-fcbd-4201-aa27-5db8b1e722fe" (UID: "90746d41-fcbd-4201-aa27-5db8b1e722fe"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:12:25.347421 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:25.347388 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90746d41-fcbd-4201-aa27-5db8b1e722fe-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "90746d41-fcbd-4201-aa27-5db8b1e722fe" (UID: "90746d41-fcbd-4201-aa27-5db8b1e722fe"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:12:25.349167 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:25.349134 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90746d41-fcbd-4201-aa27-5db8b1e722fe-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "90746d41-fcbd-4201-aa27-5db8b1e722fe" (UID: "90746d41-fcbd-4201-aa27-5db8b1e722fe"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 22:12:25.349293 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:25.349208 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90746d41-fcbd-4201-aa27-5db8b1e722fe-kube-api-access-gfr6d" (OuterVolumeSpecName: "kube-api-access-gfr6d") pod "90746d41-fcbd-4201-aa27-5db8b1e722fe" (UID: "90746d41-fcbd-4201-aa27-5db8b1e722fe"). InnerVolumeSpecName "kube-api-access-gfr6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:12:25.349293 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:25.349217 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90746d41-fcbd-4201-aa27-5db8b1e722fe-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "90746d41-fcbd-4201-aa27-5db8b1e722fe" (UID: "90746d41-fcbd-4201-aa27-5db8b1e722fe"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 22:12:25.448048 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:25.448016 2550 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/90746d41-fcbd-4201-aa27-5db8b1e722fe-console-serving-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:12:25.448048 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:25.448042 2550 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/90746d41-fcbd-4201-aa27-5db8b1e722fe-oauth-serving-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:12:25.448048 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:25.448051 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfr6d\" (UniqueName: \"kubernetes.io/projected/90746d41-fcbd-4201-aa27-5db8b1e722fe-kube-api-access-gfr6d\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:12:25.448255 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:25.448060 2550 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/90746d41-fcbd-4201-aa27-5db8b1e722fe-console-config\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:12:25.448255 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:25.448069 2550 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/90746d41-fcbd-4201-aa27-5db8b1e722fe-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:12:25.448255 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:25.448076 2550 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90746d41-fcbd-4201-aa27-5db8b1e722fe-trusted-ca-bundle\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:12:25.448255 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:25.448084 2550 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/90746d41-fcbd-4201-aa27-5db8b1e722fe-console-oauth-config\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:12:25.932247 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:25.932213 2550 generic.go:356] "Generic (PLEG): container finished" podID="90746d41-fcbd-4201-aa27-5db8b1e722fe" containerID="83799fda9ee1e6c69fdba000b08b022858f45e762cc17fd0a82b7e21db5e2667" exitCode=0 Jul 14 22:12:25.932712 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:25.932254 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-865959987d-tqnzd" Jul 14 22:12:25.932712 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:25.932282 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-865959987d-tqnzd" event={"ID":"90746d41-fcbd-4201-aa27-5db8b1e722fe","Type":"ContainerDied","Data":"83799fda9ee1e6c69fdba000b08b022858f45e762cc17fd0a82b7e21db5e2667"} Jul 14 22:12:25.932712 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:25.932329 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-865959987d-tqnzd" event={"ID":"90746d41-fcbd-4201-aa27-5db8b1e722fe","Type":"ContainerDied","Data":"ca8c77405d69c0346a0e47b480e195a70b03656521a5b6a45a3fe54097db2af6"} Jul 14 22:12:25.932712 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:25.932358 2550 scope.go:117] "RemoveContainer" containerID="83799fda9ee1e6c69fdba000b08b022858f45e762cc17fd0a82b7e21db5e2667" Jul 14 22:12:25.942489 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:25.942360 2550 scope.go:117] "RemoveContainer" containerID="83799fda9ee1e6c69fdba000b08b022858f45e762cc17fd0a82b7e21db5e2667" Jul 14 22:12:25.942682 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:12:25.942662 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83799fda9ee1e6c69fdba000b08b022858f45e762cc17fd0a82b7e21db5e2667\": container with ID starting with 83799fda9ee1e6c69fdba000b08b022858f45e762cc17fd0a82b7e21db5e2667 not found: ID does not exist" containerID="83799fda9ee1e6c69fdba000b08b022858f45e762cc17fd0a82b7e21db5e2667" Jul 14 22:12:25.942742 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:25.942689 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83799fda9ee1e6c69fdba000b08b022858f45e762cc17fd0a82b7e21db5e2667"} err="failed to get container status \"83799fda9ee1e6c69fdba000b08b022858f45e762cc17fd0a82b7e21db5e2667\": rpc error: code = NotFound desc = could not find container \"83799fda9ee1e6c69fdba000b08b022858f45e762cc17fd0a82b7e21db5e2667\": container with ID starting with 83799fda9ee1e6c69fdba000b08b022858f45e762cc17fd0a82b7e21db5e2667 not found: ID does not exist" Jul 14 22:12:25.957837 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:25.957810 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-865959987d-tqnzd"] Jul 14 22:12:25.965678 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:25.965658 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-865959987d-tqnzd"] Jul 14 22:12:26.194370 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:26.194281 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90746d41-fcbd-4201-aa27-5db8b1e722fe" path="/var/lib/kubelet/pods/90746d41-fcbd-4201-aa27-5db8b1e722fe/volumes" Jul 14 22:12:27.252864 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:27.252833 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:12:43.761342 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:43.761295 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:12:55.369405 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:55.369373 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:12:55.513691 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:55.513661 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:12:59.690919 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:59.690887 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f664d7795-jhqz7"] Jul 14 22:12:59.691278 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:59.691155 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90746d41-fcbd-4201-aa27-5db8b1e722fe" containerName="console" Jul 14 22:12:59.691278 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:59.691164 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="90746d41-fcbd-4201-aa27-5db8b1e722fe" containerName="console" Jul 14 22:12:59.691278 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:59.691210 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="90746d41-fcbd-4201-aa27-5db8b1e722fe" containerName="console" Jul 14 22:12:59.692965 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:59.692944 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f664d7795-jhqz7" Jul 14 22:12:59.706137 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:59.706115 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f664d7795-jhqz7"] Jul 14 22:12:59.743211 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:59.743185 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:12:59.797033 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:59.797010 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b386957-e8b9-415f-8097-66cb8a84e9ac-trusted-ca-bundle\") pod \"console-f664d7795-jhqz7\" (UID: \"3b386957-e8b9-415f-8097-66cb8a84e9ac\") " pod="openshift-console/console-f664d7795-jhqz7" Jul 14 22:12:59.797135 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:59.797040 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b386957-e8b9-415f-8097-66cb8a84e9ac-console-serving-cert\") pod \"console-f664d7795-jhqz7\" (UID: \"3b386957-e8b9-415f-8097-66cb8a84e9ac\") " pod="openshift-console/console-f664d7795-jhqz7" Jul 14 22:12:59.797135 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:59.797069 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b386957-e8b9-415f-8097-66cb8a84e9ac-oauth-serving-cert\") pod \"console-f664d7795-jhqz7\" (UID: \"3b386957-e8b9-415f-8097-66cb8a84e9ac\") " pod="openshift-console/console-f664d7795-jhqz7" Jul 14 22:12:59.797135 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:59.797107 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbjln\" (UniqueName: \"kubernetes.io/projected/3b386957-e8b9-415f-8097-66cb8a84e9ac-kube-api-access-rbjln\") pod \"console-f664d7795-jhqz7\" (UID: \"3b386957-e8b9-415f-8097-66cb8a84e9ac\") " pod="openshift-console/console-f664d7795-jhqz7" Jul 14 22:12:59.797253 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:59.797164 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b386957-e8b9-415f-8097-66cb8a84e9ac-console-config\") pod \"console-f664d7795-jhqz7\" (UID: \"3b386957-e8b9-415f-8097-66cb8a84e9ac\") " pod="openshift-console/console-f664d7795-jhqz7" Jul 14 22:12:59.797253 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:59.797184 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b386957-e8b9-415f-8097-66cb8a84e9ac-console-oauth-config\") pod \"console-f664d7795-jhqz7\" (UID: \"3b386957-e8b9-415f-8097-66cb8a84e9ac\") " pod="openshift-console/console-f664d7795-jhqz7" Jul 14 22:12:59.797253 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:59.797202 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b386957-e8b9-415f-8097-66cb8a84e9ac-service-ca\") pod \"console-f664d7795-jhqz7\" (UID: \"3b386957-e8b9-415f-8097-66cb8a84e9ac\") " pod="openshift-console/console-f664d7795-jhqz7" Jul 14 22:12:59.845473 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:59.845440 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:12:59.897787 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:59.897758 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b386957-e8b9-415f-8097-66cb8a84e9ac-console-config\") pod \"console-f664d7795-jhqz7\" (UID: \"3b386957-e8b9-415f-8097-66cb8a84e9ac\") " pod="openshift-console/console-f664d7795-jhqz7" Jul 14 22:12:59.897886 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:59.897790 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b386957-e8b9-415f-8097-66cb8a84e9ac-console-oauth-config\") pod \"console-f664d7795-jhqz7\" (UID: \"3b386957-e8b9-415f-8097-66cb8a84e9ac\") " pod="openshift-console/console-f664d7795-jhqz7" Jul 14 22:12:59.897886 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:59.897810 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b386957-e8b9-415f-8097-66cb8a84e9ac-service-ca\") pod \"console-f664d7795-jhqz7\" (UID: \"3b386957-e8b9-415f-8097-66cb8a84e9ac\") " pod="openshift-console/console-f664d7795-jhqz7" Jul 14 22:12:59.897886 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:59.897844 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b386957-e8b9-415f-8097-66cb8a84e9ac-trusted-ca-bundle\") pod \"console-f664d7795-jhqz7\" (UID: \"3b386957-e8b9-415f-8097-66cb8a84e9ac\") " pod="openshift-console/console-f664d7795-jhqz7" Jul 14 22:12:59.897886 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:59.897868 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b386957-e8b9-415f-8097-66cb8a84e9ac-console-serving-cert\") pod \"console-f664d7795-jhqz7\" (UID: \"3b386957-e8b9-415f-8097-66cb8a84e9ac\") " pod="openshift-console/console-f664d7795-jhqz7" Jul 14 22:12:59.898029 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:59.897899 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b386957-e8b9-415f-8097-66cb8a84e9ac-oauth-serving-cert\") pod \"console-f664d7795-jhqz7\" (UID: \"3b386957-e8b9-415f-8097-66cb8a84e9ac\") " pod="openshift-console/console-f664d7795-jhqz7" Jul 14 22:12:59.898029 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:59.897920 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbjln\" (UniqueName: \"kubernetes.io/projected/3b386957-e8b9-415f-8097-66cb8a84e9ac-kube-api-access-rbjln\") pod \"console-f664d7795-jhqz7\" (UID: \"3b386957-e8b9-415f-8097-66cb8a84e9ac\") " pod="openshift-console/console-f664d7795-jhqz7" Jul 14 22:12:59.898590 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:59.898538 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b386957-e8b9-415f-8097-66cb8a84e9ac-service-ca\") pod \"console-f664d7795-jhqz7\" (UID: \"3b386957-e8b9-415f-8097-66cb8a84e9ac\") " pod="openshift-console/console-f664d7795-jhqz7" Jul 14 22:12:59.898683 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:59.898540 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b386957-e8b9-415f-8097-66cb8a84e9ac-console-config\") pod \"console-f664d7795-jhqz7\" (UID: \"3b386957-e8b9-415f-8097-66cb8a84e9ac\") " pod="openshift-console/console-f664d7795-jhqz7" Jul 14 22:12:59.898742 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:59.898720 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b386957-e8b9-415f-8097-66cb8a84e9ac-oauth-serving-cert\") pod \"console-f664d7795-jhqz7\" (UID: \"3b386957-e8b9-415f-8097-66cb8a84e9ac\") " pod="openshift-console/console-f664d7795-jhqz7" Jul 14 22:12:59.898789 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:59.898750 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b386957-e8b9-415f-8097-66cb8a84e9ac-trusted-ca-bundle\") pod \"console-f664d7795-jhqz7\" (UID: \"3b386957-e8b9-415f-8097-66cb8a84e9ac\") " pod="openshift-console/console-f664d7795-jhqz7" Jul 14 22:12:59.900334 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:59.900317 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b386957-e8b9-415f-8097-66cb8a84e9ac-console-serving-cert\") pod \"console-f664d7795-jhqz7\" (UID: \"3b386957-e8b9-415f-8097-66cb8a84e9ac\") " pod="openshift-console/console-f664d7795-jhqz7" Jul 14 22:12:59.900439 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:59.900420 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b386957-e8b9-415f-8097-66cb8a84e9ac-console-oauth-config\") pod \"console-f664d7795-jhqz7\" (UID: \"3b386957-e8b9-415f-8097-66cb8a84e9ac\") " pod="openshift-console/console-f664d7795-jhqz7" Jul 14 22:12:59.906154 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:59.906135 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbjln\" (UniqueName: \"kubernetes.io/projected/3b386957-e8b9-415f-8097-66cb8a84e9ac-kube-api-access-rbjln\") pod \"console-f664d7795-jhqz7\" (UID: \"3b386957-e8b9-415f-8097-66cb8a84e9ac\") " pod="openshift-console/console-f664d7795-jhqz7" Jul 14 22:12:59.944267 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:12:59.944211 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:13:00.001667 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:00.001645 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f664d7795-jhqz7" Jul 14 22:13:00.129374 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:00.129345 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f664d7795-jhqz7"] Jul 14 22:13:00.132624 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:13:00.132598 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b386957_e8b9_415f_8097_66cb8a84e9ac.slice/crio-99708f8ce194f5292bd98260b322cbc0ea1ab267bac37d12ad7035bc0a2e7e3a WatchSource:0}: Error finding container 99708f8ce194f5292bd98260b322cbc0ea1ab267bac37d12ad7035bc0a2e7e3a: Status 404 returned error can't find the container with id 99708f8ce194f5292bd98260b322cbc0ea1ab267bac37d12ad7035bc0a2e7e3a Jul 14 22:13:00.498682 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:00.498657 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:13:00.561344 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:00.561314 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:13:01.055129 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:01.055093 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f664d7795-jhqz7" event={"ID":"3b386957-e8b9-415f-8097-66cb8a84e9ac","Type":"ContainerStarted","Data":"92082ebc7545e685b67a20796df255493fbea42b7a1fc41f3ac738ea44265f10"} Jul 14 22:13:01.055607 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:01.055174 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f664d7795-jhqz7" event={"ID":"3b386957-e8b9-415f-8097-66cb8a84e9ac","Type":"ContainerStarted","Data":"99708f8ce194f5292bd98260b322cbc0ea1ab267bac37d12ad7035bc0a2e7e3a"} Jul 14 22:13:01.077183 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:01.077132 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f664d7795-jhqz7" podStartSLOduration=2.077117032 podStartE2EDuration="2.077117032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:13:01.07627598 +0000 UTC m=+211.435422417" watchObservedRunningTime="2026-07-14 22:13:01.077117032 +0000 UTC m=+211.436263506" Jul 14 22:13:07.456470 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:07.456440 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:13:07.521202 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:07.521170 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:13:10.002257 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:10.002224 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f664d7795-jhqz7" Jul 14 22:13:10.002652 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:10.002278 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f664d7795-jhqz7" Jul 14 22:13:10.006780 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:10.006759 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f664d7795-jhqz7" Jul 14 22:13:10.091071 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:10.091045 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f664d7795-jhqz7" Jul 14 22:13:10.151329 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:10.151301 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-654ff486f-vsmql"] Jul 14 22:13:10.339466 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:10.339395 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:13:10.433910 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:10.433882 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:13:15.371520 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:15.371489 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:13:19.958009 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:19.957976 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:13:20.058421 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:20.058395 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:13:35.171443 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:35.171393 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-654ff486f-vsmql" podUID="e382f7b4-6d31-482e-ac9d-392e614a2e49" containerName="console" containerID="cri-o://9bbb76b48907ac393a5d9550753256a73bd7420c24de8c922425de3fa1e07834" gracePeriod=15 Jul 14 22:13:35.405479 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:35.405450 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-654ff486f-vsmql" Jul 14 22:13:35.569052 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:35.568978 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e382f7b4-6d31-482e-ac9d-392e614a2e49-oauth-serving-cert\") pod \"e382f7b4-6d31-482e-ac9d-392e614a2e49\" (UID: \"e382f7b4-6d31-482e-ac9d-392e614a2e49\") " Jul 14 22:13:35.569052 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:35.569027 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4j25\" (UniqueName: \"kubernetes.io/projected/e382f7b4-6d31-482e-ac9d-392e614a2e49-kube-api-access-w4j25\") pod \"e382f7b4-6d31-482e-ac9d-392e614a2e49\" (UID: \"e382f7b4-6d31-482e-ac9d-392e614a2e49\") " Jul 14 22:13:35.569241 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:35.569152 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e382f7b4-6d31-482e-ac9d-392e614a2e49-console-config\") pod \"e382f7b4-6d31-482e-ac9d-392e614a2e49\" (UID: \"e382f7b4-6d31-482e-ac9d-392e614a2e49\") " Jul 14 22:13:35.569241 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:35.569181 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e382f7b4-6d31-482e-ac9d-392e614a2e49-console-oauth-config\") pod \"e382f7b4-6d31-482e-ac9d-392e614a2e49\" (UID: \"e382f7b4-6d31-482e-ac9d-392e614a2e49\") " Jul 14 22:13:35.569241 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:35.569201 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e382f7b4-6d31-482e-ac9d-392e614a2e49-service-ca\") pod \"e382f7b4-6d31-482e-ac9d-392e614a2e49\" (UID: \"e382f7b4-6d31-482e-ac9d-392e614a2e49\") " Jul 14 22:13:35.569241 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:35.569216 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e382f7b4-6d31-482e-ac9d-392e614a2e49-console-serving-cert\") pod \"e382f7b4-6d31-482e-ac9d-392e614a2e49\" (UID: \"e382f7b4-6d31-482e-ac9d-392e614a2e49\") " Jul 14 22:13:35.569420 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:35.569248 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e382f7b4-6d31-482e-ac9d-392e614a2e49-trusted-ca-bundle\") pod \"e382f7b4-6d31-482e-ac9d-392e614a2e49\" (UID: \"e382f7b4-6d31-482e-ac9d-392e614a2e49\") " Jul 14 22:13:35.569420 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:35.569323 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e382f7b4-6d31-482e-ac9d-392e614a2e49-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e382f7b4-6d31-482e-ac9d-392e614a2e49" (UID: "e382f7b4-6d31-482e-ac9d-392e614a2e49"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:13:35.569626 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:35.569514 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e382f7b4-6d31-482e-ac9d-392e614a2e49-console-config" (OuterVolumeSpecName: "console-config") pod "e382f7b4-6d31-482e-ac9d-392e614a2e49" (UID: "e382f7b4-6d31-482e-ac9d-392e614a2e49"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:13:35.569626 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:35.569521 2550 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e382f7b4-6d31-482e-ac9d-392e614a2e49-oauth-serving-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:13:35.569735 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:35.569710 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e382f7b4-6d31-482e-ac9d-392e614a2e49-service-ca" (OuterVolumeSpecName: "service-ca") pod "e382f7b4-6d31-482e-ac9d-392e614a2e49" (UID: "e382f7b4-6d31-482e-ac9d-392e614a2e49"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:13:35.569838 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:35.569801 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e382f7b4-6d31-482e-ac9d-392e614a2e49-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e382f7b4-6d31-482e-ac9d-392e614a2e49" (UID: "e382f7b4-6d31-482e-ac9d-392e614a2e49"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:13:35.571190 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:35.571165 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e382f7b4-6d31-482e-ac9d-392e614a2e49-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e382f7b4-6d31-482e-ac9d-392e614a2e49" (UID: "e382f7b4-6d31-482e-ac9d-392e614a2e49"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 22:13:35.571335 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:35.571320 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e382f7b4-6d31-482e-ac9d-392e614a2e49-kube-api-access-w4j25" (OuterVolumeSpecName: "kube-api-access-w4j25") pod "e382f7b4-6d31-482e-ac9d-392e614a2e49" (UID: "e382f7b4-6d31-482e-ac9d-392e614a2e49"). InnerVolumeSpecName "kube-api-access-w4j25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:13:35.571399 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:35.571375 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e382f7b4-6d31-482e-ac9d-392e614a2e49-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e382f7b4-6d31-482e-ac9d-392e614a2e49" (UID: "e382f7b4-6d31-482e-ac9d-392e614a2e49"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 22:13:35.670053 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:35.670026 2550 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e382f7b4-6d31-482e-ac9d-392e614a2e49-console-config\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:13:35.670053 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:35.670047 2550 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e382f7b4-6d31-482e-ac9d-392e614a2e49-console-oauth-config\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:13:35.670053 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:35.670057 2550 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e382f7b4-6d31-482e-ac9d-392e614a2e49-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:13:35.670229 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:35.670065 2550 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e382f7b4-6d31-482e-ac9d-392e614a2e49-console-serving-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:13:35.670229 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:35.670073 2550 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e382f7b4-6d31-482e-ac9d-392e614a2e49-trusted-ca-bundle\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:13:35.670229 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:35.670081 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4j25\" (UniqueName: \"kubernetes.io/projected/e382f7b4-6d31-482e-ac9d-392e614a2e49-kube-api-access-w4j25\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:13:36.179034 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:36.179003 2550 generic.go:356] "Generic (PLEG): container finished" podID="e382f7b4-6d31-482e-ac9d-392e614a2e49" containerID="9bbb76b48907ac393a5d9550753256a73bd7420c24de8c922425de3fa1e07834" exitCode=0 Jul 14 22:13:36.179462 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:36.179045 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-654ff486f-vsmql" Jul 14 22:13:36.179462 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:36.179078 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-654ff486f-vsmql" event={"ID":"e382f7b4-6d31-482e-ac9d-392e614a2e49","Type":"ContainerDied","Data":"9bbb76b48907ac393a5d9550753256a73bd7420c24de8c922425de3fa1e07834"} Jul 14 22:13:36.179462 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:36.179117 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-654ff486f-vsmql" event={"ID":"e382f7b4-6d31-482e-ac9d-392e614a2e49","Type":"ContainerDied","Data":"1a6fb2cc991b78af0066c35de960e2ba3dec482e1eb7e226c0f97602eb935c1d"} Jul 14 22:13:36.179462 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:36.179138 2550 scope.go:117] "RemoveContainer" containerID="9bbb76b48907ac393a5d9550753256a73bd7420c24de8c922425de3fa1e07834" Jul 14 22:13:36.187081 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:36.187065 2550 scope.go:117] "RemoveContainer" containerID="9bbb76b48907ac393a5d9550753256a73bd7420c24de8c922425de3fa1e07834" Jul 14 22:13:36.187312 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:13:36.187295 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bbb76b48907ac393a5d9550753256a73bd7420c24de8c922425de3fa1e07834\": container with ID starting with 9bbb76b48907ac393a5d9550753256a73bd7420c24de8c922425de3fa1e07834 not found: ID does not exist" containerID="9bbb76b48907ac393a5d9550753256a73bd7420c24de8c922425de3fa1e07834" Jul 14 22:13:36.187370 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:36.187320 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bbb76b48907ac393a5d9550753256a73bd7420c24de8c922425de3fa1e07834"} err="failed to get container status \"9bbb76b48907ac393a5d9550753256a73bd7420c24de8c922425de3fa1e07834\": rpc error: code = NotFound desc = could not find container \"9bbb76b48907ac393a5d9550753256a73bd7420c24de8c922425de3fa1e07834\": container with ID starting with 9bbb76b48907ac393a5d9550753256a73bd7420c24de8c922425de3fa1e07834 not found: ID does not exist" Jul 14 22:13:36.201416 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:36.201384 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-654ff486f-vsmql"] Jul 14 22:13:36.204834 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:36.204816 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-654ff486f-vsmql"] Jul 14 22:13:38.197627 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:13:38.197597 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e382f7b4-6d31-482e-ac9d-392e614a2e49" path="/var/lib/kubelet/pods/e382f7b4-6d31-482e-ac9d-392e614a2e49/volumes" Jul 14 22:14:06.315154 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:14:06.315056 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:14:07.474764 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:14:07.474734 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:14:14.605543 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:14:14.605502 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:14:14.666084 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:14:14.666061 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:14:29.987497 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:14:29.987468 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zj4sb_0884baa5-9206-4196-a7c9-b5686623311b/ovn-acl-logging/0.log" Jul 14 22:14:29.989810 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:14:29.989791 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zj4sb_0884baa5-9206-4196-a7c9-b5686623311b/ovn-acl-logging/0.log" Jul 14 22:14:29.990125 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:14:29.990099 2550 kubelet.go:1648] "Image garbage collection succeeded" Jul 14 22:15:10.111882 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:10.111847 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:15:25.018768 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:25.018650 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vpcnj"] Jul 14 22:15:25.019254 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:25.019062 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e382f7b4-6d31-482e-ac9d-392e614a2e49" containerName="console" Jul 14 22:15:25.019254 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:25.019078 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e382f7b4-6d31-482e-ac9d-392e614a2e49" containerName="console" Jul 14 22:15:25.019254 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:25.019147 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e382f7b4-6d31-482e-ac9d-392e614a2e49" containerName="console" Jul 14 22:15:25.021238 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:25.021222 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vpcnj" Jul 14 22:15:25.023950 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:25.023929 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 14 22:15:25.024238 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:25.024222 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 14 22:15:25.025114 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:25.025099 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jj9nv\"" Jul 14 22:15:25.038625 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:25.038601 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vpcnj"] Jul 14 22:15:25.048618 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:25.048591 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm7bj\" (UniqueName: \"kubernetes.io/projected/8667ec8b-6a0c-4cd4-99c4-99d874792e7a-kube-api-access-xm7bj\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vpcnj\" (UID: \"8667ec8b-6a0c-4cd4-99c4-99d874792e7a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vpcnj" Jul 14 22:15:25.048727 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:25.048627 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8667ec8b-6a0c-4cd4-99c4-99d874792e7a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vpcnj\" (UID: \"8667ec8b-6a0c-4cd4-99c4-99d874792e7a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vpcnj" Jul 14 22:15:25.048727 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:25.048717 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8667ec8b-6a0c-4cd4-99c4-99d874792e7a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vpcnj\" (UID: \"8667ec8b-6a0c-4cd4-99c4-99d874792e7a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vpcnj" Jul 14 22:15:25.149353 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:25.149326 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8667ec8b-6a0c-4cd4-99c4-99d874792e7a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vpcnj\" (UID: \"8667ec8b-6a0c-4cd4-99c4-99d874792e7a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vpcnj" Jul 14 22:15:25.149529 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:25.149378 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8667ec8b-6a0c-4cd4-99c4-99d874792e7a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vpcnj\" (UID: \"8667ec8b-6a0c-4cd4-99c4-99d874792e7a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vpcnj" Jul 14 22:15:25.149529 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:25.149422 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm7bj\" (UniqueName: \"kubernetes.io/projected/8667ec8b-6a0c-4cd4-99c4-99d874792e7a-kube-api-access-xm7bj\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vpcnj\" (UID: \"8667ec8b-6a0c-4cd4-99c4-99d874792e7a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vpcnj" Jul 14 22:15:25.149741 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:25.149722 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8667ec8b-6a0c-4cd4-99c4-99d874792e7a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vpcnj\" (UID: \"8667ec8b-6a0c-4cd4-99c4-99d874792e7a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vpcnj" Jul 14 22:15:25.149783 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:25.149751 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8667ec8b-6a0c-4cd4-99c4-99d874792e7a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vpcnj\" (UID: \"8667ec8b-6a0c-4cd4-99c4-99d874792e7a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vpcnj" Jul 14 22:15:25.157895 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:25.157871 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm7bj\" (UniqueName: \"kubernetes.io/projected/8667ec8b-6a0c-4cd4-99c4-99d874792e7a-kube-api-access-xm7bj\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vpcnj\" (UID: \"8667ec8b-6a0c-4cd4-99c4-99d874792e7a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vpcnj" Jul 14 22:15:25.329313 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:25.329225 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vpcnj" Jul 14 22:15:25.451142 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:25.451042 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vpcnj"] Jul 14 22:15:25.454053 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:15:25.454025 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8667ec8b_6a0c_4cd4_99c4_99d874792e7a.slice/crio-aedcbb8fe7bead800370932efa3c3291444817a81ac2102820bf532b42eeb5e0 WatchSource:0}: Error finding container aedcbb8fe7bead800370932efa3c3291444817a81ac2102820bf532b42eeb5e0: Status 404 returned error can't find the container with id aedcbb8fe7bead800370932efa3c3291444817a81ac2102820bf532b42eeb5e0 Jul 14 22:15:25.456271 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:25.456257 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 22:15:25.556436 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:25.556407 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vpcnj" event={"ID":"8667ec8b-6a0c-4cd4-99c4-99d874792e7a","Type":"ContainerStarted","Data":"aedcbb8fe7bead800370932efa3c3291444817a81ac2102820bf532b42eeb5e0"} Jul 14 22:15:32.585885 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:32.585797 2550 generic.go:356] "Generic (PLEG): container finished" podID="8667ec8b-6a0c-4cd4-99c4-99d874792e7a" containerID="cb73f050fee6b6aab22d8e8abfa85f9225a002e2f9844e20afbf2a51d11e5e3c" exitCode=0 Jul 14 22:15:32.586234 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:32.585890 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vpcnj" event={"ID":"8667ec8b-6a0c-4cd4-99c4-99d874792e7a","Type":"ContainerDied","Data":"cb73f050fee6b6aab22d8e8abfa85f9225a002e2f9844e20afbf2a51d11e5e3c"} Jul 14 22:15:34.596730 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:34.596698 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vpcnj" event={"ID":"8667ec8b-6a0c-4cd4-99c4-99d874792e7a","Type":"ContainerStarted","Data":"8cdc11507784b91dc923a2cf7274a5326262792a7fff7a5797474d271db89ae3"} Jul 14 22:15:35.601843 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:35.601801 2550 generic.go:356] "Generic (PLEG): container finished" podID="8667ec8b-6a0c-4cd4-99c4-99d874792e7a" containerID="8cdc11507784b91dc923a2cf7274a5326262792a7fff7a5797474d271db89ae3" exitCode=0 Jul 14 22:15:35.602214 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:35.601879 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vpcnj" event={"ID":"8667ec8b-6a0c-4cd4-99c4-99d874792e7a","Type":"ContainerDied","Data":"8cdc11507784b91dc923a2cf7274a5326262792a7fff7a5797474d271db89ae3"} Jul 14 22:15:43.635079 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:43.634996 2550 generic.go:356] "Generic (PLEG): container finished" podID="8667ec8b-6a0c-4cd4-99c4-99d874792e7a" containerID="7717b03b5cd5c25c73c492ee788d57912fd61c102a09a2b0bfe9a7bcc4672ed3" exitCode=0 Jul 14 22:15:43.635437 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:43.635081 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vpcnj" event={"ID":"8667ec8b-6a0c-4cd4-99c4-99d874792e7a","Type":"ContainerDied","Data":"7717b03b5cd5c25c73c492ee788d57912fd61c102a09a2b0bfe9a7bcc4672ed3"} Jul 14 22:15:44.762422 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:44.762396 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vpcnj" Jul 14 22:15:44.913342 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:44.913310 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8667ec8b-6a0c-4cd4-99c4-99d874792e7a-util\") pod \"8667ec8b-6a0c-4cd4-99c4-99d874792e7a\" (UID: \"8667ec8b-6a0c-4cd4-99c4-99d874792e7a\") " Jul 14 22:15:44.913512 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:44.913366 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xm7bj\" (UniqueName: \"kubernetes.io/projected/8667ec8b-6a0c-4cd4-99c4-99d874792e7a-kube-api-access-xm7bj\") pod \"8667ec8b-6a0c-4cd4-99c4-99d874792e7a\" (UID: \"8667ec8b-6a0c-4cd4-99c4-99d874792e7a\") " Jul 14 22:15:44.913512 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:44.913383 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8667ec8b-6a0c-4cd4-99c4-99d874792e7a-bundle\") pod \"8667ec8b-6a0c-4cd4-99c4-99d874792e7a\" (UID: \"8667ec8b-6a0c-4cd4-99c4-99d874792e7a\") " Jul 14 22:15:44.914108 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:44.914079 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8667ec8b-6a0c-4cd4-99c4-99d874792e7a-bundle" (OuterVolumeSpecName: "bundle") pod "8667ec8b-6a0c-4cd4-99c4-99d874792e7a" (UID: "8667ec8b-6a0c-4cd4-99c4-99d874792e7a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:15:44.915511 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:44.915490 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8667ec8b-6a0c-4cd4-99c4-99d874792e7a-kube-api-access-xm7bj" (OuterVolumeSpecName: "kube-api-access-xm7bj") pod "8667ec8b-6a0c-4cd4-99c4-99d874792e7a" (UID: "8667ec8b-6a0c-4cd4-99c4-99d874792e7a"). InnerVolumeSpecName "kube-api-access-xm7bj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:15:44.917643 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:44.917622 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8667ec8b-6a0c-4cd4-99c4-99d874792e7a-util" (OuterVolumeSpecName: "util") pod "8667ec8b-6a0c-4cd4-99c4-99d874792e7a" (UID: "8667ec8b-6a0c-4cd4-99c4-99d874792e7a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:15:45.014269 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:45.014226 2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8667ec8b-6a0c-4cd4-99c4-99d874792e7a-util\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:15:45.014269 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:45.014265 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm7bj\" (UniqueName: \"kubernetes.io/projected/8667ec8b-6a0c-4cd4-99c4-99d874792e7a-kube-api-access-xm7bj\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:15:45.014269 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:45.014275 2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8667ec8b-6a0c-4cd4-99c4-99d874792e7a-bundle\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:15:45.643091 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:45.643062 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vpcnj" Jul 14 22:15:45.643282 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:45.643059 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vpcnj" event={"ID":"8667ec8b-6a0c-4cd4-99c4-99d874792e7a","Type":"ContainerDied","Data":"aedcbb8fe7bead800370932efa3c3291444817a81ac2102820bf532b42eeb5e0"} Jul 14 22:15:45.643282 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:45.643175 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aedcbb8fe7bead800370932efa3c3291444817a81ac2102820bf532b42eeb5e0" Jul 14 22:15:48.077637 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:48.077608 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:15:48.210017 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:48.209987 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:15:50.076838 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:50.076806 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-l246m"] Jul 14 22:15:50.077274 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:50.077060 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8667ec8b-6a0c-4cd4-99c4-99d874792e7a" containerName="util" Jul 14 22:15:50.077274 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:50.077069 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="8667ec8b-6a0c-4cd4-99c4-99d874792e7a" containerName="util" Jul 14 22:15:50.077274 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:50.077079 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8667ec8b-6a0c-4cd4-99c4-99d874792e7a" containerName="extract" Jul 14 22:15:50.077274 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:50.077084 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="8667ec8b-6a0c-4cd4-99c4-99d874792e7a" containerName="extract" Jul 14 22:15:50.077274 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:50.077093 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8667ec8b-6a0c-4cd4-99c4-99d874792e7a" containerName="pull" Jul 14 22:15:50.077274 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:50.077099 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="8667ec8b-6a0c-4cd4-99c4-99d874792e7a" containerName="pull" Jul 14 22:15:50.077274 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:50.077138 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="8667ec8b-6a0c-4cd4-99c4-99d874792e7a" containerName="extract" Jul 14 22:15:50.080774 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:50.080756 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-l246m" Jul 14 22:15:50.083366 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:50.083348 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 14 22:15:50.083929 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:50.083910 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 14 22:15:50.084029 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:50.083985 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-d6k56\"" Jul 14 22:15:50.092457 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:50.092433 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-l246m"] Jul 14 22:15:50.150849 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:50.150820 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56e84f87-4cfb-45f7-ab8b-9d0f6c5da0fd-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-l246m\" (UID: \"56e84f87-4cfb-45f7-ab8b-9d0f6c5da0fd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-l246m" Jul 14 22:15:50.150993 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:50.150855 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxxbz\" (UniqueName: \"kubernetes.io/projected/56e84f87-4cfb-45f7-ab8b-9d0f6c5da0fd-kube-api-access-fxxbz\") pod \"cert-manager-operator-controller-manager-55c444bf86-l246m\" (UID: \"56e84f87-4cfb-45f7-ab8b-9d0f6c5da0fd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-l246m" Jul 14 22:15:50.251459 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:50.251427 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56e84f87-4cfb-45f7-ab8b-9d0f6c5da0fd-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-l246m\" (UID: \"56e84f87-4cfb-45f7-ab8b-9d0f6c5da0fd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-l246m" Jul 14 22:15:50.251721 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:50.251702 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxxbz\" (UniqueName: \"kubernetes.io/projected/56e84f87-4cfb-45f7-ab8b-9d0f6c5da0fd-kube-api-access-fxxbz\") pod \"cert-manager-operator-controller-manager-55c444bf86-l246m\" (UID: \"56e84f87-4cfb-45f7-ab8b-9d0f6c5da0fd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-l246m" Jul 14 22:15:50.252347 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:50.252276 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56e84f87-4cfb-45f7-ab8b-9d0f6c5da0fd-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-l246m\" (UID: \"56e84f87-4cfb-45f7-ab8b-9d0f6c5da0fd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-l246m" Jul 14 22:15:50.265861 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:50.265833 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxxbz\" (UniqueName: \"kubernetes.io/projected/56e84f87-4cfb-45f7-ab8b-9d0f6c5da0fd-kube-api-access-fxxbz\") pod \"cert-manager-operator-controller-manager-55c444bf86-l246m\" (UID: \"56e84f87-4cfb-45f7-ab8b-9d0f6c5da0fd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-l246m" Jul 14 22:15:50.389972 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:50.389874 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-l246m" Jul 14 22:15:50.539089 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:50.539056 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-l246m"] Jul 14 22:15:50.542188 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:15:50.542161 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56e84f87_4cfb_45f7_ab8b_9d0f6c5da0fd.slice/crio-3ea4fa2cf30ee5e1a9148d979c9aaf280632dc7893f3b8b83a752598ae6b21d0 WatchSource:0}: Error finding container 3ea4fa2cf30ee5e1a9148d979c9aaf280632dc7893f3b8b83a752598ae6b21d0: Status 404 returned error can't find the container with id 3ea4fa2cf30ee5e1a9148d979c9aaf280632dc7893f3b8b83a752598ae6b21d0 Jul 14 22:15:50.666532 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:50.666504 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-l246m" event={"ID":"56e84f87-4cfb-45f7-ab8b-9d0f6c5da0fd","Type":"ContainerStarted","Data":"3ea4fa2cf30ee5e1a9148d979c9aaf280632dc7893f3b8b83a752598ae6b21d0"} Jul 14 22:15:51.880675 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:51.880634 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:15:51.986649 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:51.986610 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:15:52.676210 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:52.676181 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-l246m" event={"ID":"56e84f87-4cfb-45f7-ab8b-9d0f6c5da0fd","Type":"ContainerStarted","Data":"4ef7ce6f52693168807f93e56d9fb8f8884fc423ac4ddf230d266b99b42ee656"} Jul 14 22:15:52.711039 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:52.710963 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-l246m" podStartSLOduration=1.016057188 podStartE2EDuration="2.710944497s" totalImagesPullingTime="1.694887309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:15:52.709765622 +0000 UTC m=+383.068912063" watchObservedRunningTime="2026-07-14 22:15:52.710944497 +0000 UTC m=+383.070090936" Jul 14 22:15:53.196038 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:53.196012 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:15:53.426971 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:53.426943 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:15:53.618479 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:53.618408 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmxmrp"] Jul 14 22:15:53.620899 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:53.620880 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmxmrp" Jul 14 22:15:53.624188 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:53.624172 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 14 22:15:53.624317 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:53.624298 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jj9nv\"" Jul 14 22:15:53.625311 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:53.625299 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 14 22:15:53.640139 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:53.640113 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmxmrp"] Jul 14 22:15:53.780981 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:53.780952 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9777439-a3e0-4d42-aefc-e09e23a7877b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmxmrp\" (UID: \"f9777439-a3e0-4d42-aefc-e09e23a7877b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmxmrp" Jul 14 22:15:53.781129 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:53.780986 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9777439-a3e0-4d42-aefc-e09e23a7877b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmxmrp\" (UID: \"f9777439-a3e0-4d42-aefc-e09e23a7877b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmxmrp" Jul 14 22:15:53.781129 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:53.781025 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdw6q\" (UniqueName: \"kubernetes.io/projected/f9777439-a3e0-4d42-aefc-e09e23a7877b-kube-api-access-fdw6q\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmxmrp\" (UID: \"f9777439-a3e0-4d42-aefc-e09e23a7877b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmxmrp" Jul 14 22:15:53.881877 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:53.881791 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9777439-a3e0-4d42-aefc-e09e23a7877b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmxmrp\" (UID: \"f9777439-a3e0-4d42-aefc-e09e23a7877b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmxmrp" Jul 14 22:15:53.881877 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:53.881832 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9777439-a3e0-4d42-aefc-e09e23a7877b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmxmrp\" (UID: \"f9777439-a3e0-4d42-aefc-e09e23a7877b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmxmrp" Jul 14 22:15:53.882089 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:53.881968 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdw6q\" (UniqueName: \"kubernetes.io/projected/f9777439-a3e0-4d42-aefc-e09e23a7877b-kube-api-access-fdw6q\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmxmrp\" (UID: \"f9777439-a3e0-4d42-aefc-e09e23a7877b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmxmrp" Jul 14 22:15:53.882186 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:53.882166 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9777439-a3e0-4d42-aefc-e09e23a7877b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmxmrp\" (UID: \"f9777439-a3e0-4d42-aefc-e09e23a7877b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmxmrp" Jul 14 22:15:53.882263 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:53.882229 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9777439-a3e0-4d42-aefc-e09e23a7877b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmxmrp\" (UID: \"f9777439-a3e0-4d42-aefc-e09e23a7877b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmxmrp" Jul 14 22:15:53.895545 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:53.895520 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdw6q\" (UniqueName: \"kubernetes.io/projected/f9777439-a3e0-4d42-aefc-e09e23a7877b-kube-api-access-fdw6q\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmxmrp\" (UID: \"f9777439-a3e0-4d42-aefc-e09e23a7877b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmxmrp" Jul 14 22:15:53.929646 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:53.929611 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmxmrp" Jul 14 22:15:53.974824 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:53.974793 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:15:54.068927 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:54.068837 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmxmrp"] Jul 14 22:15:54.071568 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:15:54.071525 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9777439_a3e0_4d42_aefc_e09e23a7877b.slice/crio-a224d9415c2b0784eb6e4f0d7626eb58d4e32db7a4cb9b9e28099d2cf147c0e8 WatchSource:0}: Error finding container a224d9415c2b0784eb6e4f0d7626eb58d4e32db7a4cb9b9e28099d2cf147c0e8: Status 404 returned error can't find the container with id a224d9415c2b0784eb6e4f0d7626eb58d4e32db7a4cb9b9e28099d2cf147c0e8 Jul 14 22:15:54.073455 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:54.073433 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:15:54.137498 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:54.137439 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-npm6c"] Jul 14 22:15:54.139435 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:54.139419 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-npm6c" Jul 14 22:15:54.143761 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:54.143743 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-nx4c7\"" Jul 14 22:15:54.144517 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:54.144496 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 14 22:15:54.144970 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:54.144949 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 14 22:15:54.152639 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:54.152614 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-npm6c"] Jul 14 22:15:54.285658 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:54.285622 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f5e61b8a-61d5-450c-83f4-14dbe53d15e1-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-npm6c\" (UID: \"f5e61b8a-61d5-450c-83f4-14dbe53d15e1\") " pod="cert-manager/cert-manager-webhook-6486ffd997-npm6c" Jul 14 22:15:54.286011 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:54.285689 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vsvt\" (UniqueName: \"kubernetes.io/projected/f5e61b8a-61d5-450c-83f4-14dbe53d15e1-kube-api-access-5vsvt\") pod \"cert-manager-webhook-6486ffd997-npm6c\" (UID: \"f5e61b8a-61d5-450c-83f4-14dbe53d15e1\") " pod="cert-manager/cert-manager-webhook-6486ffd997-npm6c" Jul 14 22:15:54.386610 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:54.386576 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vsvt\" (UniqueName: \"kubernetes.io/projected/f5e61b8a-61d5-450c-83f4-14dbe53d15e1-kube-api-access-5vsvt\") pod \"cert-manager-webhook-6486ffd997-npm6c\" (UID: \"f5e61b8a-61d5-450c-83f4-14dbe53d15e1\") " pod="cert-manager/cert-manager-webhook-6486ffd997-npm6c" Jul 14 22:15:54.386765 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:54.386655 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f5e61b8a-61d5-450c-83f4-14dbe53d15e1-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-npm6c\" (UID: \"f5e61b8a-61d5-450c-83f4-14dbe53d15e1\") " pod="cert-manager/cert-manager-webhook-6486ffd997-npm6c" Jul 14 22:15:54.403233 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:54.403184 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vsvt\" (UniqueName: \"kubernetes.io/projected/f5e61b8a-61d5-450c-83f4-14dbe53d15e1-kube-api-access-5vsvt\") pod \"cert-manager-webhook-6486ffd997-npm6c\" (UID: \"f5e61b8a-61d5-450c-83f4-14dbe53d15e1\") " pod="cert-manager/cert-manager-webhook-6486ffd997-npm6c" Jul 14 22:15:54.415107 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:54.415090 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f5e61b8a-61d5-450c-83f4-14dbe53d15e1-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-npm6c\" (UID: \"f5e61b8a-61d5-450c-83f4-14dbe53d15e1\") " pod="cert-manager/cert-manager-webhook-6486ffd997-npm6c" Jul 14 22:15:54.460391 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:54.460366 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-npm6c" Jul 14 22:15:54.585456 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:54.585432 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-npm6c"] Jul 14 22:15:54.587799 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:15:54.587770 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5e61b8a_61d5_450c_83f4_14dbe53d15e1.slice/crio-ef3b74e4f2ac5642436b90be1af4cb687675031c031c4c40e2f4638312549205 WatchSource:0}: Error finding container ef3b74e4f2ac5642436b90be1af4cb687675031c031c4c40e2f4638312549205: Status 404 returned error can't find the container with id ef3b74e4f2ac5642436b90be1af4cb687675031c031c4c40e2f4638312549205 Jul 14 22:15:54.685656 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:54.685627 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-npm6c" event={"ID":"f5e61b8a-61d5-450c-83f4-14dbe53d15e1","Type":"ContainerStarted","Data":"ef3b74e4f2ac5642436b90be1af4cb687675031c031c4c40e2f4638312549205"} Jul 14 22:15:54.686947 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:54.686927 2550 generic.go:356] "Generic (PLEG): container finished" podID="f9777439-a3e0-4d42-aefc-e09e23a7877b" containerID="b93092d98db9147040edad3e84ca701b450b8e49e6c20046794d0305eb8357de" exitCode=0 Jul 14 22:15:54.687040 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:54.686979 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmxmrp" event={"ID":"f9777439-a3e0-4d42-aefc-e09e23a7877b","Type":"ContainerDied","Data":"b93092d98db9147040edad3e84ca701b450b8e49e6c20046794d0305eb8357de"} Jul 14 22:15:54.687040 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:54.687008 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmxmrp" event={"ID":"f9777439-a3e0-4d42-aefc-e09e23a7877b","Type":"ContainerStarted","Data":"a224d9415c2b0784eb6e4f0d7626eb58d4e32db7a4cb9b9e28099d2cf147c0e8"} Jul 14 22:15:56.366312 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:56.366281 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:15:56.593211 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:56.593173 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:15:58.706725 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:58.706685 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-npm6c" event={"ID":"f5e61b8a-61d5-450c-83f4-14dbe53d15e1","Type":"ContainerStarted","Data":"fb5e264d857487cc3d55013d901e994820eaa58d2a1eba7544a3b68f65625a89"} Jul 14 22:15:58.707162 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:58.706763 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-npm6c" Jul 14 22:15:58.708373 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:58.708350 2550 generic.go:356] "Generic (PLEG): container finished" podID="f9777439-a3e0-4d42-aefc-e09e23a7877b" containerID="69ce952072dffa2c521630e87c9344f51ae4f9ab78f96ae91ed10e13cd608d37" exitCode=0 Jul 14 22:15:58.708500 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:58.708404 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmxmrp" event={"ID":"f9777439-a3e0-4d42-aefc-e09e23a7877b","Type":"ContainerDied","Data":"69ce952072dffa2c521630e87c9344f51ae4f9ab78f96ae91ed10e13cd608d37"} Jul 14 22:15:58.724871 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:58.724835 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-npm6c" podStartSLOduration=1.57794776 podStartE2EDuration="4.72481929s" totalImagesPullingTime="3.14687153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:15:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:15:58.723683918 +0000 UTC m=+389.082830392" watchObservedRunningTime="2026-07-14 22:15:58.72481929 +0000 UTC m=+389.083965726" Jul 14 22:15:58.862896 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:58.862871 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:15:58.962783 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:58.962706 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:15:59.714603 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:59.714573 2550 generic.go:356] "Generic (PLEG): container finished" podID="f9777439-a3e0-4d42-aefc-e09e23a7877b" containerID="1bed52450c0632930e6505d9720d77cfd5f40362aa236c334e69c2115bb1a3ca" exitCode=0 Jul 14 22:15:59.715069 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:15:59.714643 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmxmrp" event={"ID":"f9777439-a3e0-4d42-aefc-e09e23a7877b","Type":"ContainerDied","Data":"1bed52450c0632930e6505d9720d77cfd5f40362aa236c334e69c2115bb1a3ca"} Jul 14 22:16:00.837981 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:00.837959 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmxmrp" Jul 14 22:16:00.941625 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:00.941592 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdw6q\" (UniqueName: \"kubernetes.io/projected/f9777439-a3e0-4d42-aefc-e09e23a7877b-kube-api-access-fdw6q\") pod \"f9777439-a3e0-4d42-aefc-e09e23a7877b\" (UID: \"f9777439-a3e0-4d42-aefc-e09e23a7877b\") " Jul 14 22:16:00.941810 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:00.941668 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9777439-a3e0-4d42-aefc-e09e23a7877b-bundle\") pod \"f9777439-a3e0-4d42-aefc-e09e23a7877b\" (UID: \"f9777439-a3e0-4d42-aefc-e09e23a7877b\") " Jul 14 22:16:00.941810 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:00.941686 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9777439-a3e0-4d42-aefc-e09e23a7877b-util\") pod \"f9777439-a3e0-4d42-aefc-e09e23a7877b\" (UID: \"f9777439-a3e0-4d42-aefc-e09e23a7877b\") " Jul 14 22:16:00.942070 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:00.942047 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9777439-a3e0-4d42-aefc-e09e23a7877b-bundle" (OuterVolumeSpecName: "bundle") pod "f9777439-a3e0-4d42-aefc-e09e23a7877b" (UID: "f9777439-a3e0-4d42-aefc-e09e23a7877b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:16:00.943767 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:00.943747 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9777439-a3e0-4d42-aefc-e09e23a7877b-kube-api-access-fdw6q" (OuterVolumeSpecName: "kube-api-access-fdw6q") pod "f9777439-a3e0-4d42-aefc-e09e23a7877b" (UID: "f9777439-a3e0-4d42-aefc-e09e23a7877b"). InnerVolumeSpecName "kube-api-access-fdw6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:16:00.946291 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:00.946273 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9777439-a3e0-4d42-aefc-e09e23a7877b-util" (OuterVolumeSpecName: "util") pod "f9777439-a3e0-4d42-aefc-e09e23a7877b" (UID: "f9777439-a3e0-4d42-aefc-e09e23a7877b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:16:01.042949 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:01.042873 2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9777439-a3e0-4d42-aefc-e09e23a7877b-bundle\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:16:01.042949 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:01.042900 2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9777439-a3e0-4d42-aefc-e09e23a7877b-util\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:16:01.042949 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:01.042910 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdw6q\" (UniqueName: \"kubernetes.io/projected/f9777439-a3e0-4d42-aefc-e09e23a7877b-kube-api-access-fdw6q\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:16:01.723812 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:01.723782 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmxmrp" event={"ID":"f9777439-a3e0-4d42-aefc-e09e23a7877b","Type":"ContainerDied","Data":"a224d9415c2b0784eb6e4f0d7626eb58d4e32db7a4cb9b9e28099d2cf147c0e8"} Jul 14 22:16:01.723812 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:01.723810 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmxmrp" Jul 14 22:16:01.724022 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:01.723827 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a224d9415c2b0784eb6e4f0d7626eb58d4e32db7a4cb9b9e28099d2cf147c0e8" Jul 14 22:16:02.241129 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:02.241095 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-q2vq5"] Jul 14 22:16:02.241496 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:02.241334 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9777439-a3e0-4d42-aefc-e09e23a7877b" containerName="util" Jul 14 22:16:02.241496 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:02.241344 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9777439-a3e0-4d42-aefc-e09e23a7877b" containerName="util" Jul 14 22:16:02.241496 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:02.241362 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9777439-a3e0-4d42-aefc-e09e23a7877b" containerName="extract" Jul 14 22:16:02.241496 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:02.241368 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9777439-a3e0-4d42-aefc-e09e23a7877b" containerName="extract" Jul 14 22:16:02.241496 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:02.241375 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9777439-a3e0-4d42-aefc-e09e23a7877b" containerName="pull" Jul 14 22:16:02.241496 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:02.241380 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9777439-a3e0-4d42-aefc-e09e23a7877b" containerName="pull" Jul 14 22:16:02.241496 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:02.241425 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9777439-a3e0-4d42-aefc-e09e23a7877b" containerName="extract" Jul 14 22:16:02.243299 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:02.243281 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-q2vq5" Jul 14 22:16:02.245999 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:02.245976 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-62q6d\"" Jul 14 22:16:02.257873 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:02.257852 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-q2vq5"] Jul 14 22:16:02.352897 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:02.352864 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps9cl\" (UniqueName: \"kubernetes.io/projected/b76c7d2f-eac5-4117-a101-71630d1f6c5e-kube-api-access-ps9cl\") pod \"cert-manager-cainjector-559cdd8c87-q2vq5\" (UID: \"b76c7d2f-eac5-4117-a101-71630d1f6c5e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-q2vq5" Jul 14 22:16:02.352897 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:02.352899 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b76c7d2f-eac5-4117-a101-71630d1f6c5e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-q2vq5\" (UID: \"b76c7d2f-eac5-4117-a101-71630d1f6c5e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-q2vq5" Jul 14 22:16:02.453901 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:02.453866 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b76c7d2f-eac5-4117-a101-71630d1f6c5e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-q2vq5\" (UID: \"b76c7d2f-eac5-4117-a101-71630d1f6c5e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-q2vq5" Jul 14 22:16:02.454049 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:02.453950 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps9cl\" (UniqueName: \"kubernetes.io/projected/b76c7d2f-eac5-4117-a101-71630d1f6c5e-kube-api-access-ps9cl\") pod \"cert-manager-cainjector-559cdd8c87-q2vq5\" (UID: \"b76c7d2f-eac5-4117-a101-71630d1f6c5e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-q2vq5" Jul 14 22:16:02.463788 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:02.463756 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b76c7d2f-eac5-4117-a101-71630d1f6c5e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-q2vq5\" (UID: \"b76c7d2f-eac5-4117-a101-71630d1f6c5e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-q2vq5" Jul 14 22:16:02.463890 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:02.463870 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps9cl\" (UniqueName: \"kubernetes.io/projected/b76c7d2f-eac5-4117-a101-71630d1f6c5e-kube-api-access-ps9cl\") pod \"cert-manager-cainjector-559cdd8c87-q2vq5\" (UID: \"b76c7d2f-eac5-4117-a101-71630d1f6c5e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-q2vq5" Jul 14 22:16:02.551823 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:02.551723 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-q2vq5" Jul 14 22:16:02.678379 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:02.678348 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-q2vq5"] Jul 14 22:16:02.682178 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:16:02.682152 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb76c7d2f_eac5_4117_a101_71630d1f6c5e.slice/crio-e97edba3669b8e939dbcb3d93454c0c28f4c7cdefa3a896759c6247ee3d02387 WatchSource:0}: Error finding container e97edba3669b8e939dbcb3d93454c0c28f4c7cdefa3a896759c6247ee3d02387: Status 404 returned error can't find the container with id e97edba3669b8e939dbcb3d93454c0c28f4c7cdefa3a896759c6247ee3d02387 Jul 14 22:16:02.728854 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:02.728825 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-q2vq5" event={"ID":"b76c7d2f-eac5-4117-a101-71630d1f6c5e","Type":"ContainerStarted","Data":"e97edba3669b8e939dbcb3d93454c0c28f4c7cdefa3a896759c6247ee3d02387"} Jul 14 22:16:03.618032 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:03.618001 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:03.733897 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:03.733865 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-q2vq5" event={"ID":"b76c7d2f-eac5-4117-a101-71630d1f6c5e","Type":"ContainerStarted","Data":"8b7a8038d4f2fe6926059b98b6b1b7921b2f9ad2d0f6eea97037e01661b4833c"} Jul 14 22:16:03.772717 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:03.772675 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-q2vq5" podStartSLOduration=1.772659186 podStartE2EDuration="1.772659186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:16:03.771376949 +0000 UTC m=+394.130523385" watchObservedRunningTime="2026-07-14 22:16:03.772659186 +0000 UTC m=+394.131805623" Jul 14 22:16:03.793762 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:03.793736 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:03.992871 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:03.992843 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:04.717158 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:04.717126 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-npm6c" Jul 14 22:16:05.760982 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:05.760951 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:05.895265 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:05.895234 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:07.229864 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:07.229833 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ndz9d"] Jul 14 22:16:07.232099 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:07.232082 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ndz9d" Jul 14 22:16:07.237139 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:07.237120 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 14 22:16:07.237237 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:07.237198 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 14 22:16:07.237237 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:07.237217 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-gpr9v\"" Jul 14 22:16:07.246065 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:07.246046 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ndz9d"] Jul 14 22:16:07.396176 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:07.396139 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p59g\" (UniqueName: \"kubernetes.io/projected/3d25f945-dabe-42ac-af03-f36dfbd0e213-kube-api-access-7p59g\") pod \"openshift-lws-operator-fd8ccff4c-ndz9d\" (UID: \"3d25f945-dabe-42ac-af03-f36dfbd0e213\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ndz9d" Jul 14 22:16:07.396326 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:07.396187 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d25f945-dabe-42ac-af03-f36dfbd0e213-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ndz9d\" (UID: \"3d25f945-dabe-42ac-af03-f36dfbd0e213\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ndz9d" Jul 14 22:16:07.496814 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:07.496731 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d25f945-dabe-42ac-af03-f36dfbd0e213-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ndz9d\" (UID: \"3d25f945-dabe-42ac-af03-f36dfbd0e213\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ndz9d" Jul 14 22:16:07.496814 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:07.496803 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p59g\" (UniqueName: \"kubernetes.io/projected/3d25f945-dabe-42ac-af03-f36dfbd0e213-kube-api-access-7p59g\") pod \"openshift-lws-operator-fd8ccff4c-ndz9d\" (UID: \"3d25f945-dabe-42ac-af03-f36dfbd0e213\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ndz9d" Jul 14 22:16:07.497121 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:07.497101 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d25f945-dabe-42ac-af03-f36dfbd0e213-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ndz9d\" (UID: \"3d25f945-dabe-42ac-af03-f36dfbd0e213\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ndz9d" Jul 14 22:16:07.512500 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:07.512470 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p59g\" (UniqueName: \"kubernetes.io/projected/3d25f945-dabe-42ac-af03-f36dfbd0e213-kube-api-access-7p59g\") pod \"openshift-lws-operator-fd8ccff4c-ndz9d\" (UID: \"3d25f945-dabe-42ac-af03-f36dfbd0e213\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ndz9d" Jul 14 22:16:07.541285 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:07.541252 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ndz9d" Jul 14 22:16:07.679452 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:07.679428 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ndz9d"] Jul 14 22:16:07.681364 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:16:07.681326 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d25f945_dabe_42ac_af03_f36dfbd0e213.slice/crio-b20e690efdbd75addadeea9b471fef0dfe31ea6140cf31b6861910a9c0cdeab1 WatchSource:0}: Error finding container b20e690efdbd75addadeea9b471fef0dfe31ea6140cf31b6861910a9c0cdeab1: Status 404 returned error can't find the container with id b20e690efdbd75addadeea9b471fef0dfe31ea6140cf31b6861910a9c0cdeab1 Jul 14 22:16:07.751191 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:07.751109 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ndz9d" event={"ID":"3d25f945-dabe-42ac-af03-f36dfbd0e213","Type":"ContainerStarted","Data":"b20e690efdbd75addadeea9b471fef0dfe31ea6140cf31b6861910a9c0cdeab1"} Jul 14 22:16:09.761422 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:09.761373 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ndz9d" event={"ID":"3d25f945-dabe-42ac-af03-f36dfbd0e213","Type":"ContainerStarted","Data":"0f23ca3cbb963b61ee5653017cb5d28b8c26e8f0cd00e951a2d2f79669aa1dbc"} Jul 14 22:16:09.781730 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:09.781686 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ndz9d" podStartSLOduration=1.1660379650000001 podStartE2EDuration="2.781670297s" totalImagesPullingTime="1.615632332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:16:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:16:09.779613796 +0000 UTC m=+400.138760234" watchObservedRunningTime="2026-07-14 22:16:09.781670297 +0000 UTC m=+400.140816733" Jul 14 22:16:10.580521 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:10.580490 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:10.681176 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:10.681144 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:12.330574 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:12.330524 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9vvb6"] Jul 14 22:16:12.333791 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:12.333774 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9vvb6" Jul 14 22:16:12.337099 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:12.337074 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jj9nv\"" Jul 14 22:16:12.337211 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:12.337124 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 14 22:16:12.337211 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:12.337124 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 14 22:16:12.356024 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:12.355999 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9vvb6"] Jul 14 22:16:12.433988 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:12.433946 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a762a844-c8ad-42df-9722-b805efd0adba-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9vvb6\" (UID: \"a762a844-c8ad-42df-9722-b805efd0adba\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9vvb6" Jul 14 22:16:12.433988 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:12.433991 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a762a844-c8ad-42df-9722-b805efd0adba-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9vvb6\" (UID: \"a762a844-c8ad-42df-9722-b805efd0adba\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9vvb6" Jul 14 22:16:12.434219 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:12.434086 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26s48\" (UniqueName: \"kubernetes.io/projected/a762a844-c8ad-42df-9722-b805efd0adba-kube-api-access-26s48\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9vvb6\" (UID: \"a762a844-c8ad-42df-9722-b805efd0adba\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9vvb6" Jul 14 22:16:12.524342 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:12.524312 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:12.534534 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:12.534507 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26s48\" (UniqueName: \"kubernetes.io/projected/a762a844-c8ad-42df-9722-b805efd0adba-kube-api-access-26s48\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9vvb6\" (UID: \"a762a844-c8ad-42df-9722-b805efd0adba\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9vvb6" Jul 14 22:16:12.534696 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:12.534583 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a762a844-c8ad-42df-9722-b805efd0adba-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9vvb6\" (UID: \"a762a844-c8ad-42df-9722-b805efd0adba\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9vvb6" Jul 14 22:16:12.534696 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:12.534611 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a762a844-c8ad-42df-9722-b805efd0adba-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9vvb6\" (UID: \"a762a844-c8ad-42df-9722-b805efd0adba\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9vvb6" Jul 14 22:16:12.534941 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:12.534920 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a762a844-c8ad-42df-9722-b805efd0adba-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9vvb6\" (UID: \"a762a844-c8ad-42df-9722-b805efd0adba\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9vvb6" Jul 14 22:16:12.535005 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:12.534968 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a762a844-c8ad-42df-9722-b805efd0adba-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9vvb6\" (UID: \"a762a844-c8ad-42df-9722-b805efd0adba\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9vvb6" Jul 14 22:16:12.546942 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:12.546913 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26s48\" (UniqueName: \"kubernetes.io/projected/a762a844-c8ad-42df-9722-b805efd0adba-kube-api-access-26s48\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9vvb6\" (UID: \"a762a844-c8ad-42df-9722-b805efd0adba\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9vvb6" Jul 14 22:16:12.642930 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:12.642853 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9vvb6" Jul 14 22:16:12.776665 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:12.776639 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9vvb6"] Jul 14 22:16:12.778566 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:16:12.778529 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda762a844_c8ad_42df_9722_b805efd0adba.slice/crio-55f0631aa6844180b5fcf485c86cb42c8d65ebfb5abc8b4102c468268747deb6 WatchSource:0}: Error finding container 55f0631aa6844180b5fcf485c86cb42c8d65ebfb5abc8b4102c468268747deb6: Status 404 returned error can't find the container with id 55f0631aa6844180b5fcf485c86cb42c8d65ebfb5abc8b4102c468268747deb6 Jul 14 22:16:13.782065 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:13.782036 2550 generic.go:356] "Generic (PLEG): container finished" podID="a762a844-c8ad-42df-9722-b805efd0adba" containerID="b09bf4e729812ee4d66705d087cdc1b8b3165e9e5cc03d7d6ac913a3105cdc47" exitCode=0 Jul 14 22:16:13.782437 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:13.782131 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9vvb6" event={"ID":"a762a844-c8ad-42df-9722-b805efd0adba","Type":"ContainerDied","Data":"b09bf4e729812ee4d66705d087cdc1b8b3165e9e5cc03d7d6ac913a3105cdc47"} Jul 14 22:16:13.782437 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:13.782227 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9vvb6" event={"ID":"a762a844-c8ad-42df-9722-b805efd0adba","Type":"ContainerStarted","Data":"55f0631aa6844180b5fcf485c86cb42c8d65ebfb5abc8b4102c468268747deb6"} Jul 14 22:16:14.788765 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:14.788731 2550 generic.go:356] "Generic (PLEG): container finished" podID="a762a844-c8ad-42df-9722-b805efd0adba" containerID="da0c7a099c88944f45c4f2c637a1c0fc0b72e514c6b01933c954f1a2c918de48" exitCode=0 Jul 14 22:16:14.789123 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:14.788813 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9vvb6" event={"ID":"a762a844-c8ad-42df-9722-b805efd0adba","Type":"ContainerDied","Data":"da0c7a099c88944f45c4f2c637a1c0fc0b72e514c6b01933c954f1a2c918de48"} Jul 14 22:16:15.794783 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:15.794703 2550 generic.go:356] "Generic (PLEG): container finished" podID="a762a844-c8ad-42df-9722-b805efd0adba" containerID="8d3a9e0d5d09fae059646601ed04fa423f320c1e81b2ccaeb4a78ac3df84aa48" exitCode=0 Jul 14 22:16:15.795123 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:15.794781 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9vvb6" event={"ID":"a762a844-c8ad-42df-9722-b805efd0adba","Type":"ContainerDied","Data":"8d3a9e0d5d09fae059646601ed04fa423f320c1e81b2ccaeb4a78ac3df84aa48"} Jul 14 22:16:16.937079 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:16.937051 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9vvb6" Jul 14 22:16:16.968581 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:16.968546 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a762a844-c8ad-42df-9722-b805efd0adba-bundle\") pod \"a762a844-c8ad-42df-9722-b805efd0adba\" (UID: \"a762a844-c8ad-42df-9722-b805efd0adba\") " Jul 14 22:16:16.968736 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:16.968637 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a762a844-c8ad-42df-9722-b805efd0adba-util\") pod \"a762a844-c8ad-42df-9722-b805efd0adba\" (UID: \"a762a844-c8ad-42df-9722-b805efd0adba\") " Jul 14 22:16:16.968736 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:16.968660 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26s48\" (UniqueName: \"kubernetes.io/projected/a762a844-c8ad-42df-9722-b805efd0adba-kube-api-access-26s48\") pod \"a762a844-c8ad-42df-9722-b805efd0adba\" (UID: \"a762a844-c8ad-42df-9722-b805efd0adba\") " Jul 14 22:16:16.969251 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:16.969222 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a762a844-c8ad-42df-9722-b805efd0adba-bundle" (OuterVolumeSpecName: "bundle") pod "a762a844-c8ad-42df-9722-b805efd0adba" (UID: "a762a844-c8ad-42df-9722-b805efd0adba"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:16:16.970817 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:16.970794 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a762a844-c8ad-42df-9722-b805efd0adba-kube-api-access-26s48" (OuterVolumeSpecName: "kube-api-access-26s48") pod "a762a844-c8ad-42df-9722-b805efd0adba" (UID: "a762a844-c8ad-42df-9722-b805efd0adba"). InnerVolumeSpecName "kube-api-access-26s48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:16:16.974260 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:16.974238 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a762a844-c8ad-42df-9722-b805efd0adba-util" (OuterVolumeSpecName: "util") pod "a762a844-c8ad-42df-9722-b805efd0adba" (UID: "a762a844-c8ad-42df-9722-b805efd0adba"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:16:17.069914 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:17.069841 2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a762a844-c8ad-42df-9722-b805efd0adba-util\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:16:17.069914 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:17.069866 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26s48\" (UniqueName: \"kubernetes.io/projected/a762a844-c8ad-42df-9722-b805efd0adba-kube-api-access-26s48\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:16:17.069914 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:17.069875 2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a762a844-c8ad-42df-9722-b805efd0adba-bundle\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:16:17.807053 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:17.807019 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9vvb6" event={"ID":"a762a844-c8ad-42df-9722-b805efd0adba","Type":"ContainerDied","Data":"55f0631aa6844180b5fcf485c86cb42c8d65ebfb5abc8b4102c468268747deb6"} Jul 14 22:16:17.807243 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:17.807074 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55f0631aa6844180b5fcf485c86cb42c8d65ebfb5abc8b4102c468268747deb6" Jul 14 22:16:17.807243 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:17.807029 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9vvb6" Jul 14 22:16:24.599450 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:24.599417 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:24.698983 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:24.698954 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:25.173876 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:25.173850 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:25.751891 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:25.751856 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:25.882035 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:25.882008 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:25.891766 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:25.891731 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86c5998494-g2p5d"] Jul 14 22:16:25.892072 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:25.892056 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a762a844-c8ad-42df-9722-b805efd0adba" containerName="pull" Jul 14 22:16:25.892154 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:25.892075 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a762a844-c8ad-42df-9722-b805efd0adba" containerName="pull" Jul 14 22:16:25.892154 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:25.892089 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a762a844-c8ad-42df-9722-b805efd0adba" containerName="extract" Jul 14 22:16:25.892154 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:25.892097 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a762a844-c8ad-42df-9722-b805efd0adba" containerName="extract" Jul 14 22:16:25.892154 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:25.892117 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a762a844-c8ad-42df-9722-b805efd0adba" containerName="util" Jul 14 22:16:25.892154 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:25.892126 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a762a844-c8ad-42df-9722-b805efd0adba" containerName="util" Jul 14 22:16:25.892389 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:25.892205 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="a762a844-c8ad-42df-9722-b805efd0adba" containerName="extract" Jul 14 22:16:25.896048 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:25.896025 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-g2p5d" Jul 14 22:16:25.906287 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:25.906268 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-9kkhx\"" Jul 14 22:16:25.906382 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:25.906301 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 14 22:16:25.906547 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:25.906530 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 14 22:16:25.908143 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:25.908126 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 14 22:16:25.908198 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:25.908151 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 14 22:16:25.937004 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:25.936974 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ccad00f-8074-4fb0-bfa9-c89d07edede6-tmp\") pod \"opendatahub-operator-controller-manager-86c5998494-g2p5d\" (UID: \"5ccad00f-8074-4fb0-bfa9-c89d07edede6\") " pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-g2p5d" Jul 14 22:16:25.937169 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:25.937028 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5ccad00f-8074-4fb0-bfa9-c89d07edede6-manifests\") pod \"opendatahub-operator-controller-manager-86c5998494-g2p5d\" (UID: \"5ccad00f-8074-4fb0-bfa9-c89d07edede6\") " pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-g2p5d" Jul 14 22:16:25.937169 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:25.937101 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5ccad00f-8074-4fb0-bfa9-c89d07edede6-webhook-cert\") pod \"opendatahub-operator-controller-manager-86c5998494-g2p5d\" (UID: \"5ccad00f-8074-4fb0-bfa9-c89d07edede6\") " pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-g2p5d" Jul 14 22:16:25.937169 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:25.937145 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttv58\" (UniqueName: \"kubernetes.io/projected/5ccad00f-8074-4fb0-bfa9-c89d07edede6-kube-api-access-ttv58\") pod \"opendatahub-operator-controller-manager-86c5998494-g2p5d\" (UID: \"5ccad00f-8074-4fb0-bfa9-c89d07edede6\") " pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-g2p5d" Jul 14 22:16:25.937169 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:25.937165 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5ccad00f-8074-4fb0-bfa9-c89d07edede6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86c5998494-g2p5d\" (UID: \"5ccad00f-8074-4fb0-bfa9-c89d07edede6\") " pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-g2p5d" Jul 14 22:16:25.944347 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:25.944321 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86c5998494-g2p5d"] Jul 14 22:16:26.038307 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:26.038221 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5ccad00f-8074-4fb0-bfa9-c89d07edede6-webhook-cert\") pod \"opendatahub-operator-controller-manager-86c5998494-g2p5d\" (UID: \"5ccad00f-8074-4fb0-bfa9-c89d07edede6\") " pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-g2p5d" Jul 14 22:16:26.038307 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:26.038266 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttv58\" (UniqueName: \"kubernetes.io/projected/5ccad00f-8074-4fb0-bfa9-c89d07edede6-kube-api-access-ttv58\") pod \"opendatahub-operator-controller-manager-86c5998494-g2p5d\" (UID: \"5ccad00f-8074-4fb0-bfa9-c89d07edede6\") " pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-g2p5d" Jul 14 22:16:26.038307 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:26.038296 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5ccad00f-8074-4fb0-bfa9-c89d07edede6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86c5998494-g2p5d\" (UID: \"5ccad00f-8074-4fb0-bfa9-c89d07edede6\") " pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-g2p5d" Jul 14 22:16:26.038601 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:26.038320 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ccad00f-8074-4fb0-bfa9-c89d07edede6-tmp\") pod \"opendatahub-operator-controller-manager-86c5998494-g2p5d\" (UID: \"5ccad00f-8074-4fb0-bfa9-c89d07edede6\") " pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-g2p5d" Jul 14 22:16:26.038601 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:26.038378 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5ccad00f-8074-4fb0-bfa9-c89d07edede6-manifests\") pod \"opendatahub-operator-controller-manager-86c5998494-g2p5d\" (UID: \"5ccad00f-8074-4fb0-bfa9-c89d07edede6\") " pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-g2p5d" Jul 14 22:16:26.038815 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:26.038794 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5ccad00f-8074-4fb0-bfa9-c89d07edede6-manifests\") pod \"opendatahub-operator-controller-manager-86c5998494-g2p5d\" (UID: \"5ccad00f-8074-4fb0-bfa9-c89d07edede6\") " pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-g2p5d" Jul 14 22:16:26.040651 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:26.040626 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ccad00f-8074-4fb0-bfa9-c89d07edede6-tmp\") pod \"opendatahub-operator-controller-manager-86c5998494-g2p5d\" (UID: \"5ccad00f-8074-4fb0-bfa9-c89d07edede6\") " pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-g2p5d" Jul 14 22:16:26.040837 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:26.040820 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5ccad00f-8074-4fb0-bfa9-c89d07edede6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86c5998494-g2p5d\" (UID: \"5ccad00f-8074-4fb0-bfa9-c89d07edede6\") " pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-g2p5d" Jul 14 22:16:26.040879 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:26.040823 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5ccad00f-8074-4fb0-bfa9-c89d07edede6-webhook-cert\") pod \"opendatahub-operator-controller-manager-86c5998494-g2p5d\" (UID: \"5ccad00f-8074-4fb0-bfa9-c89d07edede6\") " pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-g2p5d" Jul 14 22:16:26.051140 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:26.051117 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttv58\" (UniqueName: \"kubernetes.io/projected/5ccad00f-8074-4fb0-bfa9-c89d07edede6-kube-api-access-ttv58\") pod \"opendatahub-operator-controller-manager-86c5998494-g2p5d\" (UID: \"5ccad00f-8074-4fb0-bfa9-c89d07edede6\") " pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-g2p5d" Jul 14 22:16:26.185811 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:26.185782 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:26.205304 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:26.205278 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-g2p5d" Jul 14 22:16:26.314434 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:26.314362 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:26.391582 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:26.391560 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86c5998494-g2p5d"] Jul 14 22:16:26.393764 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:16:26.393738 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ccad00f_8074_4fb0_bfa9_c89d07edede6.slice/crio-7fb75eaea77e9cea541df9b9f7a09d887b4f48c6750ab310281feb5d2e94d55b WatchSource:0}: Error finding container 7fb75eaea77e9cea541df9b9f7a09d887b4f48c6750ab310281feb5d2e94d55b: Status 404 returned error can't find the container with id 7fb75eaea77e9cea541df9b9f7a09d887b4f48c6750ab310281feb5d2e94d55b Jul 14 22:16:26.449150 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:26.449122 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:26.557732 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:26.557640 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:26.667830 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:26.667801 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:26.845169 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:26.845086 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-g2p5d" event={"ID":"5ccad00f-8074-4fb0-bfa9-c89d07edede6","Type":"ContainerStarted","Data":"7fb75eaea77e9cea541df9b9f7a09d887b4f48c6750ab310281feb5d2e94d55b"} Jul 14 22:16:29.862112 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:29.862033 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-g2p5d" event={"ID":"5ccad00f-8074-4fb0-bfa9-c89d07edede6","Type":"ContainerStarted","Data":"65a73b7cd3afd7fd3e87ceba7ddd317daf0d5a7bf3a3f50bc60b1e030440b674"} Jul 14 22:16:30.867261 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:30.867222 2550 generic.go:356] "Generic (PLEG): container finished" podID="5ccad00f-8074-4fb0-bfa9-c89d07edede6" containerID="65a73b7cd3afd7fd3e87ceba7ddd317daf0d5a7bf3a3f50bc60b1e030440b674" exitCode=0 Jul 14 22:16:30.867665 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:30.867309 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-g2p5d" event={"ID":"5ccad00f-8074-4fb0-bfa9-c89d07edede6","Type":"ContainerDied","Data":"65a73b7cd3afd7fd3e87ceba7ddd317daf0d5a7bf3a3f50bc60b1e030440b674"} Jul 14 22:16:31.873461 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:31.873409 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-g2p5d" event={"ID":"5ccad00f-8074-4fb0-bfa9-c89d07edede6","Type":"ContainerStarted","Data":"84c6bb599cbf44cb81e5d20e7f5fdcd3a5525da4760df54286dac10be30310aa"} Jul 14 22:16:31.873878 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:31.873509 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-g2p5d" Jul 14 22:16:31.919909 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:31.919851 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-g2p5d" podStartSLOduration=2.383263912 podStartE2EDuration="6.919835766s" totalImagesPullingTime="3.49584242s" totalInitContainerRuntime="1.040729434s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:16:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:16:31.918475466 +0000 UTC m=+422.277621903" watchObservedRunningTime="2026-07-14 22:16:31.919835766 +0000 UTC m=+422.278982202" Jul 14 22:16:38.739721 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:38.739683 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:38.837841 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:38.837812 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:42.880649 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:42.880620 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-86c5998494-g2p5d" Jul 14 22:16:45.096907 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:45.096875 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-66545d98-lq6fq"] Jul 14 22:16:45.103716 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:45.103692 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-66545d98-lq6fq" Jul 14 22:16:45.107637 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:45.107610 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mtqf9\"" Jul 14 22:16:45.107770 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:45.107661 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 14 22:16:45.107770 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:45.107738 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 14 22:16:45.107770 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:45.107747 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 14 22:16:45.124046 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:45.124021 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-66545d98-lq6fq"] Jul 14 22:16:45.198804 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:45.198770 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2312297c-7f42-47c9-b0cf-a1474ae6e6d4-cert\") pod \"lws-controller-manager-66545d98-lq6fq\" (UID: \"2312297c-7f42-47c9-b0cf-a1474ae6e6d4\") " pod="openshift-lws-operator/lws-controller-manager-66545d98-lq6fq" Jul 14 22:16:45.198977 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:45.198820 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2312297c-7f42-47c9-b0cf-a1474ae6e6d4-metrics-cert\") pod \"lws-controller-manager-66545d98-lq6fq\" (UID: \"2312297c-7f42-47c9-b0cf-a1474ae6e6d4\") " pod="openshift-lws-operator/lws-controller-manager-66545d98-lq6fq" Jul 14 22:16:45.198977 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:45.198840 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2312297c-7f42-47c9-b0cf-a1474ae6e6d4-manager-config\") pod \"lws-controller-manager-66545d98-lq6fq\" (UID: \"2312297c-7f42-47c9-b0cf-a1474ae6e6d4\") " pod="openshift-lws-operator/lws-controller-manager-66545d98-lq6fq" Jul 14 22:16:45.198977 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:45.198864 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk8mt\" (UniqueName: \"kubernetes.io/projected/2312297c-7f42-47c9-b0cf-a1474ae6e6d4-kube-api-access-lk8mt\") pod \"lws-controller-manager-66545d98-lq6fq\" (UID: \"2312297c-7f42-47c9-b0cf-a1474ae6e6d4\") " pod="openshift-lws-operator/lws-controller-manager-66545d98-lq6fq" Jul 14 22:16:45.299869 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:45.299823 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2312297c-7f42-47c9-b0cf-a1474ae6e6d4-cert\") pod \"lws-controller-manager-66545d98-lq6fq\" (UID: \"2312297c-7f42-47c9-b0cf-a1474ae6e6d4\") " pod="openshift-lws-operator/lws-controller-manager-66545d98-lq6fq" Jul 14 22:16:45.299869 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:45.299875 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2312297c-7f42-47c9-b0cf-a1474ae6e6d4-metrics-cert\") pod \"lws-controller-manager-66545d98-lq6fq\" (UID: \"2312297c-7f42-47c9-b0cf-a1474ae6e6d4\") " pod="openshift-lws-operator/lws-controller-manager-66545d98-lq6fq" Jul 14 22:16:45.300113 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:45.299898 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2312297c-7f42-47c9-b0cf-a1474ae6e6d4-manager-config\") pod \"lws-controller-manager-66545d98-lq6fq\" (UID: \"2312297c-7f42-47c9-b0cf-a1474ae6e6d4\") " pod="openshift-lws-operator/lws-controller-manager-66545d98-lq6fq" Jul 14 22:16:45.300113 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:45.299920 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk8mt\" (UniqueName: \"kubernetes.io/projected/2312297c-7f42-47c9-b0cf-a1474ae6e6d4-kube-api-access-lk8mt\") pod \"lws-controller-manager-66545d98-lq6fq\" (UID: \"2312297c-7f42-47c9-b0cf-a1474ae6e6d4\") " pod="openshift-lws-operator/lws-controller-manager-66545d98-lq6fq" Jul 14 22:16:45.300717 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:45.300685 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2312297c-7f42-47c9-b0cf-a1474ae6e6d4-manager-config\") pod \"lws-controller-manager-66545d98-lq6fq\" (UID: \"2312297c-7f42-47c9-b0cf-a1474ae6e6d4\") " pod="openshift-lws-operator/lws-controller-manager-66545d98-lq6fq" Jul 14 22:16:45.302896 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:45.302873 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2312297c-7f42-47c9-b0cf-a1474ae6e6d4-cert\") pod \"lws-controller-manager-66545d98-lq6fq\" (UID: \"2312297c-7f42-47c9-b0cf-a1474ae6e6d4\") " pod="openshift-lws-operator/lws-controller-manager-66545d98-lq6fq" Jul 14 22:16:45.302995 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:45.302894 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2312297c-7f42-47c9-b0cf-a1474ae6e6d4-metrics-cert\") pod \"lws-controller-manager-66545d98-lq6fq\" (UID: \"2312297c-7f42-47c9-b0cf-a1474ae6e6d4\") " pod="openshift-lws-operator/lws-controller-manager-66545d98-lq6fq" Jul 14 22:16:45.325311 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:45.325281 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk8mt\" (UniqueName: \"kubernetes.io/projected/2312297c-7f42-47c9-b0cf-a1474ae6e6d4-kube-api-access-lk8mt\") pod \"lws-controller-manager-66545d98-lq6fq\" (UID: \"2312297c-7f42-47c9-b0cf-a1474ae6e6d4\") " pod="openshift-lws-operator/lws-controller-manager-66545d98-lq6fq" Jul 14 22:16:45.413079 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:45.413054 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-66545d98-lq6fq" Jul 14 22:16:45.613160 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:45.613127 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-66545d98-lq6fq"] Jul 14 22:16:45.621143 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:16:45.621120 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2312297c_7f42_47c9_b0cf_a1474ae6e6d4.slice/crio-da0b26f28000599c482316713720e6a3e48e96f35b3e7c37558382a79f02a6be WatchSource:0}: Error finding container da0b26f28000599c482316713720e6a3e48e96f35b3e7c37558382a79f02a6be: Status 404 returned error can't find the container with id da0b26f28000599c482316713720e6a3e48e96f35b3e7c37558382a79f02a6be Jul 14 22:16:45.678399 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:45.678306 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:45.937893 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:45.937811 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-66545d98-lq6fq" event={"ID":"2312297c-7f42-47c9-b0cf-a1474ae6e6d4","Type":"ContainerStarted","Data":"da0b26f28000599c482316713720e6a3e48e96f35b3e7c37558382a79f02a6be"} Jul 14 22:16:46.036348 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:46.036312 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:46.526401 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:46.526361 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:46.725501 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:46.725448 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:46.978933 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:46.978874 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:47.240278 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:47.240207 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:47.592299 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:47.592226 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:47.798891 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:47.798855 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:47.956077 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:47.956038 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-66545d98-lq6fq" event={"ID":"2312297c-7f42-47c9-b0cf-a1474ae6e6d4","Type":"ContainerStarted","Data":"da27df231160bee7851258eb2091b386180e46adbf5af4b3c6b21d9fbc5542cf"} Jul 14 22:16:47.956253 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:47.956222 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-66545d98-lq6fq" Jul 14 22:16:48.600820 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:48.600785 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:48.761455 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:48.761416 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:49.571163 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:49.571131 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:49.730368 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:49.730330 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:51.064588 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:51.064434 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-66545d98-lq6fq" podStartSLOduration=4.562894443 podStartE2EDuration="6.064414668s" totalImagesPullingTime="1.501520225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:16:47.995888523 +0000 UTC m=+438.355034997" watchObservedRunningTime="2026-07-14 22:16:51.064414668 +0000 UTC m=+441.423561096" Jul 14 22:16:51.066076 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:51.066055 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d5d545ff5-9t7ht"] Jul 14 22:16:51.069189 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:51.069169 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d5d545ff5-9t7ht" Jul 14 22:16:51.069381 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:51.069359 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:51.089308 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:51.089276 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d5d545ff5-9t7ht"] Jul 14 22:16:51.153234 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:51.153207 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cf8274b-6860-426f-98e9-a2af0eb5d8ed-oauth-serving-cert\") pod \"console-7d5d545ff5-9t7ht\" (UID: \"9cf8274b-6860-426f-98e9-a2af0eb5d8ed\") " pod="openshift-console/console-7d5d545ff5-9t7ht" Jul 14 22:16:51.153398 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:51.153241 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cf8274b-6860-426f-98e9-a2af0eb5d8ed-console-serving-cert\") pod \"console-7d5d545ff5-9t7ht\" (UID: \"9cf8274b-6860-426f-98e9-a2af0eb5d8ed\") " pod="openshift-console/console-7d5d545ff5-9t7ht" Jul 14 22:16:51.153398 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:51.153268 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cf8274b-6860-426f-98e9-a2af0eb5d8ed-console-config\") pod \"console-7d5d545ff5-9t7ht\" (UID: \"9cf8274b-6860-426f-98e9-a2af0eb5d8ed\") " pod="openshift-console/console-7d5d545ff5-9t7ht" Jul 14 22:16:51.153398 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:51.153291 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cf8274b-6860-426f-98e9-a2af0eb5d8ed-service-ca\") pod \"console-7d5d545ff5-9t7ht\" (UID: \"9cf8274b-6860-426f-98e9-a2af0eb5d8ed\") " pod="openshift-console/console-7d5d545ff5-9t7ht" Jul 14 22:16:51.153398 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:51.153325 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cf8274b-6860-426f-98e9-a2af0eb5d8ed-trusted-ca-bundle\") pod \"console-7d5d545ff5-9t7ht\" (UID: \"9cf8274b-6860-426f-98e9-a2af0eb5d8ed\") " pod="openshift-console/console-7d5d545ff5-9t7ht" Jul 14 22:16:51.153398 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:51.153359 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cf8274b-6860-426f-98e9-a2af0eb5d8ed-console-oauth-config\") pod \"console-7d5d545ff5-9t7ht\" (UID: \"9cf8274b-6860-426f-98e9-a2af0eb5d8ed\") " pod="openshift-console/console-7d5d545ff5-9t7ht" Jul 14 22:16:51.153398 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:51.153377 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75xpv\" (UniqueName: \"kubernetes.io/projected/9cf8274b-6860-426f-98e9-a2af0eb5d8ed-kube-api-access-75xpv\") pod \"console-7d5d545ff5-9t7ht\" (UID: \"9cf8274b-6860-426f-98e9-a2af0eb5d8ed\") " pod="openshift-console/console-7d5d545ff5-9t7ht" Jul 14 22:16:51.195421 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:51.195391 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:51.254502 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:51.254470 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cf8274b-6860-426f-98e9-a2af0eb5d8ed-oauth-serving-cert\") pod \"console-7d5d545ff5-9t7ht\" (UID: \"9cf8274b-6860-426f-98e9-a2af0eb5d8ed\") " pod="openshift-console/console-7d5d545ff5-9t7ht" Jul 14 22:16:51.254502 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:51.254507 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cf8274b-6860-426f-98e9-a2af0eb5d8ed-console-serving-cert\") pod \"console-7d5d545ff5-9t7ht\" (UID: \"9cf8274b-6860-426f-98e9-a2af0eb5d8ed\") " pod="openshift-console/console-7d5d545ff5-9t7ht" Jul 14 22:16:51.254736 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:51.254536 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cf8274b-6860-426f-98e9-a2af0eb5d8ed-console-config\") pod \"console-7d5d545ff5-9t7ht\" (UID: \"9cf8274b-6860-426f-98e9-a2af0eb5d8ed\") " pod="openshift-console/console-7d5d545ff5-9t7ht" Jul 14 22:16:51.254736 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:51.254577 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cf8274b-6860-426f-98e9-a2af0eb5d8ed-service-ca\") pod \"console-7d5d545ff5-9t7ht\" (UID: \"9cf8274b-6860-426f-98e9-a2af0eb5d8ed\") " pod="openshift-console/console-7d5d545ff5-9t7ht" Jul 14 22:16:51.254846 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:51.254755 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cf8274b-6860-426f-98e9-a2af0eb5d8ed-trusted-ca-bundle\") pod \"console-7d5d545ff5-9t7ht\" (UID: \"9cf8274b-6860-426f-98e9-a2af0eb5d8ed\") " pod="openshift-console/console-7d5d545ff5-9t7ht" Jul 14 22:16:51.254846 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:51.254836 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cf8274b-6860-426f-98e9-a2af0eb5d8ed-console-oauth-config\") pod \"console-7d5d545ff5-9t7ht\" (UID: \"9cf8274b-6860-426f-98e9-a2af0eb5d8ed\") " pod="openshift-console/console-7d5d545ff5-9t7ht" Jul 14 22:16:51.254940 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:51.254869 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75xpv\" (UniqueName: \"kubernetes.io/projected/9cf8274b-6860-426f-98e9-a2af0eb5d8ed-kube-api-access-75xpv\") pod \"console-7d5d545ff5-9t7ht\" (UID: \"9cf8274b-6860-426f-98e9-a2af0eb5d8ed\") " pod="openshift-console/console-7d5d545ff5-9t7ht" Jul 14 22:16:51.255354 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:51.255329 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cf8274b-6860-426f-98e9-a2af0eb5d8ed-service-ca\") pod \"console-7d5d545ff5-9t7ht\" (UID: \"9cf8274b-6860-426f-98e9-a2af0eb5d8ed\") " pod="openshift-console/console-7d5d545ff5-9t7ht" Jul 14 22:16:51.255535 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:51.255371 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cf8274b-6860-426f-98e9-a2af0eb5d8ed-console-config\") pod \"console-7d5d545ff5-9t7ht\" (UID: \"9cf8274b-6860-426f-98e9-a2af0eb5d8ed\") " pod="openshift-console/console-7d5d545ff5-9t7ht" Jul 14 22:16:51.255535 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:51.255352 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cf8274b-6860-426f-98e9-a2af0eb5d8ed-oauth-serving-cert\") pod \"console-7d5d545ff5-9t7ht\" (UID: \"9cf8274b-6860-426f-98e9-a2af0eb5d8ed\") " pod="openshift-console/console-7d5d545ff5-9t7ht" Jul 14 22:16:51.255739 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:51.255713 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cf8274b-6860-426f-98e9-a2af0eb5d8ed-trusted-ca-bundle\") pod \"console-7d5d545ff5-9t7ht\" (UID: \"9cf8274b-6860-426f-98e9-a2af0eb5d8ed\") " pod="openshift-console/console-7d5d545ff5-9t7ht" Jul 14 22:16:51.257185 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:51.257163 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cf8274b-6860-426f-98e9-a2af0eb5d8ed-console-serving-cert\") pod \"console-7d5d545ff5-9t7ht\" (UID: \"9cf8274b-6860-426f-98e9-a2af0eb5d8ed\") " pod="openshift-console/console-7d5d545ff5-9t7ht" Jul 14 22:16:51.257291 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:51.257250 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cf8274b-6860-426f-98e9-a2af0eb5d8ed-console-oauth-config\") pod \"console-7d5d545ff5-9t7ht\" (UID: \"9cf8274b-6860-426f-98e9-a2af0eb5d8ed\") " pod="openshift-console/console-7d5d545ff5-9t7ht" Jul 14 22:16:51.265723 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:51.265697 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75xpv\" (UniqueName: \"kubernetes.io/projected/9cf8274b-6860-426f-98e9-a2af0eb5d8ed-kube-api-access-75xpv\") pod \"console-7d5d545ff5-9t7ht\" (UID: \"9cf8274b-6860-426f-98e9-a2af0eb5d8ed\") " pod="openshift-console/console-7d5d545ff5-9t7ht" Jul 14 22:16:51.379295 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:51.379205 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d5d545ff5-9t7ht" Jul 14 22:16:51.515114 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:51.515084 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d5d545ff5-9t7ht"] Jul 14 22:16:51.519942 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:16:51.519917 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cf8274b_6860_426f_98e9_a2af0eb5d8ed.slice/crio-399167dbd7efefe7e7b995493c04b00c12dd5b8ef31e60c4bcf71a07be757cf4 WatchSource:0}: Error finding container 399167dbd7efefe7e7b995493c04b00c12dd5b8ef31e60c4bcf71a07be757cf4: Status 404 returned error can't find the container with id 399167dbd7efefe7e7b995493c04b00c12dd5b8ef31e60c4bcf71a07be757cf4 Jul 14 22:16:51.976573 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:51.976517 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d5d545ff5-9t7ht" event={"ID":"9cf8274b-6860-426f-98e9-a2af0eb5d8ed","Type":"ContainerStarted","Data":"1d48d21ac51ed99156c0cbef7e941d176f6c5142e73a5a4ef0bd9157941b26f8"} Jul 14 22:16:51.976757 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:51.976593 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d5d545ff5-9t7ht" event={"ID":"9cf8274b-6860-426f-98e9-a2af0eb5d8ed","Type":"ContainerStarted","Data":"399167dbd7efefe7e7b995493c04b00c12dd5b8ef31e60c4bcf71a07be757cf4"} Jul 14 22:16:51.997525 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:51.997476 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d5d545ff5-9t7ht" podStartSLOduration=0.997460748 podStartE2EDuration="997.460748ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:16:51.995926418 +0000 UTC m=+442.355072858" watchObservedRunningTime="2026-07-14 22:16:51.997460748 +0000 UTC m=+442.356607185" Jul 14 22:16:54.058325 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:54.058292 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:54.259954 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:54.259905 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:54.706376 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:54.706341 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-484t2"] Jul 14 22:16:54.709281 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:54.709260 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-484t2" Jul 14 22:16:54.712108 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:54.712086 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 14 22:16:54.712221 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:54.712087 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-g64zc\"" Jul 14 22:16:54.712542 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:54.712515 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 14 22:16:54.713161 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:54.713145 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 14 22:16:54.713800 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:54.713772 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 14 22:16:54.714709 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:54.714691 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 14 22:16:54.714980 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:54.714963 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 14 22:16:54.729498 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:54.729475 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-484t2"] Jul 14 22:16:54.887870 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:54.887835 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/140164c5-c024-4fb4-91da-9ff07e0c9406-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-484t2\" (UID: \"140164c5-c024-4fb4-91da-9ff07e0c9406\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-484t2" Jul 14 22:16:54.887870 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:54.887872 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/140164c5-c024-4fb4-91da-9ff07e0c9406-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-484t2\" (UID: \"140164c5-c024-4fb4-91da-9ff07e0c9406\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-484t2" Jul 14 22:16:54.888080 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:54.887893 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/140164c5-c024-4fb4-91da-9ff07e0c9406-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-484t2\" (UID: \"140164c5-c024-4fb4-91da-9ff07e0c9406\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-484t2" Jul 14 22:16:54.888080 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:54.887992 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/140164c5-c024-4fb4-91da-9ff07e0c9406-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-484t2\" (UID: \"140164c5-c024-4fb4-91da-9ff07e0c9406\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-484t2" Jul 14 22:16:54.888080 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:54.888020 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/140164c5-c024-4fb4-91da-9ff07e0c9406-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-484t2\" (UID: \"140164c5-c024-4fb4-91da-9ff07e0c9406\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-484t2" Jul 14 22:16:54.888080 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:54.888056 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlm8h\" (UniqueName: \"kubernetes.io/projected/140164c5-c024-4fb4-91da-9ff07e0c9406-kube-api-access-wlm8h\") pod \"istiod-openshift-gateway-94bb8fbfd-484t2\" (UID: \"140164c5-c024-4fb4-91da-9ff07e0c9406\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-484t2" Jul 14 22:16:54.888201 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:54.888105 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/140164c5-c024-4fb4-91da-9ff07e0c9406-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-484t2\" (UID: \"140164c5-c024-4fb4-91da-9ff07e0c9406\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-484t2" Jul 14 22:16:54.988517 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:54.988431 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/140164c5-c024-4fb4-91da-9ff07e0c9406-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-484t2\" (UID: \"140164c5-c024-4fb4-91da-9ff07e0c9406\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-484t2" Jul 14 22:16:54.988517 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:54.988470 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/140164c5-c024-4fb4-91da-9ff07e0c9406-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-484t2\" (UID: \"140164c5-c024-4fb4-91da-9ff07e0c9406\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-484t2" Jul 14 22:16:54.988517 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:54.988503 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/140164c5-c024-4fb4-91da-9ff07e0c9406-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-484t2\" (UID: \"140164c5-c024-4fb4-91da-9ff07e0c9406\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-484t2" Jul 14 22:16:54.988802 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:54.988571 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/140164c5-c024-4fb4-91da-9ff07e0c9406-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-484t2\" (UID: \"140164c5-c024-4fb4-91da-9ff07e0c9406\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-484t2" Jul 14 22:16:54.988802 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:54.988600 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/140164c5-c024-4fb4-91da-9ff07e0c9406-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-484t2\" (UID: \"140164c5-c024-4fb4-91da-9ff07e0c9406\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-484t2" Jul 14 22:16:54.988802 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:54.988642 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlm8h\" (UniqueName: \"kubernetes.io/projected/140164c5-c024-4fb4-91da-9ff07e0c9406-kube-api-access-wlm8h\") pod \"istiod-openshift-gateway-94bb8fbfd-484t2\" (UID: \"140164c5-c024-4fb4-91da-9ff07e0c9406\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-484t2" Jul 14 22:16:54.988802 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:54.988693 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/140164c5-c024-4fb4-91da-9ff07e0c9406-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-484t2\" (UID: \"140164c5-c024-4fb4-91da-9ff07e0c9406\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-484t2" Jul 14 22:16:54.989432 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:54.989401 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/140164c5-c024-4fb4-91da-9ff07e0c9406-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-484t2\" (UID: \"140164c5-c024-4fb4-91da-9ff07e0c9406\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-484t2" Jul 14 22:16:54.990952 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:54.990927 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/140164c5-c024-4fb4-91da-9ff07e0c9406-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-484t2\" (UID: \"140164c5-c024-4fb4-91da-9ff07e0c9406\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-484t2" Jul 14 22:16:54.991138 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:54.991120 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/140164c5-c024-4fb4-91da-9ff07e0c9406-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-484t2\" (UID: \"140164c5-c024-4fb4-91da-9ff07e0c9406\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-484t2" Jul 14 22:16:54.991368 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:54.991351 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/140164c5-c024-4fb4-91da-9ff07e0c9406-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-484t2\" (UID: \"140164c5-c024-4fb4-91da-9ff07e0c9406\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-484t2" Jul 14 22:16:54.991412 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:54.991357 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/140164c5-c024-4fb4-91da-9ff07e0c9406-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-484t2\" (UID: \"140164c5-c024-4fb4-91da-9ff07e0c9406\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-484t2" Jul 14 22:16:55.001060 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:55.001034 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/140164c5-c024-4fb4-91da-9ff07e0c9406-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-484t2\" (UID: \"140164c5-c024-4fb4-91da-9ff07e0c9406\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-484t2" Jul 14 22:16:55.001370 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:55.001353 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlm8h\" (UniqueName: \"kubernetes.io/projected/140164c5-c024-4fb4-91da-9ff07e0c9406-kube-api-access-wlm8h\") pod \"istiod-openshift-gateway-94bb8fbfd-484t2\" (UID: \"140164c5-c024-4fb4-91da-9ff07e0c9406\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-484t2" Jul 14 22:16:55.020363 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:55.020344 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-484t2" Jul 14 22:16:55.120466 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:55.120437 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:55.199795 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:55.199768 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-484t2"] Jul 14 22:16:55.202256 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:16:55.202229 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod140164c5_c024_4fb4_91da_9ff07e0c9406.slice/crio-bb08f9221cfc548556a1cf2417bc202acc2653158c0116c2d6bce0a6c11a133b WatchSource:0}: Error finding container bb08f9221cfc548556a1cf2417bc202acc2653158c0116c2d6bce0a6c11a133b: Status 404 returned error can't find the container with id bb08f9221cfc548556a1cf2417bc202acc2653158c0116c2d6bce0a6c11a133b Jul 14 22:16:55.380857 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:55.380767 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:55.578982 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:55.578950 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:55.998457 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:55.998384 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-484t2" event={"ID":"140164c5-c024-4fb4-91da-9ff07e0c9406","Type":"ContainerStarted","Data":"bb08f9221cfc548556a1cf2417bc202acc2653158c0116c2d6bce0a6c11a133b"} Jul 14 22:16:57.528411 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:57.528373 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:16:57.528702 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:57.528441 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 22:16:58.010375 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:58.010341 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-484t2" event={"ID":"140164c5-c024-4fb4-91da-9ff07e0c9406","Type":"ContainerStarted","Data":"dc3fa332b0124d7a890371d917ffb2815fb60ed06d1045e8e5c91a1745e980e4"} Jul 14 22:16:58.010641 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:58.010592 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-484t2" Jul 14 22:16:58.012112 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:58.012090 2550 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-484t2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 22:16:58.012234 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:58.012169 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-484t2" podUID="140164c5-c024-4fb4-91da-9ff07e0c9406" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 22:16:58.078175 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:58.078127 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-484t2" podStartSLOduration=1.75422134 podStartE2EDuration="4.078112214s" totalImagesPullingTime="2.323890874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:16:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:16:58.074122709 +0000 UTC m=+448.433269146" watchObservedRunningTime="2026-07-14 22:16:58.078112214 +0000 UTC m=+448.437258650" Jul 14 22:16:58.893136 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:58.893111 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:58.962973 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:58.962947 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-66545d98-lq6fq" Jul 14 22:16:59.015992 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:59.015967 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-484t2" Jul 14 22:16:59.328949 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:59.328919 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:16:59.571566 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:16:59.571529 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:17:00.681718 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:00.681688 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:17:00.883784 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:00.883756 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:17:01.379866 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:01.379829 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d5d545ff5-9t7ht" Jul 14 22:17:01.380043 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:01.379909 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d5d545ff5-9t7ht" Jul 14 22:17:01.384615 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:01.384592 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d5d545ff5-9t7ht" Jul 14 22:17:02.033326 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:02.033301 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d5d545ff5-9t7ht" Jul 14 22:17:02.123073 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:02.123037 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f664d7795-jhqz7"] Jul 14 22:17:02.283071 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:02.283026 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:17:02.458067 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:02.458034 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:17:02.600441 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:02.600397 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:17:27.143595 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:27.143527 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-f664d7795-jhqz7" podUID="3b386957-e8b9-415f-8097-66cb8a84e9ac" containerName="console" containerID="cri-o://92082ebc7545e685b67a20796df255493fbea42b7a1fc41f3ac738ea44265f10" gracePeriod=15 Jul 14 22:17:27.390928 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:27.390907 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f664d7795-jhqz7" Jul 14 22:17:27.472605 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:27.472570 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b386957-e8b9-415f-8097-66cb8a84e9ac-trusted-ca-bundle\") pod \"3b386957-e8b9-415f-8097-66cb8a84e9ac\" (UID: \"3b386957-e8b9-415f-8097-66cb8a84e9ac\") " Jul 14 22:17:27.472781 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:27.472631 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b386957-e8b9-415f-8097-66cb8a84e9ac-service-ca\") pod \"3b386957-e8b9-415f-8097-66cb8a84e9ac\" (UID: \"3b386957-e8b9-415f-8097-66cb8a84e9ac\") " Jul 14 22:17:27.472781 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:27.472681 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b386957-e8b9-415f-8097-66cb8a84e9ac-console-config\") pod \"3b386957-e8b9-415f-8097-66cb8a84e9ac\" (UID: \"3b386957-e8b9-415f-8097-66cb8a84e9ac\") " Jul 14 22:17:27.472781 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:27.472721 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbjln\" (UniqueName: \"kubernetes.io/projected/3b386957-e8b9-415f-8097-66cb8a84e9ac-kube-api-access-rbjln\") pod \"3b386957-e8b9-415f-8097-66cb8a84e9ac\" (UID: \"3b386957-e8b9-415f-8097-66cb8a84e9ac\") " Jul 14 22:17:27.472781 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:27.472756 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b386957-e8b9-415f-8097-66cb8a84e9ac-oauth-serving-cert\") pod \"3b386957-e8b9-415f-8097-66cb8a84e9ac\" (UID: \"3b386957-e8b9-415f-8097-66cb8a84e9ac\") " Jul 14 22:17:27.473042 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:27.472789 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b386957-e8b9-415f-8097-66cb8a84e9ac-console-oauth-config\") pod \"3b386957-e8b9-415f-8097-66cb8a84e9ac\" (UID: \"3b386957-e8b9-415f-8097-66cb8a84e9ac\") " Jul 14 22:17:27.473042 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:27.472815 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b386957-e8b9-415f-8097-66cb8a84e9ac-console-serving-cert\") pod \"3b386957-e8b9-415f-8097-66cb8a84e9ac\" (UID: \"3b386957-e8b9-415f-8097-66cb8a84e9ac\") " Jul 14 22:17:27.473042 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:27.472899 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b386957-e8b9-415f-8097-66cb8a84e9ac-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3b386957-e8b9-415f-8097-66cb8a84e9ac" (UID: "3b386957-e8b9-415f-8097-66cb8a84e9ac"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:17:27.473163 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:27.473115 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b386957-e8b9-415f-8097-66cb8a84e9ac-service-ca" (OuterVolumeSpecName: "service-ca") pod "3b386957-e8b9-415f-8097-66cb8a84e9ac" (UID: "3b386957-e8b9-415f-8097-66cb8a84e9ac"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:17:27.473230 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:27.473202 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b386957-e8b9-415f-8097-66cb8a84e9ac-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3b386957-e8b9-415f-8097-66cb8a84e9ac" (UID: "3b386957-e8b9-415f-8097-66cb8a84e9ac"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:17:27.473230 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:27.473205 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b386957-e8b9-415f-8097-66cb8a84e9ac-console-config" (OuterVolumeSpecName: "console-config") pod "3b386957-e8b9-415f-8097-66cb8a84e9ac" (UID: "3b386957-e8b9-415f-8097-66cb8a84e9ac"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:17:27.473350 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:27.473336 2550 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b386957-e8b9-415f-8097-66cb8a84e9ac-oauth-serving-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:17:27.473405 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:27.473353 2550 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b386957-e8b9-415f-8097-66cb8a84e9ac-trusted-ca-bundle\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:17:27.473405 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:27.473366 2550 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b386957-e8b9-415f-8097-66cb8a84e9ac-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:17:27.473405 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:27.473375 2550 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b386957-e8b9-415f-8097-66cb8a84e9ac-console-config\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:17:27.474971 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:27.474939 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b386957-e8b9-415f-8097-66cb8a84e9ac-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3b386957-e8b9-415f-8097-66cb8a84e9ac" (UID: "3b386957-e8b9-415f-8097-66cb8a84e9ac"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 22:17:27.475065 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:27.474973 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b386957-e8b9-415f-8097-66cb8a84e9ac-kube-api-access-rbjln" (OuterVolumeSpecName: "kube-api-access-rbjln") pod "3b386957-e8b9-415f-8097-66cb8a84e9ac" (UID: "3b386957-e8b9-415f-8097-66cb8a84e9ac"). InnerVolumeSpecName "kube-api-access-rbjln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:17:27.475065 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:27.474976 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b386957-e8b9-415f-8097-66cb8a84e9ac-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3b386957-e8b9-415f-8097-66cb8a84e9ac" (UID: "3b386957-e8b9-415f-8097-66cb8a84e9ac"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 22:17:27.574763 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:27.574730 2550 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b386957-e8b9-415f-8097-66cb8a84e9ac-console-oauth-config\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:17:27.574763 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:27.574755 2550 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b386957-e8b9-415f-8097-66cb8a84e9ac-console-serving-cert\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:17:27.574763 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:27.574765 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbjln\" (UniqueName: \"kubernetes.io/projected/3b386957-e8b9-415f-8097-66cb8a84e9ac-kube-api-access-rbjln\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:17:28.146138 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:28.146104 2550 generic.go:356] "Generic (PLEG): container finished" podID="3b386957-e8b9-415f-8097-66cb8a84e9ac" containerID="92082ebc7545e685b67a20796df255493fbea42b7a1fc41f3ac738ea44265f10" exitCode=0 Jul 14 22:17:28.146528 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:28.146145 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f664d7795-jhqz7" Jul 14 22:17:28.146528 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:28.146193 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f664d7795-jhqz7" event={"ID":"3b386957-e8b9-415f-8097-66cb8a84e9ac","Type":"ContainerDied","Data":"92082ebc7545e685b67a20796df255493fbea42b7a1fc41f3ac738ea44265f10"} Jul 14 22:17:28.146528 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:28.146239 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f664d7795-jhqz7" event={"ID":"3b386957-e8b9-415f-8097-66cb8a84e9ac","Type":"ContainerDied","Data":"99708f8ce194f5292bd98260b322cbc0ea1ab267bac37d12ad7035bc0a2e7e3a"} Jul 14 22:17:28.146528 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:28.146262 2550 scope.go:117] "RemoveContainer" containerID="92082ebc7545e685b67a20796df255493fbea42b7a1fc41f3ac738ea44265f10" Jul 14 22:17:28.154823 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:28.154806 2550 scope.go:117] "RemoveContainer" containerID="92082ebc7545e685b67a20796df255493fbea42b7a1fc41f3ac738ea44265f10" Jul 14 22:17:28.155057 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:17:28.155039 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92082ebc7545e685b67a20796df255493fbea42b7a1fc41f3ac738ea44265f10\": container with ID starting with 92082ebc7545e685b67a20796df255493fbea42b7a1fc41f3ac738ea44265f10 not found: ID does not exist" containerID="92082ebc7545e685b67a20796df255493fbea42b7a1fc41f3ac738ea44265f10" Jul 14 22:17:28.155098 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:28.155066 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92082ebc7545e685b67a20796df255493fbea42b7a1fc41f3ac738ea44265f10"} err="failed to get container status \"92082ebc7545e685b67a20796df255493fbea42b7a1fc41f3ac738ea44265f10\": rpc error: code = NotFound desc = could not find container \"92082ebc7545e685b67a20796df255493fbea42b7a1fc41f3ac738ea44265f10\": container with ID starting with 92082ebc7545e685b67a20796df255493fbea42b7a1fc41f3ac738ea44265f10 not found: ID does not exist" Jul 14 22:17:28.170862 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:28.170838 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f664d7795-jhqz7"] Jul 14 22:17:28.174578 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:28.174541 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f664d7795-jhqz7"] Jul 14 22:17:28.194937 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:28.194914 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b386957-e8b9-415f-8097-66cb8a84e9ac" path="/var/lib/kubelet/pods/3b386957-e8b9-415f-8097-66cb8a84e9ac/volumes" Jul 14 22:17:30.180302 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:30.180272 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:17:47.247220 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:47.247184 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:17:47.395197 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:47.395168 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:17:47.583753 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:17:47.583654 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:18:02.735593 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:02.735564 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m"] Jul 14 22:18:02.736052 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:02.735937 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b386957-e8b9-415f-8097-66cb8a84e9ac" containerName="console" Jul 14 22:18:02.736052 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:02.735956 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b386957-e8b9-415f-8097-66cb8a84e9ac" containerName="console" Jul 14 22:18:02.736052 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:02.736032 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b386957-e8b9-415f-8097-66cb8a84e9ac" containerName="console" Jul 14 22:18:02.739146 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:02.739131 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m" Jul 14 22:18:02.741863 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:02.741845 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 14 22:18:02.742929 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:02.742906 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mvnzc\"" Jul 14 22:18:02.743028 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:02.742956 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 14 22:18:02.755947 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:02.755921 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m"] Jul 14 22:18:02.776010 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:02.775983 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m\" (UID: \"2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m" Jul 14 22:18:02.776115 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:02.776018 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m\" (UID: \"2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m" Jul 14 22:18:02.776115 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:02.776086 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww692\" (UniqueName: \"kubernetes.io/projected/2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf-kube-api-access-ww692\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m\" (UID: \"2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m" Jul 14 22:18:02.876451 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:02.876420 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m\" (UID: \"2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m" Jul 14 22:18:02.876602 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:02.876456 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m\" (UID: \"2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m" Jul 14 22:18:02.876602 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:02.876492 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww692\" (UniqueName: \"kubernetes.io/projected/2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf-kube-api-access-ww692\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m\" (UID: \"2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m" Jul 14 22:18:02.876938 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:02.876819 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m\" (UID: \"2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m" Jul 14 22:18:02.876938 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:02.876869 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m\" (UID: \"2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m" Jul 14 22:18:02.885921 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:02.885902 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww692\" (UniqueName: \"kubernetes.io/projected/2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf-kube-api-access-ww692\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m\" (UID: \"2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m" Jul 14 22:18:03.047723 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:03.047642 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m" Jul 14 22:18:03.179496 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:03.179468 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m"] Jul 14 22:18:03.184504 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:18:03.184476 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c7d18a6_3c83_4a6d_9bb0_9b34eccdd1bf.slice/crio-bdc66f696190caf33d07bd22619dc6893c42ba33613b8162f9d1ab4caf24f3ae WatchSource:0}: Error finding container bdc66f696190caf33d07bd22619dc6893c42ba33613b8162f9d1ab4caf24f3ae: Status 404 returned error can't find the container with id bdc66f696190caf33d07bd22619dc6893c42ba33613b8162f9d1ab4caf24f3ae Jul 14 22:18:03.311335 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:03.311263 2550 generic.go:356] "Generic (PLEG): container finished" podID="2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf" containerID="a15ae0a87708b0e9685572a2a938b84e93d0c6c92cb68cc86aac9900b4aba2b6" exitCode=0 Jul 14 22:18:03.311459 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:03.311348 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m" event={"ID":"2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf","Type":"ContainerDied","Data":"a15ae0a87708b0e9685572a2a938b84e93d0c6c92cb68cc86aac9900b4aba2b6"} Jul 14 22:18:03.311459 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:03.311421 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m" event={"ID":"2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf","Type":"ContainerStarted","Data":"bdc66f696190caf33d07bd22619dc6893c42ba33613b8162f9d1ab4caf24f3ae"} Jul 14 22:18:03.318623 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:03.318601 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx"] Jul 14 22:18:03.321933 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:03.321918 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx" Jul 14 22:18:03.330826 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:03.330806 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx"] Jul 14 22:18:03.380446 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:03.380421 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34d9904f-1e3b-478a-bde9-9a1967ee85b5-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx\" (UID: \"34d9904f-1e3b-478a-bde9-9a1967ee85b5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx" Jul 14 22:18:03.380576 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:03.380472 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34d9904f-1e3b-478a-bde9-9a1967ee85b5-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx\" (UID: \"34d9904f-1e3b-478a-bde9-9a1967ee85b5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx" Jul 14 22:18:03.380630 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:03.380573 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m56hh\" (UniqueName: \"kubernetes.io/projected/34d9904f-1e3b-478a-bde9-9a1967ee85b5-kube-api-access-m56hh\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx\" (UID: \"34d9904f-1e3b-478a-bde9-9a1967ee85b5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx" Jul 14 22:18:03.481795 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:03.481752 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m56hh\" (UniqueName: \"kubernetes.io/projected/34d9904f-1e3b-478a-bde9-9a1967ee85b5-kube-api-access-m56hh\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx\" (UID: \"34d9904f-1e3b-478a-bde9-9a1967ee85b5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx" Jul 14 22:18:03.481795 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:03.481808 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34d9904f-1e3b-478a-bde9-9a1967ee85b5-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx\" (UID: \"34d9904f-1e3b-478a-bde9-9a1967ee85b5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx" Jul 14 22:18:03.482029 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:03.481835 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34d9904f-1e3b-478a-bde9-9a1967ee85b5-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx\" (UID: \"34d9904f-1e3b-478a-bde9-9a1967ee85b5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx" Jul 14 22:18:03.482167 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:03.482150 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34d9904f-1e3b-478a-bde9-9a1967ee85b5-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx\" (UID: \"34d9904f-1e3b-478a-bde9-9a1967ee85b5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx" Jul 14 22:18:03.482233 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:03.482202 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34d9904f-1e3b-478a-bde9-9a1967ee85b5-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx\" (UID: \"34d9904f-1e3b-478a-bde9-9a1967ee85b5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx" Jul 14 22:18:03.490893 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:03.490873 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m56hh\" (UniqueName: \"kubernetes.io/projected/34d9904f-1e3b-478a-bde9-9a1967ee85b5-kube-api-access-m56hh\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx\" (UID: \"34d9904f-1e3b-478a-bde9-9a1967ee85b5\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx" Jul 14 22:18:03.654589 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:03.654508 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx" Jul 14 22:18:03.713117 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:03.713090 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c"] Jul 14 22:18:03.716826 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:03.716805 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c" Jul 14 22:18:03.726383 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:03.726359 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c"] Jul 14 22:18:03.785024 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:03.784995 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa6b4ac5-8086-4b39-98cb-11db6e876fd0-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c\" (UID: \"fa6b4ac5-8086-4b39-98cb-11db6e876fd0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c" Jul 14 22:18:03.785024 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:03.785026 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa6b4ac5-8086-4b39-98cb-11db6e876fd0-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c\" (UID: \"fa6b4ac5-8086-4b39-98cb-11db6e876fd0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c" Jul 14 22:18:03.785431 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:03.785046 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g95v\" (UniqueName: \"kubernetes.io/projected/fa6b4ac5-8086-4b39-98cb-11db6e876fd0-kube-api-access-2g95v\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c\" (UID: \"fa6b4ac5-8086-4b39-98cb-11db6e876fd0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c" Jul 14 22:18:03.790734 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:03.790713 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx"] Jul 14 22:18:03.792500 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:18:03.792480 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d9904f_1e3b_478a_bde9_9a1967ee85b5.slice/crio-5d22df356c94cfcebafa5179b5b8ee4e251c590d2be169c110bc2d265d7a7469 WatchSource:0}: Error finding container 5d22df356c94cfcebafa5179b5b8ee4e251c590d2be169c110bc2d265d7a7469: Status 404 returned error can't find the container with id 5d22df356c94cfcebafa5179b5b8ee4e251c590d2be169c110bc2d265d7a7469 Jul 14 22:18:03.886533 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:03.886505 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa6b4ac5-8086-4b39-98cb-11db6e876fd0-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c\" (UID: \"fa6b4ac5-8086-4b39-98cb-11db6e876fd0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c" Jul 14 22:18:03.886746 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:03.886538 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa6b4ac5-8086-4b39-98cb-11db6e876fd0-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c\" (UID: \"fa6b4ac5-8086-4b39-98cb-11db6e876fd0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c" Jul 14 22:18:03.886746 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:03.886573 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g95v\" (UniqueName: \"kubernetes.io/projected/fa6b4ac5-8086-4b39-98cb-11db6e876fd0-kube-api-access-2g95v\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c\" (UID: \"fa6b4ac5-8086-4b39-98cb-11db6e876fd0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c" Jul 14 22:18:03.886968 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:03.886935 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa6b4ac5-8086-4b39-98cb-11db6e876fd0-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c\" (UID: \"fa6b4ac5-8086-4b39-98cb-11db6e876fd0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c" Jul 14 22:18:03.887085 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:03.886994 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa6b4ac5-8086-4b39-98cb-11db6e876fd0-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c\" (UID: \"fa6b4ac5-8086-4b39-98cb-11db6e876fd0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c" Jul 14 22:18:03.895853 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:03.895832 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g95v\" (UniqueName: \"kubernetes.io/projected/fa6b4ac5-8086-4b39-98cb-11db6e876fd0-kube-api-access-2g95v\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c\" (UID: \"fa6b4ac5-8086-4b39-98cb-11db6e876fd0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c" Jul 14 22:18:04.027149 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:04.027120 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c" Jul 14 22:18:04.107207 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:04.107181 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6"] Jul 14 22:18:04.110830 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:04.110807 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6" Jul 14 22:18:04.119546 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:04.119524 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6"] Jul 14 22:18:04.188696 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:04.188650 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17b5cd52-9924-4249-ad92-3d80b85ac197-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6\" (UID: \"17b5cd52-9924-4249-ad92-3d80b85ac197\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6" Jul 14 22:18:04.188880 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:04.188716 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s475p\" (UniqueName: \"kubernetes.io/projected/17b5cd52-9924-4249-ad92-3d80b85ac197-kube-api-access-s475p\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6\" (UID: \"17b5cd52-9924-4249-ad92-3d80b85ac197\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6" Jul 14 22:18:04.188880 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:04.188763 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17b5cd52-9924-4249-ad92-3d80b85ac197-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6\" (UID: \"17b5cd52-9924-4249-ad92-3d80b85ac197\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6" Jul 14 22:18:04.196842 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:18:04.196792 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa6b4ac5_8086_4b39_98cb_11db6e876fd0.slice/crio-652a8c937ffc232ad9e16cbc8aea229b591eb652a3681f91cc3b15deee890b03 WatchSource:0}: Error finding container 652a8c937ffc232ad9e16cbc8aea229b591eb652a3681f91cc3b15deee890b03: Status 404 returned error can't find the container with id 652a8c937ffc232ad9e16cbc8aea229b591eb652a3681f91cc3b15deee890b03 Jul 14 22:18:04.199049 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:04.199029 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c"] Jul 14 22:18:04.289510 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:04.289490 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17b5cd52-9924-4249-ad92-3d80b85ac197-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6\" (UID: \"17b5cd52-9924-4249-ad92-3d80b85ac197\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6" Jul 14 22:18:04.289626 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:04.289528 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s475p\" (UniqueName: \"kubernetes.io/projected/17b5cd52-9924-4249-ad92-3d80b85ac197-kube-api-access-s475p\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6\" (UID: \"17b5cd52-9924-4249-ad92-3d80b85ac197\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6" Jul 14 22:18:04.289626 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:04.289574 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17b5cd52-9924-4249-ad92-3d80b85ac197-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6\" (UID: \"17b5cd52-9924-4249-ad92-3d80b85ac197\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6" Jul 14 22:18:04.289835 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:04.289818 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17b5cd52-9924-4249-ad92-3d80b85ac197-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6\" (UID: \"17b5cd52-9924-4249-ad92-3d80b85ac197\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6" Jul 14 22:18:04.289894 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:04.289878 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17b5cd52-9924-4249-ad92-3d80b85ac197-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6\" (UID: \"17b5cd52-9924-4249-ad92-3d80b85ac197\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6" Jul 14 22:18:04.298340 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:04.298318 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s475p\" (UniqueName: \"kubernetes.io/projected/17b5cd52-9924-4249-ad92-3d80b85ac197-kube-api-access-s475p\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6\" (UID: \"17b5cd52-9924-4249-ad92-3d80b85ac197\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6" Jul 14 22:18:04.318252 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:04.318226 2550 generic.go:356] "Generic (PLEG): container finished" podID="2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf" containerID="300e59d337604617e1115574936e066f6583538345ad68f85fe357c2b2d900c1" exitCode=0 Jul 14 22:18:04.318361 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:04.318303 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m" event={"ID":"2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf","Type":"ContainerDied","Data":"300e59d337604617e1115574936e066f6583538345ad68f85fe357c2b2d900c1"} Jul 14 22:18:04.320076 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:04.319988 2550 generic.go:356] "Generic (PLEG): container finished" podID="34d9904f-1e3b-478a-bde9-9a1967ee85b5" containerID="c36cf89d41e549e858e8aaa943b8ccf466248d218770015810045e4122063248" exitCode=0 Jul 14 22:18:04.320147 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:04.320071 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx" event={"ID":"34d9904f-1e3b-478a-bde9-9a1967ee85b5","Type":"ContainerDied","Data":"c36cf89d41e549e858e8aaa943b8ccf466248d218770015810045e4122063248"} Jul 14 22:18:04.320147 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:04.320126 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx" event={"ID":"34d9904f-1e3b-478a-bde9-9a1967ee85b5","Type":"ContainerStarted","Data":"5d22df356c94cfcebafa5179b5b8ee4e251c590d2be169c110bc2d265d7a7469"} Jul 14 22:18:04.321875 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:04.321850 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c" event={"ID":"fa6b4ac5-8086-4b39-98cb-11db6e876fd0","Type":"ContainerStarted","Data":"5bf8bd574273fb75205e37c7c38a84d538ea36c901062ab30dfc5be0eb06e18f"} Jul 14 22:18:04.321988 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:04.321892 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c" event={"ID":"fa6b4ac5-8086-4b39-98cb-11db6e876fd0","Type":"ContainerStarted","Data":"652a8c937ffc232ad9e16cbc8aea229b591eb652a3681f91cc3b15deee890b03"} Jul 14 22:18:04.442540 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:04.442511 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6" Jul 14 22:18:04.575427 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:04.575401 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6"] Jul 14 22:18:04.576512 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:18:04.576484 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b5cd52_9924_4249_ad92_3d80b85ac197.slice/crio-7b9c003035abf7a2a26b6c384febf872624dd4e6528668bcc2f43a9ed077c394 WatchSource:0}: Error finding container 7b9c003035abf7a2a26b6c384febf872624dd4e6528668bcc2f43a9ed077c394: Status 404 returned error can't find the container with id 7b9c003035abf7a2a26b6c384febf872624dd4e6528668bcc2f43a9ed077c394 Jul 14 22:18:05.328380 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:05.328351 2550 generic.go:356] "Generic (PLEG): container finished" podID="2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf" containerID="39104fecff6f5ccd19e2aa9f6607b19ce175599c56f660cc2729c98d74169f9a" exitCode=0 Jul 14 22:18:05.328725 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:05.328441 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m" event={"ID":"2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf","Type":"ContainerDied","Data":"39104fecff6f5ccd19e2aa9f6607b19ce175599c56f660cc2729c98d74169f9a"} Jul 14 22:18:05.330352 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:05.330328 2550 generic.go:356] "Generic (PLEG): container finished" podID="34d9904f-1e3b-478a-bde9-9a1967ee85b5" containerID="44db64607a9e4e841a80e9056a6acfa2404a638086849d0c461cdefb112035e3" exitCode=0 Jul 14 22:18:05.330458 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:05.330414 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx" event={"ID":"34d9904f-1e3b-478a-bde9-9a1967ee85b5","Type":"ContainerDied","Data":"44db64607a9e4e841a80e9056a6acfa2404a638086849d0c461cdefb112035e3"} Jul 14 22:18:05.331942 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:05.331923 2550 generic.go:356] "Generic (PLEG): container finished" podID="17b5cd52-9924-4249-ad92-3d80b85ac197" containerID="3744397ee6a9ca432bea63542dbe97c5a24f31e6b28a726b8f622532dbc062c5" exitCode=0 Jul 14 22:18:05.332021 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:05.332009 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6" event={"ID":"17b5cd52-9924-4249-ad92-3d80b85ac197","Type":"ContainerDied","Data":"3744397ee6a9ca432bea63542dbe97c5a24f31e6b28a726b8f622532dbc062c5"} Jul 14 22:18:05.332076 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:05.332046 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6" event={"ID":"17b5cd52-9924-4249-ad92-3d80b85ac197","Type":"ContainerStarted","Data":"7b9c003035abf7a2a26b6c384febf872624dd4e6528668bcc2f43a9ed077c394"} Jul 14 22:18:05.333659 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:05.333636 2550 generic.go:356] "Generic (PLEG): container finished" podID="fa6b4ac5-8086-4b39-98cb-11db6e876fd0" containerID="5bf8bd574273fb75205e37c7c38a84d538ea36c901062ab30dfc5be0eb06e18f" exitCode=0 Jul 14 22:18:05.334293 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:05.333668 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c" event={"ID":"fa6b4ac5-8086-4b39-98cb-11db6e876fd0","Type":"ContainerDied","Data":"5bf8bd574273fb75205e37c7c38a84d538ea36c901062ab30dfc5be0eb06e18f"} Jul 14 22:18:06.342107 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:06.342070 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6" event={"ID":"17b5cd52-9924-4249-ad92-3d80b85ac197","Type":"ContainerStarted","Data":"6ab19983cddc8d4ba53eeba757f978b93e16b6cf26765067ff46daea7376d56e"} Jul 14 22:18:06.344370 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:06.344315 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c" event={"ID":"fa6b4ac5-8086-4b39-98cb-11db6e876fd0","Type":"ContainerStarted","Data":"e9729ead83c4b03c557dd643fcd40d2950356d5496cbfc8ef7cc3bbd35f4e9bf"} Jul 14 22:18:06.347188 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:06.347159 2550 generic.go:356] "Generic (PLEG): container finished" podID="34d9904f-1e3b-478a-bde9-9a1967ee85b5" containerID="6d6540f1d18816a83b4032c35abf3965f1066cddac3981d1083f4bc4d00f5f6f" exitCode=0 Jul 14 22:18:06.347278 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:06.347236 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx" event={"ID":"34d9904f-1e3b-478a-bde9-9a1967ee85b5","Type":"ContainerDied","Data":"6d6540f1d18816a83b4032c35abf3965f1066cddac3981d1083f4bc4d00f5f6f"} Jul 14 22:18:06.472546 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:06.472525 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m" Jul 14 22:18:06.508836 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:06.508765 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf-util\") pod \"2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf\" (UID: \"2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf\") " Jul 14 22:18:06.508836 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:06.508809 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf-bundle\") pod \"2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf\" (UID: \"2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf\") " Jul 14 22:18:06.509008 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:06.508872 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ww692\" (UniqueName: \"kubernetes.io/projected/2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf-kube-api-access-ww692\") pod \"2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf\" (UID: \"2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf\") " Jul 14 22:18:06.512368 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:06.509921 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf-bundle" (OuterVolumeSpecName: "bundle") pod "2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf" (UID: "2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:18:06.512532 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:06.512507 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf-kube-api-access-ww692" (OuterVolumeSpecName: "kube-api-access-ww692") pod "2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf" (UID: "2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf"). InnerVolumeSpecName "kube-api-access-ww692". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:18:06.515784 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:06.515755 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf-util" (OuterVolumeSpecName: "util") pod "2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf" (UID: "2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:18:06.610385 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:06.610340 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww692\" (UniqueName: \"kubernetes.io/projected/2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf-kube-api-access-ww692\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:18:06.610385 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:06.610377 2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf-util\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:18:06.610385 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:06.610387 2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf-bundle\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:18:07.354840 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:07.354812 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m" Jul 14 22:18:07.354840 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:07.354828 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m" event={"ID":"2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf","Type":"ContainerDied","Data":"bdc66f696190caf33d07bd22619dc6893c42ba33613b8162f9d1ab4caf24f3ae"} Jul 14 22:18:07.355264 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:07.354871 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdc66f696190caf33d07bd22619dc6893c42ba33613b8162f9d1ab4caf24f3ae" Jul 14 22:18:07.356541 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:07.356519 2550 generic.go:356] "Generic (PLEG): container finished" podID="17b5cd52-9924-4249-ad92-3d80b85ac197" containerID="6ab19983cddc8d4ba53eeba757f978b93e16b6cf26765067ff46daea7376d56e" exitCode=0 Jul 14 22:18:07.356661 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:07.356585 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6" event={"ID":"17b5cd52-9924-4249-ad92-3d80b85ac197","Type":"ContainerDied","Data":"6ab19983cddc8d4ba53eeba757f978b93e16b6cf26765067ff46daea7376d56e"} Jul 14 22:18:07.358542 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:07.358518 2550 generic.go:356] "Generic (PLEG): container finished" podID="fa6b4ac5-8086-4b39-98cb-11db6e876fd0" containerID="e9729ead83c4b03c557dd643fcd40d2950356d5496cbfc8ef7cc3bbd35f4e9bf" exitCode=0 Jul 14 22:18:07.358641 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:07.358585 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c" event={"ID":"fa6b4ac5-8086-4b39-98cb-11db6e876fd0","Type":"ContainerDied","Data":"e9729ead83c4b03c557dd643fcd40d2950356d5496cbfc8ef7cc3bbd35f4e9bf"} Jul 14 22:18:07.496994 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:07.496970 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx" Jul 14 22:18:07.515601 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:07.515577 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34d9904f-1e3b-478a-bde9-9a1967ee85b5-util\") pod \"34d9904f-1e3b-478a-bde9-9a1967ee85b5\" (UID: \"34d9904f-1e3b-478a-bde9-9a1967ee85b5\") " Jul 14 22:18:07.515709 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:07.515666 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34d9904f-1e3b-478a-bde9-9a1967ee85b5-bundle\") pod \"34d9904f-1e3b-478a-bde9-9a1967ee85b5\" (UID: \"34d9904f-1e3b-478a-bde9-9a1967ee85b5\") " Jul 14 22:18:07.515709 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:07.515689 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m56hh\" (UniqueName: \"kubernetes.io/projected/34d9904f-1e3b-478a-bde9-9a1967ee85b5-kube-api-access-m56hh\") pod \"34d9904f-1e3b-478a-bde9-9a1967ee85b5\" (UID: \"34d9904f-1e3b-478a-bde9-9a1967ee85b5\") " Jul 14 22:18:07.516150 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:07.516123 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d9904f-1e3b-478a-bde9-9a1967ee85b5-bundle" (OuterVolumeSpecName: "bundle") pod "34d9904f-1e3b-478a-bde9-9a1967ee85b5" (UID: "34d9904f-1e3b-478a-bde9-9a1967ee85b5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:18:07.517832 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:07.517754 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34d9904f-1e3b-478a-bde9-9a1967ee85b5-kube-api-access-m56hh" (OuterVolumeSpecName: "kube-api-access-m56hh") pod "34d9904f-1e3b-478a-bde9-9a1967ee85b5" (UID: "34d9904f-1e3b-478a-bde9-9a1967ee85b5"). InnerVolumeSpecName "kube-api-access-m56hh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:18:07.521692 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:07.521668 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d9904f-1e3b-478a-bde9-9a1967ee85b5-util" (OuterVolumeSpecName: "util") pod "34d9904f-1e3b-478a-bde9-9a1967ee85b5" (UID: "34d9904f-1e3b-478a-bde9-9a1967ee85b5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:18:07.616506 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:07.616477 2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34d9904f-1e3b-478a-bde9-9a1967ee85b5-util\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:18:07.616506 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:07.616503 2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34d9904f-1e3b-478a-bde9-9a1967ee85b5-bundle\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:18:07.616670 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:07.616513 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m56hh\" (UniqueName: \"kubernetes.io/projected/34d9904f-1e3b-478a-bde9-9a1967ee85b5-kube-api-access-m56hh\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:18:08.365961 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:08.365926 2550 generic.go:356] "Generic (PLEG): container finished" podID="17b5cd52-9924-4249-ad92-3d80b85ac197" containerID="683822f24961c6ef88d6396ec492def6dd1f334c0ac4c38957a6cb5f95bd4fd8" exitCode=0 Jul 14 22:18:08.366349 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:08.366008 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6" event={"ID":"17b5cd52-9924-4249-ad92-3d80b85ac197","Type":"ContainerDied","Data":"683822f24961c6ef88d6396ec492def6dd1f334c0ac4c38957a6cb5f95bd4fd8"} Jul 14 22:18:08.368117 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:08.368096 2550 generic.go:356] "Generic (PLEG): container finished" podID="fa6b4ac5-8086-4b39-98cb-11db6e876fd0" containerID="f4db10ace4054d0fbe75c00b92783993e7c68facca57f4bb95afa500eb2452c9" exitCode=0 Jul 14 22:18:08.368240 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:08.368183 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c" event={"ID":"fa6b4ac5-8086-4b39-98cb-11db6e876fd0","Type":"ContainerDied","Data":"f4db10ace4054d0fbe75c00b92783993e7c68facca57f4bb95afa500eb2452c9"} Jul 14 22:18:08.369833 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:08.369810 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx" event={"ID":"34d9904f-1e3b-478a-bde9-9a1967ee85b5","Type":"ContainerDied","Data":"5d22df356c94cfcebafa5179b5b8ee4e251c590d2be169c110bc2d265d7a7469"} Jul 14 22:18:08.369926 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:08.369851 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx" Jul 14 22:18:08.369926 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:08.369855 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d22df356c94cfcebafa5179b5b8ee4e251c590d2be169c110bc2d265d7a7469" Jul 14 22:18:09.526966 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:09.526945 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6" Jul 14 22:18:09.529406 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:09.529387 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c" Jul 14 22:18:09.641023 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:09.640937 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa6b4ac5-8086-4b39-98cb-11db6e876fd0-bundle\") pod \"fa6b4ac5-8086-4b39-98cb-11db6e876fd0\" (UID: \"fa6b4ac5-8086-4b39-98cb-11db6e876fd0\") " Jul 14 22:18:09.641023 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:09.640975 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa6b4ac5-8086-4b39-98cb-11db6e876fd0-util\") pod \"fa6b4ac5-8086-4b39-98cb-11db6e876fd0\" (UID: \"fa6b4ac5-8086-4b39-98cb-11db6e876fd0\") " Jul 14 22:18:09.641023 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:09.641002 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s475p\" (UniqueName: \"kubernetes.io/projected/17b5cd52-9924-4249-ad92-3d80b85ac197-kube-api-access-s475p\") pod \"17b5cd52-9924-4249-ad92-3d80b85ac197\" (UID: \"17b5cd52-9924-4249-ad92-3d80b85ac197\") " Jul 14 22:18:09.641286 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:09.641045 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17b5cd52-9924-4249-ad92-3d80b85ac197-util\") pod \"17b5cd52-9924-4249-ad92-3d80b85ac197\" (UID: \"17b5cd52-9924-4249-ad92-3d80b85ac197\") " Jul 14 22:18:09.641286 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:09.641122 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2g95v\" (UniqueName: \"kubernetes.io/projected/fa6b4ac5-8086-4b39-98cb-11db6e876fd0-kube-api-access-2g95v\") pod \"fa6b4ac5-8086-4b39-98cb-11db6e876fd0\" (UID: \"fa6b4ac5-8086-4b39-98cb-11db6e876fd0\") " Jul 14 22:18:09.641286 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:09.641144 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17b5cd52-9924-4249-ad92-3d80b85ac197-bundle\") pod \"17b5cd52-9924-4249-ad92-3d80b85ac197\" (UID: \"17b5cd52-9924-4249-ad92-3d80b85ac197\") " Jul 14 22:18:09.641759 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:09.641707 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b5cd52-9924-4249-ad92-3d80b85ac197-bundle" (OuterVolumeSpecName: "bundle") pod "17b5cd52-9924-4249-ad92-3d80b85ac197" (UID: "17b5cd52-9924-4249-ad92-3d80b85ac197"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:18:09.641759 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:09.641714 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa6b4ac5-8086-4b39-98cb-11db6e876fd0-bundle" (OuterVolumeSpecName: "bundle") pod "fa6b4ac5-8086-4b39-98cb-11db6e876fd0" (UID: "fa6b4ac5-8086-4b39-98cb-11db6e876fd0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:18:09.643100 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:09.643079 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17b5cd52-9924-4249-ad92-3d80b85ac197-kube-api-access-s475p" (OuterVolumeSpecName: "kube-api-access-s475p") pod "17b5cd52-9924-4249-ad92-3d80b85ac197" (UID: "17b5cd52-9924-4249-ad92-3d80b85ac197"). InnerVolumeSpecName "kube-api-access-s475p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:18:09.643219 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:09.643203 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa6b4ac5-8086-4b39-98cb-11db6e876fd0-kube-api-access-2g95v" (OuterVolumeSpecName: "kube-api-access-2g95v") pod "fa6b4ac5-8086-4b39-98cb-11db6e876fd0" (UID: "fa6b4ac5-8086-4b39-98cb-11db6e876fd0"). InnerVolumeSpecName "kube-api-access-2g95v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:18:09.646734 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:09.646701 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b5cd52-9924-4249-ad92-3d80b85ac197-util" (OuterVolumeSpecName: "util") pod "17b5cd52-9924-4249-ad92-3d80b85ac197" (UID: "17b5cd52-9924-4249-ad92-3d80b85ac197"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:18:09.742618 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:09.742585 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g95v\" (UniqueName: \"kubernetes.io/projected/fa6b4ac5-8086-4b39-98cb-11db6e876fd0-kube-api-access-2g95v\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:18:09.742618 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:09.742612 2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17b5cd52-9924-4249-ad92-3d80b85ac197-bundle\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:18:09.742618 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:09.742623 2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa6b4ac5-8086-4b39-98cb-11db6e876fd0-bundle\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:18:09.742810 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:09.742631 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s475p\" (UniqueName: \"kubernetes.io/projected/17b5cd52-9924-4249-ad92-3d80b85ac197-kube-api-access-s475p\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:18:09.742810 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:09.742639 2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17b5cd52-9924-4249-ad92-3d80b85ac197-util\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:18:09.836000 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:09.835959 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa6b4ac5-8086-4b39-98cb-11db6e876fd0-util" (OuterVolumeSpecName: "util") pod "fa6b4ac5-8086-4b39-98cb-11db6e876fd0" (UID: "fa6b4ac5-8086-4b39-98cb-11db6e876fd0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:18:09.843003 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:09.842983 2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa6b4ac5-8086-4b39-98cb-11db6e876fd0-util\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:18:10.381292 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:10.381263 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6" Jul 14 22:18:10.381449 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:10.381263 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6" event={"ID":"17b5cd52-9924-4249-ad92-3d80b85ac197","Type":"ContainerDied","Data":"7b9c003035abf7a2a26b6c384febf872624dd4e6528668bcc2f43a9ed077c394"} Jul 14 22:18:10.381449 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:10.381381 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b9c003035abf7a2a26b6c384febf872624dd4e6528668bcc2f43a9ed077c394" Jul 14 22:18:10.383238 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:10.383187 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c" event={"ID":"fa6b4ac5-8086-4b39-98cb-11db6e876fd0","Type":"ContainerDied","Data":"652a8c937ffc232ad9e16cbc8aea229b591eb652a3681f91cc3b15deee890b03"} Jul 14 22:18:10.383238 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:10.383231 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="652a8c937ffc232ad9e16cbc8aea229b591eb652a3681f91cc3b15deee890b03" Jul 14 22:18:10.383238 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:10.383203 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c" Jul 14 22:18:16.637180 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:16.637150 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:18:21.020641 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:21.020525 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:18:23.642326 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:23.642276 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:18:24.223056 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:24.223026 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:18:24.656986 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:24.656901 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:18:26.288222 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:26.288153 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:18:27.483608 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:27.483571 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:18:28.689341 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:28.689311 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:18:32.225668 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:32.225634 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:18:33.680098 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:33.680063 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:18:33.812115 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:33.812085 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:18:35.468736 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:35.468696 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:18:35.593269 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:35.593237 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:18:41.495754 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:41.495719 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:18:41.595694 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:41.595663 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:18:43.442994 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:43.442965 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:18:43.550187 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:43.550160 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:18:44.998515 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:44.998482 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:18:45.097583 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:45.097531 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:18:51.676874 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:51.676837 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:18:51.790503 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:51.790475 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:18:52.435575 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:52.435522 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:18:54.257915 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.257882 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:18:54.345535 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.345504 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g86p5"] Jul 14 22:18:54.345948 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.345934 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa6b4ac5-8086-4b39-98cb-11db6e876fd0" containerName="util" Jul 14 22:18:54.345990 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.345952 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa6b4ac5-8086-4b39-98cb-11db6e876fd0" containerName="util" Jul 14 22:18:54.345990 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.345967 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa6b4ac5-8086-4b39-98cb-11db6e876fd0" containerName="pull" Jul 14 22:18:54.345990 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.345976 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa6b4ac5-8086-4b39-98cb-11db6e876fd0" containerName="pull" Jul 14 22:18:54.346085 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.345989 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34d9904f-1e3b-478a-bde9-9a1967ee85b5" containerName="extract" Jul 14 22:18:54.346085 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.345998 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="34d9904f-1e3b-478a-bde9-9a1967ee85b5" containerName="extract" Jul 14 22:18:54.346085 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.346010 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf" containerName="util" Jul 14 22:18:54.346085 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.346018 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf" containerName="util" Jul 14 22:18:54.346085 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.346028 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34d9904f-1e3b-478a-bde9-9a1967ee85b5" containerName="pull" Jul 14 22:18:54.346085 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.346035 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="34d9904f-1e3b-478a-bde9-9a1967ee85b5" containerName="pull" Jul 14 22:18:54.346085 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.346043 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17b5cd52-9924-4249-ad92-3d80b85ac197" containerName="extract" Jul 14 22:18:54.346085 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.346051 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="17b5cd52-9924-4249-ad92-3d80b85ac197" containerName="extract" Jul 14 22:18:54.346085 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.346059 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17b5cd52-9924-4249-ad92-3d80b85ac197" containerName="pull" Jul 14 22:18:54.346085 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.346067 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="17b5cd52-9924-4249-ad92-3d80b85ac197" containerName="pull" Jul 14 22:18:54.346085 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.346085 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf" containerName="pull" Jul 14 22:18:54.346399 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.346094 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf" containerName="pull" Jul 14 22:18:54.346399 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.346104 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa6b4ac5-8086-4b39-98cb-11db6e876fd0" containerName="extract" Jul 14 22:18:54.346399 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.346111 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa6b4ac5-8086-4b39-98cb-11db6e876fd0" containerName="extract" Jul 14 22:18:54.346399 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.346125 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34d9904f-1e3b-478a-bde9-9a1967ee85b5" containerName="util" Jul 14 22:18:54.346399 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.346133 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="34d9904f-1e3b-478a-bde9-9a1967ee85b5" containerName="util" Jul 14 22:18:54.346399 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.346147 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17b5cd52-9924-4249-ad92-3d80b85ac197" containerName="util" Jul 14 22:18:54.346399 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.346155 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="17b5cd52-9924-4249-ad92-3d80b85ac197" containerName="util" Jul 14 22:18:54.346399 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.346166 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf" containerName="extract" Jul 14 22:18:54.346399 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.346175 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf" containerName="extract" Jul 14 22:18:54.346399 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.346244 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="17b5cd52-9924-4249-ad92-3d80b85ac197" containerName="extract" Jul 14 22:18:54.346399 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.346255 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="34d9904f-1e3b-478a-bde9-9a1967ee85b5" containerName="extract" Jul 14 22:18:54.346399 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.346264 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf" containerName="extract" Jul 14 22:18:54.346399 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.346273 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa6b4ac5-8086-4b39-98cb-11db6e876fd0" containerName="extract" Jul 14 22:18:54.348523 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.348505 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g86p5" Jul 14 22:18:54.351613 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.351591 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 14 22:18:54.351733 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.351601 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-kshmc\"" Jul 14 22:18:54.352534 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.352511 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 14 22:18:54.372296 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.372267 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g86p5"] Jul 14 22:18:54.373181 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.373155 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:18:54.411523 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.411497 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl9f9\" (UniqueName: \"kubernetes.io/projected/d4311840-f902-47c1-bdb2-a106214e546e-kube-api-access-vl9f9\") pod \"limitador-operator-controller-manager-859d7b78bd-g86p5\" (UID: \"d4311840-f902-47c1-bdb2-a106214e546e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g86p5" Jul 14 22:18:54.512317 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.512224 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl9f9\" (UniqueName: \"kubernetes.io/projected/d4311840-f902-47c1-bdb2-a106214e546e-kube-api-access-vl9f9\") pod \"limitador-operator-controller-manager-859d7b78bd-g86p5\" (UID: \"d4311840-f902-47c1-bdb2-a106214e546e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g86p5" Jul 14 22:18:54.524202 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.524173 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl9f9\" (UniqueName: \"kubernetes.io/projected/d4311840-f902-47c1-bdb2-a106214e546e-kube-api-access-vl9f9\") pod \"limitador-operator-controller-manager-859d7b78bd-g86p5\" (UID: \"d4311840-f902-47c1-bdb2-a106214e546e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g86p5" Jul 14 22:18:54.659689 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.659649 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g86p5" Jul 14 22:18:54.791677 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:54.791651 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g86p5"] Jul 14 22:18:54.793693 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:18:54.793663 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4311840_f902_47c1_bdb2_a106214e546e.slice/crio-24686ff551f4f6b85719649e89e7d6bba477c0b88a9fd693a32e2b04dd3e2034 WatchSource:0}: Error finding container 24686ff551f4f6b85719649e89e7d6bba477c0b88a9fd693a32e2b04dd3e2034: Status 404 returned error can't find the container with id 24686ff551f4f6b85719649e89e7d6bba477c0b88a9fd693a32e2b04dd3e2034 Jul 14 22:18:55.610014 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:55.609976 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g86p5" event={"ID":"d4311840-f902-47c1-bdb2-a106214e546e","Type":"ContainerStarted","Data":"24686ff551f4f6b85719649e89e7d6bba477c0b88a9fd693a32e2b04dd3e2034"} Jul 14 22:18:56.616707 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:56.616671 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g86p5" event={"ID":"d4311840-f902-47c1-bdb2-a106214e546e","Type":"ContainerStarted","Data":"82b772c1d7ecb732da67041e9f62653a9b71d7a23dfd07c40df48f3cab4e2750"} Jul 14 22:18:56.617058 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:56.616751 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g86p5" Jul 14 22:18:56.636869 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:56.636823 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g86p5" podStartSLOduration=1.000816849 podStartE2EDuration="2.636807584s" totalImagesPullingTime="1.635990735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:18:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:18:56.635986616 +0000 UTC m=+566.995133053" watchObservedRunningTime="2026-07-14 22:18:56.636807584 +0000 UTC m=+566.995954020" Jul 14 22:18:59.240809 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:59.240771 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:18:59.324275 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:59.324244 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:18:59.470965 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:59.470933 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:18:59.575746 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:18:59.575654 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:19:03.288151 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:19:03.288116 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:19:03.437812 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:19:03.437783 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:19:07.623438 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:19:07.623405 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g86p5" Jul 14 22:19:11.017694 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:19:11.017665 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:19:14.705818 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:19:14.705792 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:19:14.777197 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:19:14.777156 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:19:30.020864 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:19:30.020837 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zj4sb_0884baa5-9206-4196-a7c9-b5686623311b/ovn-acl-logging/0.log" Jul 14 22:19:30.021953 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:19:30.021935 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zj4sb_0884baa5-9206-4196-a7c9-b5686623311b/ovn-acl-logging/0.log" Jul 14 22:19:40.797520 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:19:40.797485 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:19:40.974984 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:19:40.974954 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:19:44.214886 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:19:44.214848 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:19:44.320282 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:19:44.320252 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:19:44.388610 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:19:44.388583 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:19:45.105495 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:19:45.105464 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:19:45.189908 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:19:45.189878 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:20:04.146742 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:20:04.146671 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:20:04.269381 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:20:04.269351 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:20:04.669197 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:20:04.669166 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:20:12.361218 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:20:12.361182 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:20:16.824563 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:20:16.824521 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:20:17.539312 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:20:17.539280 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:20:23.550531 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:20:23.550492 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:20:25.186901 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:20:25.186868 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:20:25.393501 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:20:25.393473 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:20:25.553962 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:20:25.553890 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:20:28.223938 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:20:28.223907 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:20:28.421967 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:20:28.421932 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:20:59.160434 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:20:59.160402 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:20:59.287878 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:20:59.287846 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:20:59.704564 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:20:59.704531 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:20:59.847077 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:20:59.847042 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-d559f64cb-bsjmz"] Jul 14 22:20:59.850581 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:20:59.850543 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d559f64cb-bsjmz" Jul 14 22:20:59.853886 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:20:59.853865 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 14 22:20:59.854000 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:20:59.853961 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7v6nj\"" Jul 14 22:20:59.854885 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:20:59.854863 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 14 22:20:59.855005 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:20:59.854863 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 22:20:59.867305 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:20:59.867281 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d559f64cb-bsjmz"] Jul 14 22:20:59.940267 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:20:59.940231 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt7c6\" (UniqueName: \"kubernetes.io/projected/4b3ca406-334e-47c4-86d9-bd27fffc3a08-kube-api-access-gt7c6\") pod \"maas-api-d559f64cb-bsjmz\" (UID: \"4b3ca406-334e-47c4-86d9-bd27fffc3a08\") " pod="odh-ai-gateway-infra/maas-api-d559f64cb-bsjmz" Jul 14 22:20:59.940448 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:20:59.940302 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b3ca406-334e-47c4-86d9-bd27fffc3a08-maas-api-tls\") pod \"maas-api-d559f64cb-bsjmz\" (UID: \"4b3ca406-334e-47c4-86d9-bd27fffc3a08\") " pod="odh-ai-gateway-infra/maas-api-d559f64cb-bsjmz" Jul 14 22:20:59.940448 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:20:59.940379 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b3ca406-334e-47c4-86d9-bd27fffc3a08-openshift-service-ca\") pod \"maas-api-d559f64cb-bsjmz\" (UID: \"4b3ca406-334e-47c4-86d9-bd27fffc3a08\") " pod="odh-ai-gateway-infra/maas-api-d559f64cb-bsjmz" Jul 14 22:20:59.990762 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:20:59.990689 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:21:00.040990 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:00.040959 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b3ca406-334e-47c4-86d9-bd27fffc3a08-maas-api-tls\") pod \"maas-api-d559f64cb-bsjmz\" (UID: \"4b3ca406-334e-47c4-86d9-bd27fffc3a08\") " pod="odh-ai-gateway-infra/maas-api-d559f64cb-bsjmz" Jul 14 22:21:00.041137 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:00.041004 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b3ca406-334e-47c4-86d9-bd27fffc3a08-openshift-service-ca\") pod \"maas-api-d559f64cb-bsjmz\" (UID: \"4b3ca406-334e-47c4-86d9-bd27fffc3a08\") " pod="odh-ai-gateway-infra/maas-api-d559f64cb-bsjmz" Jul 14 22:21:00.041137 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:00.041030 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt7c6\" (UniqueName: \"kubernetes.io/projected/4b3ca406-334e-47c4-86d9-bd27fffc3a08-kube-api-access-gt7c6\") pod \"maas-api-d559f64cb-bsjmz\" (UID: \"4b3ca406-334e-47c4-86d9-bd27fffc3a08\") " pod="odh-ai-gateway-infra/maas-api-d559f64cb-bsjmz" Jul 14 22:21:00.041672 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:00.041651 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b3ca406-334e-47c4-86d9-bd27fffc3a08-openshift-service-ca\") pod \"maas-api-d559f64cb-bsjmz\" (UID: \"4b3ca406-334e-47c4-86d9-bd27fffc3a08\") " pod="odh-ai-gateway-infra/maas-api-d559f64cb-bsjmz" Jul 14 22:21:00.043352 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:00.043322 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b3ca406-334e-47c4-86d9-bd27fffc3a08-maas-api-tls\") pod \"maas-api-d559f64cb-bsjmz\" (UID: \"4b3ca406-334e-47c4-86d9-bd27fffc3a08\") " pod="odh-ai-gateway-infra/maas-api-d559f64cb-bsjmz" Jul 14 22:21:00.050342 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:00.050319 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt7c6\" (UniqueName: \"kubernetes.io/projected/4b3ca406-334e-47c4-86d9-bd27fffc3a08-kube-api-access-gt7c6\") pod \"maas-api-d559f64cb-bsjmz\" (UID: \"4b3ca406-334e-47c4-86d9-bd27fffc3a08\") " pod="odh-ai-gateway-infra/maas-api-d559f64cb-bsjmz" Jul 14 22:21:00.160604 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:00.160547 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d559f64cb-bsjmz" Jul 14 22:21:00.306760 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:00.306735 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d559f64cb-bsjmz"] Jul 14 22:21:00.308782 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:21:00.308758 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b3ca406_334e_47c4_86d9_bd27fffc3a08.slice/crio-edfaf6fa48618efbb95325d89ebc279ed8beb9d798b44c13144a52772ad49fbe WatchSource:0}: Error finding container edfaf6fa48618efbb95325d89ebc279ed8beb9d798b44c13144a52772ad49fbe: Status 404 returned error can't find the container with id edfaf6fa48618efbb95325d89ebc279ed8beb9d798b44c13144a52772ad49fbe Jul 14 22:21:00.310057 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:00.310041 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 22:21:00.510183 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:00.510101 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:21:00.603143 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:00.603111 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:21:01.206450 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:01.206367 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d559f64cb-bsjmz" event={"ID":"4b3ca406-334e-47c4-86d9-bd27fffc3a08","Type":"ContainerStarted","Data":"edfaf6fa48618efbb95325d89ebc279ed8beb9d798b44c13144a52772ad49fbe"} Jul 14 22:21:02.211895 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:02.211864 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d559f64cb-bsjmz" event={"ID":"4b3ca406-334e-47c4-86d9-bd27fffc3a08","Type":"ContainerStarted","Data":"6267b1f0646b9c46883227acfa54153d2505426eb0646570954c1f263f19bb63"} Jul 14 22:21:02.212227 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:02.212035 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-d559f64cb-bsjmz" Jul 14 22:21:02.235292 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:02.235248 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-d559f64cb-bsjmz" podStartSLOduration=1.416989447 podStartE2EDuration="3.235232575s" totalImagesPullingTime="1.818243128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:21:02.232171867 +0000 UTC m=+692.591318304" watchObservedRunningTime="2026-07-14 22:21:02.235232575 +0000 UTC m=+692.594379060" Jul 14 22:21:08.221491 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:08.221452 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-d559f64cb-bsjmz" Jul 14 22:21:09.290048 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:09.289990 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:21:09.389136 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:09.389107 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:21:23.548367 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:23.548278 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:21:37.999601 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:37.999573 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-n78xj"] Jul 14 22:21:38.002089 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:38.002070 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n78xj" Jul 14 22:21:38.005860 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:38.005841 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 14 22:21:38.006727 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:38.006711 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lhvtx\"" Jul 14 22:21:38.006800 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:38.006744 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 14 22:21:38.006847 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:38.006803 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 14 22:21:38.025526 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:38.025501 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-n78xj"] Jul 14 22:21:38.056951 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:38.056924 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:21:38.068334 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:38.068310 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f451ea84-ed76-4751-87d0-b341564aec23-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n78xj\" (UID: \"f451ea84-ed76-4751-87d0-b341564aec23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n78xj" Jul 14 22:21:38.068423 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:38.068371 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f451ea84-ed76-4751-87d0-b341564aec23-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n78xj\" (UID: \"f451ea84-ed76-4751-87d0-b341564aec23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n78xj" Jul 14 22:21:38.068423 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:38.068410 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swppx\" (UniqueName: \"kubernetes.io/projected/f451ea84-ed76-4751-87d0-b341564aec23-kube-api-access-swppx\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n78xj\" (UID: \"f451ea84-ed76-4751-87d0-b341564aec23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n78xj" Jul 14 22:21:38.068516 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:38.068430 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f451ea84-ed76-4751-87d0-b341564aec23-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n78xj\" (UID: \"f451ea84-ed76-4751-87d0-b341564aec23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n78xj" Jul 14 22:21:38.068516 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:38.068460 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f451ea84-ed76-4751-87d0-b341564aec23-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n78xj\" (UID: \"f451ea84-ed76-4751-87d0-b341564aec23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n78xj" Jul 14 22:21:38.068619 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:38.068531 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f451ea84-ed76-4751-87d0-b341564aec23-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n78xj\" (UID: \"f451ea84-ed76-4751-87d0-b341564aec23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n78xj" Jul 14 22:21:38.169716 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:38.169678 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f451ea84-ed76-4751-87d0-b341564aec23-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n78xj\" (UID: \"f451ea84-ed76-4751-87d0-b341564aec23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n78xj" Jul 14 22:21:38.169905 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:38.169731 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f451ea84-ed76-4751-87d0-b341564aec23-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n78xj\" (UID: \"f451ea84-ed76-4751-87d0-b341564aec23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n78xj" Jul 14 22:21:38.169905 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:38.169846 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f451ea84-ed76-4751-87d0-b341564aec23-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n78xj\" (UID: \"f451ea84-ed76-4751-87d0-b341564aec23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n78xj" Jul 14 22:21:38.170013 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:38.169937 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f451ea84-ed76-4751-87d0-b341564aec23-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n78xj\" (UID: \"f451ea84-ed76-4751-87d0-b341564aec23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n78xj" Jul 14 22:21:38.170013 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:38.169965 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swppx\" (UniqueName: \"kubernetes.io/projected/f451ea84-ed76-4751-87d0-b341564aec23-kube-api-access-swppx\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n78xj\" (UID: \"f451ea84-ed76-4751-87d0-b341564aec23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n78xj" Jul 14 22:21:38.170013 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:38.169992 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f451ea84-ed76-4751-87d0-b341564aec23-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n78xj\" (UID: \"f451ea84-ed76-4751-87d0-b341564aec23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n78xj" Jul 14 22:21:38.170176 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:38.170114 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f451ea84-ed76-4751-87d0-b341564aec23-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n78xj\" (UID: \"f451ea84-ed76-4751-87d0-b341564aec23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n78xj" Jul 14 22:21:38.170229 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:38.170177 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f451ea84-ed76-4751-87d0-b341564aec23-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n78xj\" (UID: \"f451ea84-ed76-4751-87d0-b341564aec23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n78xj" Jul 14 22:21:38.170229 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:38.170201 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f451ea84-ed76-4751-87d0-b341564aec23-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n78xj\" (UID: \"f451ea84-ed76-4751-87d0-b341564aec23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n78xj" Jul 14 22:21:38.172239 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:38.172221 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f451ea84-ed76-4751-87d0-b341564aec23-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n78xj\" (UID: \"f451ea84-ed76-4751-87d0-b341564aec23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n78xj" Jul 14 22:21:38.172409 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:38.172392 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f451ea84-ed76-4751-87d0-b341564aec23-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n78xj\" (UID: \"f451ea84-ed76-4751-87d0-b341564aec23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n78xj" Jul 14 22:21:38.180501 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:38.180470 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swppx\" (UniqueName: \"kubernetes.io/projected/f451ea84-ed76-4751-87d0-b341564aec23-kube-api-access-swppx\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n78xj\" (UID: \"f451ea84-ed76-4751-87d0-b341564aec23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n78xj" Jul 14 22:21:38.312398 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:38.312309 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n78xj" Jul 14 22:21:38.456845 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:38.454573 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-n78xj"] Jul 14 22:21:39.391206 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:39.391171 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n78xj" event={"ID":"f451ea84-ed76-4751-87d0-b341564aec23","Type":"ContainerStarted","Data":"98f378478f8f44164ccfec79910a6f0244931092de2302d44cf7acb37968b874"} Jul 14 22:21:40.397135 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:40.397100 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n78xj" event={"ID":"f451ea84-ed76-4751-87d0-b341564aec23","Type":"ContainerStarted","Data":"d91b98d326a927db7741388956376f32b952c5167a1714c4629db80a112827bc"} Jul 14 22:21:40.434060 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:40.434014 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n78xj" podStartSLOduration=2.47725921 podStartE2EDuration="3.433998729s" totalImagesPullingTime="956.739519ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:21:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:21:40.433530531 +0000 UTC m=+730.792676967" watchObservedRunningTime="2026-07-14 22:21:40.433998729 +0000 UTC m=+730.793145166" Jul 14 22:21:44.592824 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:44.592792 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:21:45.834543 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:45.834502 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:21:48.312641 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:48.312610 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n78xj" Jul 14 22:21:48.313018 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:48.312664 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n78xj" Jul 14 22:21:48.325056 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:48.325037 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n78xj" Jul 14 22:21:48.446497 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:48.446467 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n78xj" Jul 14 22:21:57.630708 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:21:57.630666 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:22:07.725964 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:22:07.725930 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jhjbl"] Jul 14 22:22:07.729325 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:22:07.729308 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jhjbl" Jul 14 22:22:07.737743 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:22:07.737723 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 14 22:22:07.748477 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:22:07.748449 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jhjbl"] Jul 14 22:22:07.823987 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:22:07.823955 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/787afbc7-1a70-4126-a7fb-c337b468ea7b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jhjbl\" (UID: \"787afbc7-1a70-4126-a7fb-c337b468ea7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jhjbl" Jul 14 22:22:07.824139 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:22:07.823993 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/787afbc7-1a70-4126-a7fb-c337b468ea7b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jhjbl\" (UID: \"787afbc7-1a70-4126-a7fb-c337b468ea7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jhjbl" Jul 14 22:22:07.824139 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:22:07.824087 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdfxg\" (UniqueName: \"kubernetes.io/projected/787afbc7-1a70-4126-a7fb-c337b468ea7b-kube-api-access-xdfxg\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jhjbl\" (UID: \"787afbc7-1a70-4126-a7fb-c337b468ea7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jhjbl" Jul 14 22:22:07.824139 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:22:07.824122 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/787afbc7-1a70-4126-a7fb-c337b468ea7b-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jhjbl\" (UID: \"787afbc7-1a70-4126-a7fb-c337b468ea7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jhjbl" Jul 14 22:22:07.824243 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:22:07.824188 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/787afbc7-1a70-4126-a7fb-c337b468ea7b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jhjbl\" (UID: \"787afbc7-1a70-4126-a7fb-c337b468ea7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jhjbl" Jul 14 22:22:07.824243 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:22:07.824224 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/787afbc7-1a70-4126-a7fb-c337b468ea7b-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jhjbl\" (UID: \"787afbc7-1a70-4126-a7fb-c337b468ea7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jhjbl" Jul 14 22:22:07.859150 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:22:07.859099 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:22:07.924846 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:22:07.924820 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdfxg\" (UniqueName: \"kubernetes.io/projected/787afbc7-1a70-4126-a7fb-c337b468ea7b-kube-api-access-xdfxg\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jhjbl\" (UID: \"787afbc7-1a70-4126-a7fb-c337b468ea7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jhjbl" Jul 14 22:22:07.924989 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:22:07.924857 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/787afbc7-1a70-4126-a7fb-c337b468ea7b-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jhjbl\" (UID: \"787afbc7-1a70-4126-a7fb-c337b468ea7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jhjbl" Jul 14 22:22:07.924989 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:22:07.924881 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/787afbc7-1a70-4126-a7fb-c337b468ea7b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jhjbl\" (UID: \"787afbc7-1a70-4126-a7fb-c337b468ea7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jhjbl" Jul 14 22:22:07.924989 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:22:07.924915 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/787afbc7-1a70-4126-a7fb-c337b468ea7b-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jhjbl\" (UID: \"787afbc7-1a70-4126-a7fb-c337b468ea7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jhjbl" Jul 14 22:22:07.924989 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:22:07.924965 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/787afbc7-1a70-4126-a7fb-c337b468ea7b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jhjbl\" (UID: \"787afbc7-1a70-4126-a7fb-c337b468ea7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jhjbl" Jul 14 22:22:07.925191 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:22:07.924999 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/787afbc7-1a70-4126-a7fb-c337b468ea7b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jhjbl\" (UID: \"787afbc7-1a70-4126-a7fb-c337b468ea7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jhjbl" Jul 14 22:22:07.925283 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:22:07.925261 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/787afbc7-1a70-4126-a7fb-c337b468ea7b-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jhjbl\" (UID: \"787afbc7-1a70-4126-a7fb-c337b468ea7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jhjbl" Jul 14 22:22:07.925342 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:22:07.925281 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/787afbc7-1a70-4126-a7fb-c337b468ea7b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jhjbl\" (UID: \"787afbc7-1a70-4126-a7fb-c337b468ea7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jhjbl" Jul 14 22:22:07.925400 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:22:07.925371 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/787afbc7-1a70-4126-a7fb-c337b468ea7b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jhjbl\" (UID: \"787afbc7-1a70-4126-a7fb-c337b468ea7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jhjbl" Jul 14 22:22:07.927178 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:22:07.927154 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/787afbc7-1a70-4126-a7fb-c337b468ea7b-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jhjbl\" (UID: \"787afbc7-1a70-4126-a7fb-c337b468ea7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jhjbl" Jul 14 22:22:07.927363 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:22:07.927348 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/787afbc7-1a70-4126-a7fb-c337b468ea7b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jhjbl\" (UID: \"787afbc7-1a70-4126-a7fb-c337b468ea7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jhjbl" Jul 14 22:22:07.937782 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:22:07.937757 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdfxg\" (UniqueName: \"kubernetes.io/projected/787afbc7-1a70-4126-a7fb-c337b468ea7b-kube-api-access-xdfxg\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jhjbl\" (UID: \"787afbc7-1a70-4126-a7fb-c337b468ea7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jhjbl" Jul 14 22:22:08.040960 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:22:08.040892 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jhjbl" Jul 14 22:22:08.175665 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:22:08.175641 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jhjbl"] Jul 14 22:22:08.177190 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:22:08.177152 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod787afbc7_1a70_4126_a7fb_c337b468ea7b.slice/crio-01bd98282340b5b53184819fe22c67c0cf7a572de4836ade56c8f0b3e0981633 WatchSource:0}: Error finding container 01bd98282340b5b53184819fe22c67c0cf7a572de4836ade56c8f0b3e0981633: Status 404 returned error can't find the container with id 01bd98282340b5b53184819fe22c67c0cf7a572de4836ade56c8f0b3e0981633 Jul 14 22:22:08.530426 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:22:08.530379 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jhjbl" event={"ID":"787afbc7-1a70-4126-a7fb-c337b468ea7b","Type":"ContainerStarted","Data":"80764ba2cfaaec64cb156780101b832961d9fc5829e7ba4ad625464b285a655b"} Jul 14 22:22:08.530596 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:22:08.530443 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jhjbl" event={"ID":"787afbc7-1a70-4126-a7fb-c337b468ea7b","Type":"ContainerStarted","Data":"01bd98282340b5b53184819fe22c67c0cf7a572de4836ade56c8f0b3e0981633"} Jul 14 22:22:08.558250 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:22:08.558201 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jhjbl" podStartSLOduration=1.353955261 podStartE2EDuration="1.558186069s" totalImagesPullingTime="204.230808ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:22:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:22:08.557477878 +0000 UTC m=+758.916624316" watchObservedRunningTime="2026-07-14 22:22:08.558186069 +0000 UTC m=+758.917332543" Jul 14 22:22:18.041690 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:22:18.041650 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jhjbl" Jul 14 22:22:18.042116 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:22:18.041736 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jhjbl" Jul 14 22:22:18.053906 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:22:18.053885 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jhjbl" Jul 14 22:22:18.590613 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:22:18.590588 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jhjbl" Jul 14 22:22:25.331112 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:22:25.331079 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:22:26.638203 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:22:26.638166 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:23:38.893628 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:23:38.893594 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:23:38.982218 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:23:38.982178 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:23:40.222586 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:23:40.222515 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:23:40.556720 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:23:40.556643 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:24:30.057612 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:24:30.057531 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zj4sb_0884baa5-9206-4196-a7c9-b5686623311b/ovn-acl-logging/0.log" Jul 14 22:24:30.059403 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:24:30.059381 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zj4sb_0884baa5-9206-4196-a7c9-b5686623311b/ovn-acl-logging/0.log" Jul 14 22:24:49.518897 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:24:49.518859 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:24:50.945328 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:24:50.945282 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:24:56.305237 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:24:56.305207 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:24:57.558183 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:24:57.558150 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:25:35.473782 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:25:35.473749 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:25:38.560023 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:25:38.559987 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:25:38.678858 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:25:38.678814 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:25:38.808894 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:25:38.808866 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:25:45.131677 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:25:45.131647 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:25:45.206694 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:25:45.206658 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:25:45.336144 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:25:45.336083 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:25:45.564691 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:25:45.564658 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:25:51.958616 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:25:51.958520 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:25:52.058819 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:25:52.058788 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:26:36.267452 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:26:36.267414 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:26:40.676744 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:26:40.676707 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:26:40.753843 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:26:40.753816 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:28:24.639172 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:28:24.639143 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:28:24.728455 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:28:24.728428 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:29:30.087934 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:29:30.087908 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zj4sb_0884baa5-9206-4196-a7c9-b5686623311b/ovn-acl-logging/0.log" Jul 14 22:29:30.091202 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:29:30.091183 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zj4sb_0884baa5-9206-4196-a7c9-b5686623311b/ovn-acl-logging/0.log" Jul 14 22:30:50.879311 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:30:50.879235 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:32:58.047731 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:32:58.047698 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:33:04.692369 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:33:04.692335 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:33:04.793290 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:33:04.793256 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:33:06.704090 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:33:06.704055 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:34:30.121598 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:34:30.121571 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zj4sb_0884baa5-9206-4196-a7c9-b5686623311b/ovn-acl-logging/0.log" Jul 14 22:34:30.125705 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:34:30.125684 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zj4sb_0884baa5-9206-4196-a7c9-b5686623311b/ovn-acl-logging/0.log" Jul 14 22:36:54.068216 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:36:54.068140 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:37:00.821141 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:00.821067 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:37:09.042718 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:09.042685 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:37:14.189008 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:14.188977 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:37:17.325723 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:17.325682 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:37:17.430483 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:17.430450 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:37:17.793178 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:17.793146 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:37:23.831909 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:23.831869 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:37:23.915376 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:23.915339 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:37:24.100598 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:24.100505 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:37:24.197176 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:24.197138 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:37:25.271197 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:25.271158 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:37:25.354361 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:25.354320 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:37:25.360183 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:25.360154 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg"] Jul 14 22:37:25.363484 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:25.363466 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg" Jul 14 22:37:25.366333 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:25.366314 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2b6fcd6d-serving-cert\"" Jul 14 22:37:25.375440 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:25.375422 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg"] Jul 14 22:37:25.426728 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:25.426694 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f721582e-eda2-4024-9216-dc6d561421a1-maas-api-tls\") pod \"maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg\" (UID: \"f721582e-eda2-4024-9216-dc6d561421a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg" Jul 14 22:37:25.426858 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:25.426741 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5bfg\" (UniqueName: \"kubernetes.io/projected/f721582e-eda2-4024-9216-dc6d561421a1-kube-api-access-k5bfg\") pod \"maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg\" (UID: \"f721582e-eda2-4024-9216-dc6d561421a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg" Jul 14 22:37:25.426858 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:25.426765 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f721582e-eda2-4024-9216-dc6d561421a1-openshift-service-ca\") pod \"maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg\" (UID: \"f721582e-eda2-4024-9216-dc6d561421a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg" Jul 14 22:37:25.429840 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:25.429818 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:37:25.527922 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:25.527848 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f721582e-eda2-4024-9216-dc6d561421a1-maas-api-tls\") pod \"maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg\" (UID: \"f721582e-eda2-4024-9216-dc6d561421a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg" Jul 14 22:37:25.527922 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:25.527899 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5bfg\" (UniqueName: \"kubernetes.io/projected/f721582e-eda2-4024-9216-dc6d561421a1-kube-api-access-k5bfg\") pod \"maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg\" (UID: \"f721582e-eda2-4024-9216-dc6d561421a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg" Jul 14 22:37:25.528101 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:25.527933 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f721582e-eda2-4024-9216-dc6d561421a1-openshift-service-ca\") pod \"maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg\" (UID: \"f721582e-eda2-4024-9216-dc6d561421a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg" Jul 14 22:37:25.528630 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:25.528611 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f721582e-eda2-4024-9216-dc6d561421a1-openshift-service-ca\") pod \"maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg\" (UID: \"f721582e-eda2-4024-9216-dc6d561421a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg" Jul 14 22:37:25.530198 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:25.530182 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f721582e-eda2-4024-9216-dc6d561421a1-maas-api-tls\") pod \"maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg\" (UID: \"f721582e-eda2-4024-9216-dc6d561421a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg" Jul 14 22:37:25.539771 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:25.539749 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5bfg\" (UniqueName: \"kubernetes.io/projected/f721582e-eda2-4024-9216-dc6d561421a1-kube-api-access-k5bfg\") pod \"maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg\" (UID: \"f721582e-eda2-4024-9216-dc6d561421a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg" Jul 14 22:37:25.561687 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:25.561654 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:37:25.673970 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:25.673935 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg" Jul 14 22:37:25.806985 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:25.806960 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg"] Jul 14 22:37:25.808692 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:37:25.808665 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf721582e_eda2_4024_9216_dc6d561421a1.slice/crio-15231b9633b31d8dcfb39320f2f0214f5ba87d787d02aec3c19cc7a4d84f1bf8 WatchSource:0}: Error finding container 15231b9633b31d8dcfb39320f2f0214f5ba87d787d02aec3c19cc7a4d84f1bf8: Status 404 returned error can't find the container with id 15231b9633b31d8dcfb39320f2f0214f5ba87d787d02aec3c19cc7a4d84f1bf8 Jul 14 22:37:25.810051 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:25.810031 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 22:37:26.021727 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:26.021688 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg" event={"ID":"f721582e-eda2-4024-9216-dc6d561421a1","Type":"ContainerStarted","Data":"15231b9633b31d8dcfb39320f2f0214f5ba87d787d02aec3c19cc7a4d84f1bf8"} Jul 14 22:37:26.079197 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:26.079118 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:37:26.176939 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:26.176902 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:37:27.027654 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:27.027615 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg" event={"ID":"f721582e-eda2-4024-9216-dc6d561421a1","Type":"ContainerStarted","Data":"12440293fae5bdd613cfb235bca8706f22339e4e573b15b606fa85013a52096d"} Jul 14 22:37:27.028005 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:27.027710 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg" Jul 14 22:37:27.045918 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:27.045864 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg" podStartSLOduration=1.5200284530000001 podStartE2EDuration="2.045845766s" totalImagesPullingTime="525.817313ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:37:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:37:27.045122344 +0000 UTC m=+1677.404268783" watchObservedRunningTime="2026-07-14 22:37:27.045845766 +0000 UTC m=+1677.404992204" Jul 14 22:37:28.126661 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:28.126629 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:37:33.037774 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:33.037746 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg" Jul 14 22:37:40.809071 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:40.809036 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg"] Jul 14 22:37:40.809478 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:40.809268 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg" podUID="f721582e-eda2-4024-9216-dc6d561421a1" containerName="maas-api" containerID="cri-o://12440293fae5bdd613cfb235bca8706f22339e4e573b15b606fa85013a52096d" gracePeriod=30 Jul 14 22:37:40.849320 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:40.849294 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:37:41.052159 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:41.052138 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg" Jul 14 22:37:41.099934 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:41.099855 2550 generic.go:356] "Generic (PLEG): container finished" podID="f721582e-eda2-4024-9216-dc6d561421a1" containerID="12440293fae5bdd613cfb235bca8706f22339e4e573b15b606fa85013a52096d" exitCode=0 Jul 14 22:37:41.099934 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:41.099894 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg" Jul 14 22:37:41.100099 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:41.099931 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg" event={"ID":"f721582e-eda2-4024-9216-dc6d561421a1","Type":"ContainerDied","Data":"12440293fae5bdd613cfb235bca8706f22339e4e573b15b606fa85013a52096d"} Jul 14 22:37:41.100099 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:41.099985 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg" event={"ID":"f721582e-eda2-4024-9216-dc6d561421a1","Type":"ContainerDied","Data":"15231b9633b31d8dcfb39320f2f0214f5ba87d787d02aec3c19cc7a4d84f1bf8"} Jul 14 22:37:41.100099 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:41.100015 2550 scope.go:117] "RemoveContainer" containerID="12440293fae5bdd613cfb235bca8706f22339e4e573b15b606fa85013a52096d" Jul 14 22:37:41.108285 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:41.108268 2550 scope.go:117] "RemoveContainer" containerID="12440293fae5bdd613cfb235bca8706f22339e4e573b15b606fa85013a52096d" Jul 14 22:37:41.108525 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:37:41.108507 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12440293fae5bdd613cfb235bca8706f22339e4e573b15b606fa85013a52096d\": container with ID starting with 12440293fae5bdd613cfb235bca8706f22339e4e573b15b606fa85013a52096d not found: ID does not exist" containerID="12440293fae5bdd613cfb235bca8706f22339e4e573b15b606fa85013a52096d" Jul 14 22:37:41.108599 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:41.108535 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12440293fae5bdd613cfb235bca8706f22339e4e573b15b606fa85013a52096d"} err="failed to get container status \"12440293fae5bdd613cfb235bca8706f22339e4e573b15b606fa85013a52096d\": rpc error: code = NotFound desc = could not find container \"12440293fae5bdd613cfb235bca8706f22339e4e573b15b606fa85013a52096d\": container with ID starting with 12440293fae5bdd613cfb235bca8706f22339e4e573b15b606fa85013a52096d not found: ID does not exist" Jul 14 22:37:41.171219 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:41.171195 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5bfg\" (UniqueName: \"kubernetes.io/projected/f721582e-eda2-4024-9216-dc6d561421a1-kube-api-access-k5bfg\") pod \"f721582e-eda2-4024-9216-dc6d561421a1\" (UID: \"f721582e-eda2-4024-9216-dc6d561421a1\") " Jul 14 22:37:41.171319 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:41.171288 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f721582e-eda2-4024-9216-dc6d561421a1-maas-api-tls\") pod \"f721582e-eda2-4024-9216-dc6d561421a1\" (UID: \"f721582e-eda2-4024-9216-dc6d561421a1\") " Jul 14 22:37:41.171319 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:41.171307 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f721582e-eda2-4024-9216-dc6d561421a1-openshift-service-ca\") pod \"f721582e-eda2-4024-9216-dc6d561421a1\" (UID: \"f721582e-eda2-4024-9216-dc6d561421a1\") " Jul 14 22:37:41.171709 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:41.171690 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f721582e-eda2-4024-9216-dc6d561421a1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f721582e-eda2-4024-9216-dc6d561421a1" (UID: "f721582e-eda2-4024-9216-dc6d561421a1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:37:41.173287 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:41.173267 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f721582e-eda2-4024-9216-dc6d561421a1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f721582e-eda2-4024-9216-dc6d561421a1" (UID: "f721582e-eda2-4024-9216-dc6d561421a1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 22:37:41.173393 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:41.173374 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f721582e-eda2-4024-9216-dc6d561421a1-kube-api-access-k5bfg" (OuterVolumeSpecName: "kube-api-access-k5bfg") pod "f721582e-eda2-4024-9216-dc6d561421a1" (UID: "f721582e-eda2-4024-9216-dc6d561421a1"). InnerVolumeSpecName "kube-api-access-k5bfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:37:41.272113 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:41.272082 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f721582e-eda2-4024-9216-dc6d561421a1-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:37:41.272113 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:41.272107 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f721582e-eda2-4024-9216-dc6d561421a1-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:37:41.272271 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:41.272120 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5bfg\" (UniqueName: \"kubernetes.io/projected/f721582e-eda2-4024-9216-dc6d561421a1-kube-api-access-k5bfg\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:37:41.422436 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:41.422410 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg"] Jul 14 22:37:41.428108 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:41.428084 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2b6fcd6d-8f6f77df8-wgswg"] Jul 14 22:37:42.194490 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:42.194462 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f721582e-eda2-4024-9216-dc6d561421a1" path="/var/lib/kubelet/pods/f721582e-eda2-4024-9216-dc6d561421a1/volumes" Jul 14 22:37:45.399307 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:45.399270 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:37:51.675775 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:51.675696 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:37:51.774314 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:51.774279 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:37:55.628145 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:55.628115 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:37:55.980581 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:55.980533 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:37:56.088104 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:56.088071 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:37:56.192173 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:56.192138 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:37:57.540759 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:57.540726 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:37:57.783024 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:57.782985 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-91e25165-6659657d8-jz6st"] Jul 14 22:37:57.783341 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:57.783327 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f721582e-eda2-4024-9216-dc6d561421a1" containerName="maas-api" Jul 14 22:37:57.783341 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:57.783341 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f721582e-eda2-4024-9216-dc6d561421a1" containerName="maas-api" Jul 14 22:37:57.783498 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:57.783393 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="f721582e-eda2-4024-9216-dc6d561421a1" containerName="maas-api" Jul 14 22:37:57.787635 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:57.787614 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91e25165-6659657d8-jz6st" Jul 14 22:37:57.790316 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:57.790295 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-91e25165-serving-cert\"" Jul 14 22:37:57.799110 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:57.799091 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-91e25165-6659657d8-jz6st"] Jul 14 22:37:57.816775 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:57.816754 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mk8b\" (UniqueName: \"kubernetes.io/projected/33519071-6b68-4d49-9d16-ad2aa68010b1-kube-api-access-9mk8b\") pod \"maas-api-e2e-migrate-91e25165-6659657d8-jz6st\" (UID: \"33519071-6b68-4d49-9d16-ad2aa68010b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91e25165-6659657d8-jz6st" Jul 14 22:37:57.816886 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:57.816802 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33519071-6b68-4d49-9d16-ad2aa68010b1-maas-api-tls\") pod \"maas-api-e2e-migrate-91e25165-6659657d8-jz6st\" (UID: \"33519071-6b68-4d49-9d16-ad2aa68010b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91e25165-6659657d8-jz6st" Jul 14 22:37:57.816886 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:57.816851 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33519071-6b68-4d49-9d16-ad2aa68010b1-openshift-service-ca\") pod \"maas-api-e2e-migrate-91e25165-6659657d8-jz6st\" (UID: \"33519071-6b68-4d49-9d16-ad2aa68010b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91e25165-6659657d8-jz6st" Jul 14 22:37:57.820234 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:57.820211 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:37:57.917495 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:57.917467 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33519071-6b68-4d49-9d16-ad2aa68010b1-maas-api-tls\") pod \"maas-api-e2e-migrate-91e25165-6659657d8-jz6st\" (UID: \"33519071-6b68-4d49-9d16-ad2aa68010b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91e25165-6659657d8-jz6st" Jul 14 22:37:57.917670 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:57.917506 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33519071-6b68-4d49-9d16-ad2aa68010b1-openshift-service-ca\") pod \"maas-api-e2e-migrate-91e25165-6659657d8-jz6st\" (UID: \"33519071-6b68-4d49-9d16-ad2aa68010b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91e25165-6659657d8-jz6st" Jul 14 22:37:57.917670 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:57.917569 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mk8b\" (UniqueName: \"kubernetes.io/projected/33519071-6b68-4d49-9d16-ad2aa68010b1-kube-api-access-9mk8b\") pod \"maas-api-e2e-migrate-91e25165-6659657d8-jz6st\" (UID: \"33519071-6b68-4d49-9d16-ad2aa68010b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91e25165-6659657d8-jz6st" Jul 14 22:37:57.918165 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:57.918148 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33519071-6b68-4d49-9d16-ad2aa68010b1-openshift-service-ca\") pod \"maas-api-e2e-migrate-91e25165-6659657d8-jz6st\" (UID: \"33519071-6b68-4d49-9d16-ad2aa68010b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91e25165-6659657d8-jz6st" Jul 14 22:37:57.919766 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:57.919742 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33519071-6b68-4d49-9d16-ad2aa68010b1-maas-api-tls\") pod \"maas-api-e2e-migrate-91e25165-6659657d8-jz6st\" (UID: \"33519071-6b68-4d49-9d16-ad2aa68010b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91e25165-6659657d8-jz6st" Jul 14 22:37:57.938566 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:57.938534 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mk8b\" (UniqueName: \"kubernetes.io/projected/33519071-6b68-4d49-9d16-ad2aa68010b1-kube-api-access-9mk8b\") pod \"maas-api-e2e-migrate-91e25165-6659657d8-jz6st\" (UID: \"33519071-6b68-4d49-9d16-ad2aa68010b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91e25165-6659657d8-jz6st" Jul 14 22:37:57.960626 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:57.960600 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:37:58.098325 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:58.098231 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91e25165-6659657d8-jz6st" Jul 14 22:37:58.233537 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:58.233516 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-91e25165-6659657d8-jz6st"] Jul 14 22:37:58.235375 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:37:58.235346 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33519071_6b68_4d49_9d16_ad2aa68010b1.slice/crio-e2f3fe2344e93927d631bb3a6670b17cd62881604f87b9b58688b20724afc401 WatchSource:0}: Error finding container e2f3fe2344e93927d631bb3a6670b17cd62881604f87b9b58688b20724afc401: Status 404 returned error can't find the container with id e2f3fe2344e93927d631bb3a6670b17cd62881604f87b9b58688b20724afc401 Jul 14 22:37:58.310007 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:58.309969 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:37:58.409518 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:58.409440 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:37:59.197649 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:59.197617 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91e25165-6659657d8-jz6st" event={"ID":"33519071-6b68-4d49-9d16-ad2aa68010b1","Type":"ContainerStarted","Data":"951c56d752dc16c4011d4455bcd7351d31174ec728a929dde2e2a81ddecf5644"} Jul 14 22:37:59.198016 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:59.197664 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91e25165-6659657d8-jz6st" event={"ID":"33519071-6b68-4d49-9d16-ad2aa68010b1","Type":"ContainerStarted","Data":"e2f3fe2344e93927d631bb3a6670b17cd62881604f87b9b58688b20724afc401"} Jul 14 22:37:59.198016 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:59.197754 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91e25165-6659657d8-jz6st" Jul 14 22:37:59.221215 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:37:59.221169 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91e25165-6659657d8-jz6st" podStartSLOduration=1.939272666 podStartE2EDuration="2.221154777s" totalImagesPullingTime="281.882111ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:37:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:37:59.219909157 +0000 UTC m=+1709.579055596" watchObservedRunningTime="2026-07-14 22:37:59.221154777 +0000 UTC m=+1709.580301214" Jul 14 22:38:02.019413 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:38:02.019376 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:38:02.120578 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:38:02.120526 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:38:05.208723 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:38:05.208685 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91e25165-6659657d8-jz6st" Jul 14 22:38:05.248334 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:38:05.248300 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:38:37.093118 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:38:37.093076 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:38:54.130355 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:38:54.130322 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:38:54.230980 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:38:54.230952 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:39:02.119399 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:02.119368 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:39:02.242041 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:02.242005 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:39:07.672584 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:07.672481 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-91e25165-6659657d8-jz6st"] Jul 14 22:39:07.672938 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:07.672725 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91e25165-6659657d8-jz6st" podUID="33519071-6b68-4d49-9d16-ad2aa68010b1" containerName="maas-api" containerID="cri-o://951c56d752dc16c4011d4455bcd7351d31174ec728a929dde2e2a81ddecf5644" gracePeriod=30 Jul 14 22:39:07.698607 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:07.698575 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:39:07.925475 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:07.925424 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91e25165-6659657d8-jz6st" Jul 14 22:39:08.020047 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:08.020024 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mk8b\" (UniqueName: \"kubernetes.io/projected/33519071-6b68-4d49-9d16-ad2aa68010b1-kube-api-access-9mk8b\") pod \"33519071-6b68-4d49-9d16-ad2aa68010b1\" (UID: \"33519071-6b68-4d49-9d16-ad2aa68010b1\") " Jul 14 22:39:08.020199 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:08.020069 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33519071-6b68-4d49-9d16-ad2aa68010b1-openshift-service-ca\") pod \"33519071-6b68-4d49-9d16-ad2aa68010b1\" (UID: \"33519071-6b68-4d49-9d16-ad2aa68010b1\") " Jul 14 22:39:08.020199 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:08.020098 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33519071-6b68-4d49-9d16-ad2aa68010b1-maas-api-tls\") pod \"33519071-6b68-4d49-9d16-ad2aa68010b1\" (UID: \"33519071-6b68-4d49-9d16-ad2aa68010b1\") " Jul 14 22:39:08.020479 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:08.020446 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33519071-6b68-4d49-9d16-ad2aa68010b1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "33519071-6b68-4d49-9d16-ad2aa68010b1" (UID: "33519071-6b68-4d49-9d16-ad2aa68010b1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:39:08.022148 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:08.022127 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33519071-6b68-4d49-9d16-ad2aa68010b1-kube-api-access-9mk8b" (OuterVolumeSpecName: "kube-api-access-9mk8b") pod "33519071-6b68-4d49-9d16-ad2aa68010b1" (UID: "33519071-6b68-4d49-9d16-ad2aa68010b1"). InnerVolumeSpecName "kube-api-access-9mk8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:39:08.022390 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:08.022147 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33519071-6b68-4d49-9d16-ad2aa68010b1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "33519071-6b68-4d49-9d16-ad2aa68010b1" (UID: "33519071-6b68-4d49-9d16-ad2aa68010b1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 22:39:08.121478 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:08.121440 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mk8b\" (UniqueName: \"kubernetes.io/projected/33519071-6b68-4d49-9d16-ad2aa68010b1-kube-api-access-9mk8b\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:39:08.121478 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:08.121468 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33519071-6b68-4d49-9d16-ad2aa68010b1-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:39:08.121478 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:08.121478 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33519071-6b68-4d49-9d16-ad2aa68010b1-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:39:08.532392 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:08.532299 2550 generic.go:356] "Generic (PLEG): container finished" podID="33519071-6b68-4d49-9d16-ad2aa68010b1" containerID="951c56d752dc16c4011d4455bcd7351d31174ec728a929dde2e2a81ddecf5644" exitCode=0 Jul 14 22:39:08.532392 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:08.532345 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91e25165-6659657d8-jz6st" Jul 14 22:39:08.532666 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:08.532386 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91e25165-6659657d8-jz6st" event={"ID":"33519071-6b68-4d49-9d16-ad2aa68010b1","Type":"ContainerDied","Data":"951c56d752dc16c4011d4455bcd7351d31174ec728a929dde2e2a81ddecf5644"} Jul 14 22:39:08.532666 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:08.532452 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-91e25165-6659657d8-jz6st" event={"ID":"33519071-6b68-4d49-9d16-ad2aa68010b1","Type":"ContainerDied","Data":"e2f3fe2344e93927d631bb3a6670b17cd62881604f87b9b58688b20724afc401"} Jul 14 22:39:08.532666 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:08.532485 2550 scope.go:117] "RemoveContainer" containerID="951c56d752dc16c4011d4455bcd7351d31174ec728a929dde2e2a81ddecf5644" Jul 14 22:39:08.540886 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:08.540867 2550 scope.go:117] "RemoveContainer" containerID="951c56d752dc16c4011d4455bcd7351d31174ec728a929dde2e2a81ddecf5644" Jul 14 22:39:08.541164 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:39:08.541145 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"951c56d752dc16c4011d4455bcd7351d31174ec728a929dde2e2a81ddecf5644\": container with ID starting with 951c56d752dc16c4011d4455bcd7351d31174ec728a929dde2e2a81ddecf5644 not found: ID does not exist" containerID="951c56d752dc16c4011d4455bcd7351d31174ec728a929dde2e2a81ddecf5644" Jul 14 22:39:08.541220 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:08.541173 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"951c56d752dc16c4011d4455bcd7351d31174ec728a929dde2e2a81ddecf5644"} err="failed to get container status \"951c56d752dc16c4011d4455bcd7351d31174ec728a929dde2e2a81ddecf5644\": rpc error: code = NotFound desc = could not find container \"951c56d752dc16c4011d4455bcd7351d31174ec728a929dde2e2a81ddecf5644\": container with ID starting with 951c56d752dc16c4011d4455bcd7351d31174ec728a929dde2e2a81ddecf5644 not found: ID does not exist" Jul 14 22:39:08.557112 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:08.557088 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-91e25165-6659657d8-jz6st"] Jul 14 22:39:08.565000 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:08.564981 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-91e25165-6659657d8-jz6st"] Jul 14 22:39:08.666877 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:08.666848 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:39:09.149481 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:09.149447 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:39:10.199763 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:10.197619 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33519071-6b68-4d49-9d16-ad2aa68010b1" path="/var/lib/kubelet/pods/33519071-6b68-4d49-9d16-ad2aa68010b1/volumes" Jul 14 22:39:14.623154 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:14.623113 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:39:14.723785 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:14.723751 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:39:14.918296 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:14.918265 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:39:15.011643 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:15.011608 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:39:16.128848 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:16.128817 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:39:16.320202 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:16.320167 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:39:16.331611 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:16.331586 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-577df277-86d669597d-jr56j"] Jul 14 22:39:16.331963 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:16.331947 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33519071-6b68-4d49-9d16-ad2aa68010b1" containerName="maas-api" Jul 14 22:39:16.332043 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:16.331966 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="33519071-6b68-4d49-9d16-ad2aa68010b1" containerName="maas-api" Jul 14 22:39:16.332100 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:16.332044 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="33519071-6b68-4d49-9d16-ad2aa68010b1" containerName="maas-api" Jul 14 22:39:16.336290 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:16.336272 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-577df277-86d669597d-jr56j" Jul 14 22:39:16.339179 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:16.339161 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-577df277-serving-cert\"" Jul 14 22:39:16.357950 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:16.357919 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-577df277-86d669597d-jr56j"] Jul 14 22:39:16.426983 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:16.426954 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:39:16.497725 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:16.497697 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4c6cce2-8bd9-4f18-96b1-5853540c12c3-maas-api-tls\") pod \"maas-api-e2e-ait-577df277-86d669597d-jr56j\" (UID: \"a4c6cce2-8bd9-4f18-96b1-5853540c12c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-577df277-86d669597d-jr56j" Jul 14 22:39:16.497864 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:16.497779 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dfzg\" (UniqueName: \"kubernetes.io/projected/a4c6cce2-8bd9-4f18-96b1-5853540c12c3-kube-api-access-2dfzg\") pod \"maas-api-e2e-ait-577df277-86d669597d-jr56j\" (UID: \"a4c6cce2-8bd9-4f18-96b1-5853540c12c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-577df277-86d669597d-jr56j" Jul 14 22:39:16.497864 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:16.497819 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4c6cce2-8bd9-4f18-96b1-5853540c12c3-openshift-service-ca\") pod \"maas-api-e2e-ait-577df277-86d669597d-jr56j\" (UID: \"a4c6cce2-8bd9-4f18-96b1-5853540c12c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-577df277-86d669597d-jr56j" Jul 14 22:39:16.598471 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:16.598432 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dfzg\" (UniqueName: \"kubernetes.io/projected/a4c6cce2-8bd9-4f18-96b1-5853540c12c3-kube-api-access-2dfzg\") pod \"maas-api-e2e-ait-577df277-86d669597d-jr56j\" (UID: \"a4c6cce2-8bd9-4f18-96b1-5853540c12c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-577df277-86d669597d-jr56j" Jul 14 22:39:16.598674 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:16.598482 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4c6cce2-8bd9-4f18-96b1-5853540c12c3-openshift-service-ca\") pod \"maas-api-e2e-ait-577df277-86d669597d-jr56j\" (UID: \"a4c6cce2-8bd9-4f18-96b1-5853540c12c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-577df277-86d669597d-jr56j" Jul 14 22:39:16.598674 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:16.598513 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4c6cce2-8bd9-4f18-96b1-5853540c12c3-maas-api-tls\") pod \"maas-api-e2e-ait-577df277-86d669597d-jr56j\" (UID: \"a4c6cce2-8bd9-4f18-96b1-5853540c12c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-577df277-86d669597d-jr56j" Jul 14 22:39:16.598674 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:39:16.598636 2550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-577df277-serving-cert: secret "maas-api-e2e-ait-577df277-serving-cert" not found Jul 14 22:39:16.598852 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:39:16.598699 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4c6cce2-8bd9-4f18-96b1-5853540c12c3-maas-api-tls podName:a4c6cce2-8bd9-4f18-96b1-5853540c12c3 nodeName:}" failed. No retries permitted until 2026-07-14 22:39:17.09868423 +0000 UTC m=+1787.457830645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a4c6cce2-8bd9-4f18-96b1-5853540c12c3-maas-api-tls") pod "maas-api-e2e-ait-577df277-86d669597d-jr56j" (UID: "a4c6cce2-8bd9-4f18-96b1-5853540c12c3") : secret "maas-api-e2e-ait-577df277-serving-cert" not found Jul 14 22:39:16.599164 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:16.599144 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4c6cce2-8bd9-4f18-96b1-5853540c12c3-openshift-service-ca\") pod \"maas-api-e2e-ait-577df277-86d669597d-jr56j\" (UID: \"a4c6cce2-8bd9-4f18-96b1-5853540c12c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-577df277-86d669597d-jr56j" Jul 14 22:39:16.618919 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:16.618887 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dfzg\" (UniqueName: \"kubernetes.io/projected/a4c6cce2-8bd9-4f18-96b1-5853540c12c3-kube-api-access-2dfzg\") pod \"maas-api-e2e-ait-577df277-86d669597d-jr56j\" (UID: \"a4c6cce2-8bd9-4f18-96b1-5853540c12c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-577df277-86d669597d-jr56j" Jul 14 22:39:16.840584 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:16.840482 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:39:17.104990 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:17.104911 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4c6cce2-8bd9-4f18-96b1-5853540c12c3-maas-api-tls\") pod \"maas-api-e2e-ait-577df277-86d669597d-jr56j\" (UID: \"a4c6cce2-8bd9-4f18-96b1-5853540c12c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-577df277-86d669597d-jr56j" Jul 14 22:39:17.107239 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:17.107209 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4c6cce2-8bd9-4f18-96b1-5853540c12c3-maas-api-tls\") pod \"maas-api-e2e-ait-577df277-86d669597d-jr56j\" (UID: \"a4c6cce2-8bd9-4f18-96b1-5853540c12c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-577df277-86d669597d-jr56j" Jul 14 22:39:17.138952 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:17.138924 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:39:17.240345 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:17.240314 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:39:17.246418 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:17.246397 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-577df277-86d669597d-jr56j" Jul 14 22:39:17.390304 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:17.390278 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-577df277-86d669597d-jr56j"] Jul 14 22:39:17.392216 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:39:17.392180 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4c6cce2_8bd9_4f18_96b1_5853540c12c3.slice/crio-8ac332f8b1452427eebb76ee41138c2d3a72351c1ea059ad2aab5f98dee8367d WatchSource:0}: Error finding container 8ac332f8b1452427eebb76ee41138c2d3a72351c1ea059ad2aab5f98dee8367d: Status 404 returned error can't find the container with id 8ac332f8b1452427eebb76ee41138c2d3a72351c1ea059ad2aab5f98dee8367d Jul 14 22:39:17.578242 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:17.578209 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-577df277-86d669597d-jr56j" event={"ID":"a4c6cce2-8bd9-4f18-96b1-5853540c12c3","Type":"ContainerStarted","Data":"8ac332f8b1452427eebb76ee41138c2d3a72351c1ea059ad2aab5f98dee8367d"} Jul 14 22:39:18.396004 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:18.395972 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:39:18.585734 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:18.585699 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-577df277-86d669597d-jr56j" event={"ID":"a4c6cce2-8bd9-4f18-96b1-5853540c12c3","Type":"ContainerStarted","Data":"5b39b4e4802b7f6f1e3d7aea489a872726c70fd8835d0b3c22a9c463b5eec755"} Jul 14 22:39:18.585902 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:18.585758 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-577df277-86d669597d-jr56j" Jul 14 22:39:18.606254 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:18.606210 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-577df277-86d669597d-jr56j" podStartSLOduration=2.149242782 podStartE2EDuration="2.606195246s" totalImagesPullingTime="456.952464ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:39:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:39:18.604188175 +0000 UTC m=+1788.963334614" watchObservedRunningTime="2026-07-14 22:39:18.606195246 +0000 UTC m=+1788.965341682" Jul 14 22:39:24.596307 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:24.596227 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-577df277-86d669597d-jr56j" Jul 14 22:39:30.153629 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:30.153601 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zj4sb_0884baa5-9206-4196-a7c9-b5686623311b/ovn-acl-logging/0.log" Jul 14 22:39:30.158438 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:30.158419 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zj4sb_0884baa5-9206-4196-a7c9-b5686623311b/ovn-acl-logging/0.log" Jul 14 22:39:41.955354 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:41.955324 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:39:42.118008 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:39:42.117970 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:40:20.374756 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:20.374718 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:40:20.470697 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:20.470663 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:40:22.189104 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:22.189070 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:40:22.364979 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:22.364943 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:40:22.530724 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:22.530647 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-577df277-86d669597d-jr56j"] Jul 14 22:40:22.530893 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:22.530869 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-577df277-86d669597d-jr56j" podUID="a4c6cce2-8bd9-4f18-96b1-5853540c12c3" containerName="maas-api" containerID="cri-o://5b39b4e4802b7f6f1e3d7aea489a872726c70fd8835d0b3c22a9c463b5eec755" gracePeriod=30 Jul 14 22:40:22.575084 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:22.575056 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:40:22.774717 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:22.774693 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-577df277-86d669597d-jr56j" Jul 14 22:40:22.862267 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:22.862195 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4c6cce2-8bd9-4f18-96b1-5853540c12c3-openshift-service-ca\") pod \"a4c6cce2-8bd9-4f18-96b1-5853540c12c3\" (UID: \"a4c6cce2-8bd9-4f18-96b1-5853540c12c3\") " Jul 14 22:40:22.862267 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:22.862257 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dfzg\" (UniqueName: \"kubernetes.io/projected/a4c6cce2-8bd9-4f18-96b1-5853540c12c3-kube-api-access-2dfzg\") pod \"a4c6cce2-8bd9-4f18-96b1-5853540c12c3\" (UID: \"a4c6cce2-8bd9-4f18-96b1-5853540c12c3\") " Jul 14 22:40:22.862488 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:22.862302 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4c6cce2-8bd9-4f18-96b1-5853540c12c3-maas-api-tls\") pod \"a4c6cce2-8bd9-4f18-96b1-5853540c12c3\" (UID: \"a4c6cce2-8bd9-4f18-96b1-5853540c12c3\") " Jul 14 22:40:22.862628 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:22.862607 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4c6cce2-8bd9-4f18-96b1-5853540c12c3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a4c6cce2-8bd9-4f18-96b1-5853540c12c3" (UID: "a4c6cce2-8bd9-4f18-96b1-5853540c12c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:40:22.864287 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:22.864263 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4c6cce2-8bd9-4f18-96b1-5853540c12c3-kube-api-access-2dfzg" (OuterVolumeSpecName: "kube-api-access-2dfzg") pod "a4c6cce2-8bd9-4f18-96b1-5853540c12c3" (UID: "a4c6cce2-8bd9-4f18-96b1-5853540c12c3"). InnerVolumeSpecName "kube-api-access-2dfzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:40:22.864378 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:22.864293 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4c6cce2-8bd9-4f18-96b1-5853540c12c3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a4c6cce2-8bd9-4f18-96b1-5853540c12c3" (UID: "a4c6cce2-8bd9-4f18-96b1-5853540c12c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 22:40:22.904681 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:22.904654 2550 generic.go:356] "Generic (PLEG): container finished" podID="a4c6cce2-8bd9-4f18-96b1-5853540c12c3" containerID="5b39b4e4802b7f6f1e3d7aea489a872726c70fd8835d0b3c22a9c463b5eec755" exitCode=0 Jul 14 22:40:22.904780 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:22.904699 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-577df277-86d669597d-jr56j" Jul 14 22:40:22.904780 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:22.904741 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-577df277-86d669597d-jr56j" event={"ID":"a4c6cce2-8bd9-4f18-96b1-5853540c12c3","Type":"ContainerDied","Data":"5b39b4e4802b7f6f1e3d7aea489a872726c70fd8835d0b3c22a9c463b5eec755"} Jul 14 22:40:22.904856 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:22.904796 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-577df277-86d669597d-jr56j" event={"ID":"a4c6cce2-8bd9-4f18-96b1-5853540c12c3","Type":"ContainerDied","Data":"8ac332f8b1452427eebb76ee41138c2d3a72351c1ea059ad2aab5f98dee8367d"} Jul 14 22:40:22.904856 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:22.904824 2550 scope.go:117] "RemoveContainer" containerID="5b39b4e4802b7f6f1e3d7aea489a872726c70fd8835d0b3c22a9c463b5eec755" Jul 14 22:40:22.913274 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:22.913260 2550 scope.go:117] "RemoveContainer" containerID="5b39b4e4802b7f6f1e3d7aea489a872726c70fd8835d0b3c22a9c463b5eec755" Jul 14 22:40:22.913566 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:40:22.913533 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b39b4e4802b7f6f1e3d7aea489a872726c70fd8835d0b3c22a9c463b5eec755\": container with ID starting with 5b39b4e4802b7f6f1e3d7aea489a872726c70fd8835d0b3c22a9c463b5eec755 not found: ID does not exist" containerID="5b39b4e4802b7f6f1e3d7aea489a872726c70fd8835d0b3c22a9c463b5eec755" Jul 14 22:40:22.913645 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:22.913573 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b39b4e4802b7f6f1e3d7aea489a872726c70fd8835d0b3c22a9c463b5eec755"} err="failed to get container status \"5b39b4e4802b7f6f1e3d7aea489a872726c70fd8835d0b3c22a9c463b5eec755\": rpc error: code = NotFound desc = could not find container \"5b39b4e4802b7f6f1e3d7aea489a872726c70fd8835d0b3c22a9c463b5eec755\": container with ID starting with 5b39b4e4802b7f6f1e3d7aea489a872726c70fd8835d0b3c22a9c463b5eec755 not found: ID does not exist" Jul 14 22:40:22.933617 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:22.933591 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-577df277-86d669597d-jr56j"] Jul 14 22:40:22.937441 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:22.937421 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-577df277-86d669597d-jr56j"] Jul 14 22:40:22.963653 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:22.963632 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dfzg\" (UniqueName: \"kubernetes.io/projected/a4c6cce2-8bd9-4f18-96b1-5853540c12c3-kube-api-access-2dfzg\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:40:22.963653 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:22.963650 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4c6cce2-8bd9-4f18-96b1-5853540c12c3-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:40:22.963765 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:22.963661 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4c6cce2-8bd9-4f18-96b1-5853540c12c3-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:40:24.195539 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:24.195507 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4c6cce2-8bd9-4f18-96b1-5853540c12c3" path="/var/lib/kubelet/pods/a4c6cce2-8bd9-4f18-96b1-5853540c12c3/volumes" Jul 14 22:40:26.493068 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:26.493032 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:40:34.775312 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:34.775279 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:40:34.888251 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:34.888222 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:40:37.650580 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:37.650515 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:40:38.037610 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:38.037571 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:40:39.228659 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:39.228630 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:40:39.353197 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:39.353161 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6205e423-54f664dff7-jltx5"] Jul 14 22:40:39.353610 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:39.353591 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4c6cce2-8bd9-4f18-96b1-5853540c12c3" containerName="maas-api" Jul 14 22:40:39.353610 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:39.353609 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4c6cce2-8bd9-4f18-96b1-5853540c12c3" containerName="maas-api" Jul 14 22:40:39.353758 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:39.353672 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4c6cce2-8bd9-4f18-96b1-5853540c12c3" containerName="maas-api" Jul 14 22:40:39.356747 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:39.356729 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6205e423-54f664dff7-jltx5" Jul 14 22:40:39.360706 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:39.360684 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-6205e423-serving-cert\"" Jul 14 22:40:39.393042 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:39.393016 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6205e423-54f664dff7-jltx5"] Jul 14 22:40:39.400584 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:39.400543 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac31ecfb-a438-42f6-805f-2db6a0559b49-openshift-service-ca\") pod \"maas-api-e2e-derive-6205e423-54f664dff7-jltx5\" (UID: \"ac31ecfb-a438-42f6-805f-2db6a0559b49\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6205e423-54f664dff7-jltx5" Jul 14 22:40:39.400702 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:39.400668 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac31ecfb-a438-42f6-805f-2db6a0559b49-maas-api-tls\") pod \"maas-api-e2e-derive-6205e423-54f664dff7-jltx5\" (UID: \"ac31ecfb-a438-42f6-805f-2db6a0559b49\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6205e423-54f664dff7-jltx5" Jul 14 22:40:39.400770 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:39.400697 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z69k\" (UniqueName: \"kubernetes.io/projected/ac31ecfb-a438-42f6-805f-2db6a0559b49-kube-api-access-4z69k\") pod \"maas-api-e2e-derive-6205e423-54f664dff7-jltx5\" (UID: \"ac31ecfb-a438-42f6-805f-2db6a0559b49\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6205e423-54f664dff7-jltx5" Jul 14 22:40:39.434689 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:39.434660 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:40:39.501950 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:39.501886 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac31ecfb-a438-42f6-805f-2db6a0559b49-maas-api-tls\") pod \"maas-api-e2e-derive-6205e423-54f664dff7-jltx5\" (UID: \"ac31ecfb-a438-42f6-805f-2db6a0559b49\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6205e423-54f664dff7-jltx5" Jul 14 22:40:39.501950 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:39.501916 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z69k\" (UniqueName: \"kubernetes.io/projected/ac31ecfb-a438-42f6-805f-2db6a0559b49-kube-api-access-4z69k\") pod \"maas-api-e2e-derive-6205e423-54f664dff7-jltx5\" (UID: \"ac31ecfb-a438-42f6-805f-2db6a0559b49\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6205e423-54f664dff7-jltx5" Jul 14 22:40:39.502102 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:39.501955 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac31ecfb-a438-42f6-805f-2db6a0559b49-openshift-service-ca\") pod \"maas-api-e2e-derive-6205e423-54f664dff7-jltx5\" (UID: \"ac31ecfb-a438-42f6-805f-2db6a0559b49\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6205e423-54f664dff7-jltx5" Jul 14 22:40:39.502508 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:39.502491 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac31ecfb-a438-42f6-805f-2db6a0559b49-openshift-service-ca\") pod \"maas-api-e2e-derive-6205e423-54f664dff7-jltx5\" (UID: \"ac31ecfb-a438-42f6-805f-2db6a0559b49\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6205e423-54f664dff7-jltx5" Jul 14 22:40:39.504386 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:39.504366 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac31ecfb-a438-42f6-805f-2db6a0559b49-maas-api-tls\") pod \"maas-api-e2e-derive-6205e423-54f664dff7-jltx5\" (UID: \"ac31ecfb-a438-42f6-805f-2db6a0559b49\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6205e423-54f664dff7-jltx5" Jul 14 22:40:39.513760 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:39.513735 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z69k\" (UniqueName: \"kubernetes.io/projected/ac31ecfb-a438-42f6-805f-2db6a0559b49-kube-api-access-4z69k\") pod \"maas-api-e2e-derive-6205e423-54f664dff7-jltx5\" (UID: \"ac31ecfb-a438-42f6-805f-2db6a0559b49\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6205e423-54f664dff7-jltx5" Jul 14 22:40:39.668463 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:39.668418 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6205e423-54f664dff7-jltx5" Jul 14 22:40:39.710950 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:39.710907 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:40:39.808389 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:39.808364 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6205e423-54f664dff7-jltx5"] Jul 14 22:40:39.809850 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:40:39.809826 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac31ecfb_a438_42f6_805f_2db6a0559b49.slice/crio-2d25fd37a63e2ddc36705754cf392ef50d71121c431fe22f495b6d5cf240d900 WatchSource:0}: Error finding container 2d25fd37a63e2ddc36705754cf392ef50d71121c431fe22f495b6d5cf240d900: Status 404 returned error can't find the container with id 2d25fd37a63e2ddc36705754cf392ef50d71121c431fe22f495b6d5cf240d900 Jul 14 22:40:39.987544 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:39.987509 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6205e423-54f664dff7-jltx5" event={"ID":"ac31ecfb-a438-42f6-805f-2db6a0559b49","Type":"ContainerStarted","Data":"2d25fd37a63e2ddc36705754cf392ef50d71121c431fe22f495b6d5cf240d900"} Jul 14 22:40:40.411840 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:40.411808 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:40:40.523026 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:40.522994 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:40:40.994990 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:40.994956 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6205e423-54f664dff7-jltx5" event={"ID":"ac31ecfb-a438-42f6-805f-2db6a0559b49","Type":"ContainerStarted","Data":"faf1aa0c20b38075fd7b335c1a11dc793a53ddcc14bff7a62e157fd2d4825939"} Jul 14 22:40:40.995180 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:40.995071 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6205e423-54f664dff7-jltx5" Jul 14 22:40:41.027783 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:41.027735 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6205e423-54f664dff7-jltx5" podStartSLOduration=1.5381995640000001 podStartE2EDuration="2.027721411s" totalImagesPullingTime="489.521847ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:40:41.024659006 +0000 UTC m=+1871.383805473" watchObservedRunningTime="2026-07-14 22:40:41.027721411 +0000 UTC m=+1871.386867847" Jul 14 22:40:44.375718 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:44.375684 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:40:44.494837 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:44.494803 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:40:47.011923 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:47.011896 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6205e423-54f664dff7-jltx5" Jul 14 22:40:48.166088 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:40:48.166052 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:41:44.133577 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:41:44.133513 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6205e423-54f664dff7-jltx5"] Jul 14 22:41:44.134090 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:41:44.133821 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6205e423-54f664dff7-jltx5" podUID="ac31ecfb-a438-42f6-805f-2db6a0559b49" containerName="maas-api" containerID="cri-o://faf1aa0c20b38075fd7b335c1a11dc793a53ddcc14bff7a62e157fd2d4825939" gracePeriod=30 Jul 14 22:41:44.180910 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:41:44.180875 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:41:44.312058 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:41:44.311818 2550 generic.go:356] "Generic (PLEG): container finished" podID="ac31ecfb-a438-42f6-805f-2db6a0559b49" containerID="faf1aa0c20b38075fd7b335c1a11dc793a53ddcc14bff7a62e157fd2d4825939" exitCode=0 Jul 14 22:41:44.312058 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:41:44.311995 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6205e423-54f664dff7-jltx5" event={"ID":"ac31ecfb-a438-42f6-805f-2db6a0559b49","Type":"ContainerDied","Data":"faf1aa0c20b38075fd7b335c1a11dc793a53ddcc14bff7a62e157fd2d4825939"} Jul 14 22:41:44.381236 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:41:44.381216 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6205e423-54f664dff7-jltx5" Jul 14 22:41:44.478416 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:41:44.478387 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac31ecfb-a438-42f6-805f-2db6a0559b49-openshift-service-ca\") pod \"ac31ecfb-a438-42f6-805f-2db6a0559b49\" (UID: \"ac31ecfb-a438-42f6-805f-2db6a0559b49\") " Jul 14 22:41:44.478618 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:41:44.478482 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac31ecfb-a438-42f6-805f-2db6a0559b49-maas-api-tls\") pod \"ac31ecfb-a438-42f6-805f-2db6a0559b49\" (UID: \"ac31ecfb-a438-42f6-805f-2db6a0559b49\") " Jul 14 22:41:44.478618 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:41:44.478522 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4z69k\" (UniqueName: \"kubernetes.io/projected/ac31ecfb-a438-42f6-805f-2db6a0559b49-kube-api-access-4z69k\") pod \"ac31ecfb-a438-42f6-805f-2db6a0559b49\" (UID: \"ac31ecfb-a438-42f6-805f-2db6a0559b49\") " Jul 14 22:41:44.478829 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:41:44.478793 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac31ecfb-a438-42f6-805f-2db6a0559b49-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ac31ecfb-a438-42f6-805f-2db6a0559b49" (UID: "ac31ecfb-a438-42f6-805f-2db6a0559b49"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:41:44.480454 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:41:44.480431 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac31ecfb-a438-42f6-805f-2db6a0559b49-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ac31ecfb-a438-42f6-805f-2db6a0559b49" (UID: "ac31ecfb-a438-42f6-805f-2db6a0559b49"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 22:41:44.480565 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:41:44.480458 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac31ecfb-a438-42f6-805f-2db6a0559b49-kube-api-access-4z69k" (OuterVolumeSpecName: "kube-api-access-4z69k") pod "ac31ecfb-a438-42f6-805f-2db6a0559b49" (UID: "ac31ecfb-a438-42f6-805f-2db6a0559b49"). InnerVolumeSpecName "kube-api-access-4z69k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:41:44.579141 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:41:44.579117 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac31ecfb-a438-42f6-805f-2db6a0559b49-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:41:44.579141 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:41:44.579139 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z69k\" (UniqueName: \"kubernetes.io/projected/ac31ecfb-a438-42f6-805f-2db6a0559b49-kube-api-access-4z69k\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:41:44.579289 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:41:44.579148 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac31ecfb-a438-42f6-805f-2db6a0559b49-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:41:45.318728 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:41:45.318700 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6205e423-54f664dff7-jltx5" Jul 14 22:41:45.319140 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:41:45.318732 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6205e423-54f664dff7-jltx5" event={"ID":"ac31ecfb-a438-42f6-805f-2db6a0559b49","Type":"ContainerDied","Data":"2d25fd37a63e2ddc36705754cf392ef50d71121c431fe22f495b6d5cf240d900"} Jul 14 22:41:45.319140 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:41:45.318791 2550 scope.go:117] "RemoveContainer" containerID="faf1aa0c20b38075fd7b335c1a11dc793a53ddcc14bff7a62e157fd2d4825939" Jul 14 22:41:45.343876 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:41:45.343845 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6205e423-54f664dff7-jltx5"] Jul 14 22:41:45.349118 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:41:45.349095 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6205e423-54f664dff7-jltx5"] Jul 14 22:41:46.196816 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:41:46.196774 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac31ecfb-a438-42f6-805f-2db6a0559b49" path="/var/lib/kubelet/pods/ac31ecfb-a438-42f6-805f-2db6a0559b49/volumes" Jul 14 22:41:48.713327 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:41:48.713285 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:41:55.794190 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:41:55.794157 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:41:57.101345 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:41:57.101287 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:41:58.925939 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:41:58.925905 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:41:59.614410 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:41:59.614374 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:41:59.758851 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:41:59.758816 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:42:01.331702 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:42:01.331664 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:42:05.772562 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:42:05.772519 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:42:07.723475 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:42:07.723440 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:42:07.858071 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:42:07.858038 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:42:07.959786 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:42:07.959750 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:42:08.529209 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:42:08.529173 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:42:34.586767 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:42:34.586730 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:42:41.156785 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:42:41.156749 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:42:41.247981 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:42:41.247949 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:42:41.363846 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:42:41.363800 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:42:41.462703 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:42:41.462672 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:42:57.341599 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:42:57.341544 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:43:03.591647 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:03.591610 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:43:03.700996 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:03.700964 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:43:03.792847 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:03.792805 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:43:08.201765 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:08.201731 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:43:08.302668 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:08.302634 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:43:14.393964 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:14.393927 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:43:15.266008 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:15.265973 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:43:21.907054 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:21.907023 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:43:22.029478 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:22.029442 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:43:22.125622 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:22.125585 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:43:22.226821 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:22.226783 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:43:23.064681 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:23.064644 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:43:23.164379 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:23.164348 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:43:24.941022 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:24.940989 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:43:31.730745 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:31.730706 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:43:31.895838 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:31.895806 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:43:32.422459 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:32.422414 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:43:38.784239 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:38.784200 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:43:38.867661 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:38.867634 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:43:39.021379 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:39.021347 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:43:39.371377 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:39.371348 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:43:39.750074 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:39.750046 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:43:42.270901 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:42.270867 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:43:42.341079 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:42.341037 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-de2bcd69-f45f75947-d4l45"] Jul 14 22:43:42.341415 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:42.341371 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac31ecfb-a438-42f6-805f-2db6a0559b49" containerName="maas-api" Jul 14 22:43:42.341415 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:42.341386 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac31ecfb-a438-42f6-805f-2db6a0559b49" containerName="maas-api" Jul 14 22:43:42.341541 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:42.341439 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac31ecfb-a438-42f6-805f-2db6a0559b49" containerName="maas-api" Jul 14 22:43:42.344047 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:42.344029 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de2bcd69-f45f75947-d4l45" Jul 14 22:43:42.347062 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:42.347037 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-de2bcd69-serving-cert\"" Jul 14 22:43:42.356957 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:42.356935 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-de2bcd69-f45f75947-d4l45"] Jul 14 22:43:42.369057 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:42.369027 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:43:42.387378 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:42.387357 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74780896-8a8a-4ad4-af12-b60b49806ec6-openshift-service-ca\") pod \"maas-api-e2e-mt-de2bcd69-f45f75947-d4l45\" (UID: \"74780896-8a8a-4ad4-af12-b60b49806ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-de2bcd69-f45f75947-d4l45" Jul 14 22:43:42.387493 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:42.387402 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whpvz\" (UniqueName: \"kubernetes.io/projected/74780896-8a8a-4ad4-af12-b60b49806ec6-kube-api-access-whpvz\") pod \"maas-api-e2e-mt-de2bcd69-f45f75947-d4l45\" (UID: \"74780896-8a8a-4ad4-af12-b60b49806ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-de2bcd69-f45f75947-d4l45" Jul 14 22:43:42.387493 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:42.387438 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74780896-8a8a-4ad4-af12-b60b49806ec6-maas-api-tls\") pod \"maas-api-e2e-mt-de2bcd69-f45f75947-d4l45\" (UID: \"74780896-8a8a-4ad4-af12-b60b49806ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-de2bcd69-f45f75947-d4l45" Jul 14 22:43:42.458969 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:42.458935 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:43:42.488251 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:42.488222 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whpvz\" (UniqueName: \"kubernetes.io/projected/74780896-8a8a-4ad4-af12-b60b49806ec6-kube-api-access-whpvz\") pod \"maas-api-e2e-mt-de2bcd69-f45f75947-d4l45\" (UID: \"74780896-8a8a-4ad4-af12-b60b49806ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-de2bcd69-f45f75947-d4l45" Jul 14 22:43:42.488440 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:42.488266 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74780896-8a8a-4ad4-af12-b60b49806ec6-maas-api-tls\") pod \"maas-api-e2e-mt-de2bcd69-f45f75947-d4l45\" (UID: \"74780896-8a8a-4ad4-af12-b60b49806ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-de2bcd69-f45f75947-d4l45" Jul 14 22:43:42.488440 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:42.488325 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74780896-8a8a-4ad4-af12-b60b49806ec6-openshift-service-ca\") pod \"maas-api-e2e-mt-de2bcd69-f45f75947-d4l45\" (UID: \"74780896-8a8a-4ad4-af12-b60b49806ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-de2bcd69-f45f75947-d4l45" Jul 14 22:43:42.489121 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:42.489090 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74780896-8a8a-4ad4-af12-b60b49806ec6-openshift-service-ca\") pod \"maas-api-e2e-mt-de2bcd69-f45f75947-d4l45\" (UID: \"74780896-8a8a-4ad4-af12-b60b49806ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-de2bcd69-f45f75947-d4l45" Jul 14 22:43:42.491283 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:42.491262 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74780896-8a8a-4ad4-af12-b60b49806ec6-maas-api-tls\") pod \"maas-api-e2e-mt-de2bcd69-f45f75947-d4l45\" (UID: \"74780896-8a8a-4ad4-af12-b60b49806ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-de2bcd69-f45f75947-d4l45" Jul 14 22:43:42.498852 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:42.498830 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whpvz\" (UniqueName: \"kubernetes.io/projected/74780896-8a8a-4ad4-af12-b60b49806ec6-kube-api-access-whpvz\") pod \"maas-api-e2e-mt-de2bcd69-f45f75947-d4l45\" (UID: \"74780896-8a8a-4ad4-af12-b60b49806ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-de2bcd69-f45f75947-d4l45" Jul 14 22:43:42.653772 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:42.653691 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de2bcd69-f45f75947-d4l45" Jul 14 22:43:42.807583 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:42.807543 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-de2bcd69-f45f75947-d4l45"] Jul 14 22:43:42.809360 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:43:42.809319 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74780896_8a8a_4ad4_af12_b60b49806ec6.slice/crio-9f45ca4c4d6f769f2b87fd38668754996ce6773decedc115295bd5745153ebf6 WatchSource:0}: Error finding container 9f45ca4c4d6f769f2b87fd38668754996ce6773decedc115295bd5745153ebf6: Status 404 returned error can't find the container with id 9f45ca4c4d6f769f2b87fd38668754996ce6773decedc115295bd5745153ebf6 Jul 14 22:43:42.810871 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:42.810853 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 22:43:42.903260 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:42.903229 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de2bcd69-f45f75947-d4l45" event={"ID":"74780896-8a8a-4ad4-af12-b60b49806ec6","Type":"ContainerStarted","Data":"9f45ca4c4d6f769f2b87fd38668754996ce6773decedc115295bd5745153ebf6"} Jul 14 22:43:43.145894 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:43.145864 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:43:43.241470 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:43.241438 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:43:43.909513 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:43.909426 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de2bcd69-f45f75947-d4l45" event={"ID":"74780896-8a8a-4ad4-af12-b60b49806ec6","Type":"ContainerStarted","Data":"dfe9ab86291fa537124d8ee3d88ea7d8ce207ed95761391b9c5505d4dde75f55"} Jul 14 22:43:43.909513 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:43.909511 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de2bcd69-f45f75947-d4l45" Jul 14 22:43:43.934197 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:43.934132 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de2bcd69-f45f75947-d4l45" podStartSLOduration=1.210012238 podStartE2EDuration="1.934112755s" totalImagesPullingTime="724.100517ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:43:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:43:43.933403465 +0000 UTC m=+2054.292549902" watchObservedRunningTime="2026-07-14 22:43:43.934112755 +0000 UTC m=+2054.293259193" Jul 14 22:43:46.755401 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:46.755353 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:43:47.529344 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:47.529305 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:43:47.652296 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:47.652264 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:43:48.060818 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:48.060788 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:43:49.920714 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:49.920689 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de2bcd69-f45f75947-d4l45" Jul 14 22:43:54.258927 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:54.258834 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:43:54.368425 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:43:54.368397 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:44:05.356793 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:05.356760 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-de2bcd69-f45f75947-d4l45"] Jul 14 22:44:05.357236 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:05.356950 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de2bcd69-f45f75947-d4l45" podUID="74780896-8a8a-4ad4-af12-b60b49806ec6" containerName="maas-api" containerID="cri-o://dfe9ab86291fa537124d8ee3d88ea7d8ce207ed95761391b9c5505d4dde75f55" gracePeriod=30 Jul 14 22:44:05.406484 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:05.406454 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:44:05.604726 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:05.604699 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de2bcd69-f45f75947-d4l45" Jul 14 22:44:05.690306 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:05.690280 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whpvz\" (UniqueName: \"kubernetes.io/projected/74780896-8a8a-4ad4-af12-b60b49806ec6-kube-api-access-whpvz\") pod \"74780896-8a8a-4ad4-af12-b60b49806ec6\" (UID: \"74780896-8a8a-4ad4-af12-b60b49806ec6\") " Jul 14 22:44:05.690457 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:05.690337 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74780896-8a8a-4ad4-af12-b60b49806ec6-openshift-service-ca\") pod \"74780896-8a8a-4ad4-af12-b60b49806ec6\" (UID: \"74780896-8a8a-4ad4-af12-b60b49806ec6\") " Jul 14 22:44:05.690457 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:05.690364 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74780896-8a8a-4ad4-af12-b60b49806ec6-maas-api-tls\") pod \"74780896-8a8a-4ad4-af12-b60b49806ec6\" (UID: \"74780896-8a8a-4ad4-af12-b60b49806ec6\") " Jul 14 22:44:05.690733 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:05.690710 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74780896-8a8a-4ad4-af12-b60b49806ec6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "74780896-8a8a-4ad4-af12-b60b49806ec6" (UID: "74780896-8a8a-4ad4-af12-b60b49806ec6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:44:05.692437 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:05.692406 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74780896-8a8a-4ad4-af12-b60b49806ec6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "74780896-8a8a-4ad4-af12-b60b49806ec6" (UID: "74780896-8a8a-4ad4-af12-b60b49806ec6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 22:44:05.692579 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:05.692462 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74780896-8a8a-4ad4-af12-b60b49806ec6-kube-api-access-whpvz" (OuterVolumeSpecName: "kube-api-access-whpvz") pod "74780896-8a8a-4ad4-af12-b60b49806ec6" (UID: "74780896-8a8a-4ad4-af12-b60b49806ec6"). InnerVolumeSpecName "kube-api-access-whpvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:44:05.791753 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:05.791702 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74780896-8a8a-4ad4-af12-b60b49806ec6-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:44:05.791753 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:05.791747 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whpvz\" (UniqueName: \"kubernetes.io/projected/74780896-8a8a-4ad4-af12-b60b49806ec6-kube-api-access-whpvz\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:44:05.791753 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:05.791759 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74780896-8a8a-4ad4-af12-b60b49806ec6-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:44:06.017518 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:06.017439 2550 generic.go:356] "Generic (PLEG): container finished" podID="74780896-8a8a-4ad4-af12-b60b49806ec6" containerID="dfe9ab86291fa537124d8ee3d88ea7d8ce207ed95761391b9c5505d4dde75f55" exitCode=0 Jul 14 22:44:06.017518 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:06.017491 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de2bcd69-f45f75947-d4l45" Jul 14 22:44:06.017518 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:06.017507 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de2bcd69-f45f75947-d4l45" event={"ID":"74780896-8a8a-4ad4-af12-b60b49806ec6","Type":"ContainerDied","Data":"dfe9ab86291fa537124d8ee3d88ea7d8ce207ed95761391b9c5505d4dde75f55"} Jul 14 22:44:06.017730 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:06.017616 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de2bcd69-f45f75947-d4l45" event={"ID":"74780896-8a8a-4ad4-af12-b60b49806ec6","Type":"ContainerDied","Data":"9f45ca4c4d6f769f2b87fd38668754996ce6773decedc115295bd5745153ebf6"} Jul 14 22:44:06.017730 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:06.017647 2550 scope.go:117] "RemoveContainer" containerID="dfe9ab86291fa537124d8ee3d88ea7d8ce207ed95761391b9c5505d4dde75f55" Jul 14 22:44:06.026217 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:06.026196 2550 scope.go:117] "RemoveContainer" containerID="dfe9ab86291fa537124d8ee3d88ea7d8ce207ed95761391b9c5505d4dde75f55" Jul 14 22:44:06.026462 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:44:06.026444 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfe9ab86291fa537124d8ee3d88ea7d8ce207ed95761391b9c5505d4dde75f55\": container with ID starting with dfe9ab86291fa537124d8ee3d88ea7d8ce207ed95761391b9c5505d4dde75f55 not found: ID does not exist" containerID="dfe9ab86291fa537124d8ee3d88ea7d8ce207ed95761391b9c5505d4dde75f55" Jul 14 22:44:06.026529 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:06.026475 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfe9ab86291fa537124d8ee3d88ea7d8ce207ed95761391b9c5505d4dde75f55"} err="failed to get container status \"dfe9ab86291fa537124d8ee3d88ea7d8ce207ed95761391b9c5505d4dde75f55\": rpc error: code = NotFound desc = could not find container \"dfe9ab86291fa537124d8ee3d88ea7d8ce207ed95761391b9c5505d4dde75f55\": container with ID starting with dfe9ab86291fa537124d8ee3d88ea7d8ce207ed95761391b9c5505d4dde75f55 not found: ID does not exist" Jul 14 22:44:06.042921 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:06.042897 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-de2bcd69-f45f75947-d4l45"] Jul 14 22:44:06.048086 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:06.048066 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-de2bcd69-f45f75947-d4l45"] Jul 14 22:44:06.195295 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:06.195260 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74780896-8a8a-4ad4-af12-b60b49806ec6" path="/var/lib/kubelet/pods/74780896-8a8a-4ad4-af12-b60b49806ec6/volumes" Jul 14 22:44:09.788604 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:09.788576 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:44:16.010833 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:16.010800 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:44:16.147189 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:16.147153 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:44:20.630383 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:20.630337 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:44:20.828257 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:20.828228 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:44:20.968461 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:20.968424 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:44:21.056860 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:21.056827 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:44:21.204531 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:21.204492 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:44:27.449367 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:27.449315 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:44:27.522193 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:27.522157 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:44:27.625234 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:27.625192 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:44:27.835873 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:27.835792 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:44:30.184259 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:30.184234 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zj4sb_0884baa5-9206-4196-a7c9-b5686623311b/ovn-acl-logging/0.log" Jul 14 22:44:30.189425 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:30.189404 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zj4sb_0884baa5-9206-4196-a7c9-b5686623311b/ovn-acl-logging/0.log" Jul 14 22:44:34.448216 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:34.448171 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:44:34.540218 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:34.540177 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:44:37.183762 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:37.183724 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:44:41.384849 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:41.384810 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:44:41.774707 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:41.774671 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:44:42.243386 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:42.243355 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:44:44.318684 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:44.318652 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:44:44.436458 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:44.436424 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7a211570-77d454fb46-d25fd"] Jul 14 22:44:44.436776 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:44.436764 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74780896-8a8a-4ad4-af12-b60b49806ec6" containerName="maas-api" Jul 14 22:44:44.436825 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:44.436779 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="74780896-8a8a-4ad4-af12-b60b49806ec6" containerName="maas-api" Jul 14 22:44:44.436861 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:44.436837 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="74780896-8a8a-4ad4-af12-b60b49806ec6" containerName="maas-api" Jul 14 22:44:44.439534 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:44.439517 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a211570-77d454fb46-d25fd" Jul 14 22:44:44.442676 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:44.442653 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7a211570-serving-cert\"" Jul 14 22:44:44.481706 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:44.481683 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7a211570-77d454fb46-d25fd"] Jul 14 22:44:44.482977 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:44.482955 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:44:44.525675 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:44.525653 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea7f300-7721-41ef-a5d2-cef75b89aef5-openshift-service-ca\") pod \"maas-api-e2e-mt-7a211570-77d454fb46-d25fd\" (UID: \"4ea7f300-7721-41ef-a5d2-cef75b89aef5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a211570-77d454fb46-d25fd" Jul 14 22:44:44.525793 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:44.525703 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ea7f300-7721-41ef-a5d2-cef75b89aef5-maas-api-tls\") pod \"maas-api-e2e-mt-7a211570-77d454fb46-d25fd\" (UID: \"4ea7f300-7721-41ef-a5d2-cef75b89aef5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a211570-77d454fb46-d25fd" Jul 14 22:44:44.525793 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:44.525723 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhpmf\" (UniqueName: \"kubernetes.io/projected/4ea7f300-7721-41ef-a5d2-cef75b89aef5-kube-api-access-mhpmf\") pod \"maas-api-e2e-mt-7a211570-77d454fb46-d25fd\" (UID: \"4ea7f300-7721-41ef-a5d2-cef75b89aef5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a211570-77d454fb46-d25fd" Jul 14 22:44:44.626831 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:44.626753 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ea7f300-7721-41ef-a5d2-cef75b89aef5-maas-api-tls\") pod \"maas-api-e2e-mt-7a211570-77d454fb46-d25fd\" (UID: \"4ea7f300-7721-41ef-a5d2-cef75b89aef5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a211570-77d454fb46-d25fd" Jul 14 22:44:44.626831 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:44.626790 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhpmf\" (UniqueName: \"kubernetes.io/projected/4ea7f300-7721-41ef-a5d2-cef75b89aef5-kube-api-access-mhpmf\") pod \"maas-api-e2e-mt-7a211570-77d454fb46-d25fd\" (UID: \"4ea7f300-7721-41ef-a5d2-cef75b89aef5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a211570-77d454fb46-d25fd" Jul 14 22:44:44.627006 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:44.626843 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea7f300-7721-41ef-a5d2-cef75b89aef5-openshift-service-ca\") pod \"maas-api-e2e-mt-7a211570-77d454fb46-d25fd\" (UID: \"4ea7f300-7721-41ef-a5d2-cef75b89aef5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a211570-77d454fb46-d25fd" Jul 14 22:44:44.627511 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:44.627488 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea7f300-7721-41ef-a5d2-cef75b89aef5-openshift-service-ca\") pod \"maas-api-e2e-mt-7a211570-77d454fb46-d25fd\" (UID: \"4ea7f300-7721-41ef-a5d2-cef75b89aef5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a211570-77d454fb46-d25fd" Jul 14 22:44:44.629100 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:44.629080 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ea7f300-7721-41ef-a5d2-cef75b89aef5-maas-api-tls\") pod \"maas-api-e2e-mt-7a211570-77d454fb46-d25fd\" (UID: \"4ea7f300-7721-41ef-a5d2-cef75b89aef5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a211570-77d454fb46-d25fd" Jul 14 22:44:44.646401 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:44.646371 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhpmf\" (UniqueName: \"kubernetes.io/projected/4ea7f300-7721-41ef-a5d2-cef75b89aef5-kube-api-access-mhpmf\") pod \"maas-api-e2e-mt-7a211570-77d454fb46-d25fd\" (UID: \"4ea7f300-7721-41ef-a5d2-cef75b89aef5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a211570-77d454fb46-d25fd" Jul 14 22:44:44.648669 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:44.648636 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:44:44.745096 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:44.745070 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:44:44.749286 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:44.749266 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a211570-77d454fb46-d25fd" Jul 14 22:44:44.883933 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:44.883907 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7a211570-77d454fb46-d25fd"] Jul 14 22:44:44.885854 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:44:44.885829 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea7f300_7721_41ef_a5d2_cef75b89aef5.slice/crio-c1917047db7ae7ceec042a397f98e43331271a73a61611feb3a8e82cf8867896 WatchSource:0}: Error finding container c1917047db7ae7ceec042a397f98e43331271a73a61611feb3a8e82cf8867896: Status 404 returned error can't find the container with id c1917047db7ae7ceec042a397f98e43331271a73a61611feb3a8e82cf8867896 Jul 14 22:44:45.111863 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:45.111832 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:44:45.208273 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:45.208244 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a211570-77d454fb46-d25fd" event={"ID":"4ea7f300-7721-41ef-a5d2-cef75b89aef5","Type":"ContainerStarted","Data":"c1917047db7ae7ceec042a397f98e43331271a73a61611feb3a8e82cf8867896"} Jul 14 22:44:45.211891 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:45.211870 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:44:46.213994 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:46.213962 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a211570-77d454fb46-d25fd" event={"ID":"4ea7f300-7721-41ef-a5d2-cef75b89aef5","Type":"ContainerStarted","Data":"0ddd633b2896bc3e9c2f69465c64f64467449fb7003b7319e868253d3a645bfa"} Jul 14 22:44:46.214420 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:46.214016 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a211570-77d454fb46-d25fd" Jul 14 22:44:46.239580 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:46.239521 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a211570-77d454fb46-d25fd" podStartSLOduration=1.42629362 podStartE2EDuration="2.239506847s" totalImagesPullingTime="813.213227ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:44:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:44:46.236699626 +0000 UTC m=+2116.595846064" watchObservedRunningTime="2026-07-14 22:44:46.239506847 +0000 UTC m=+2116.598653306" Jul 14 22:44:48.864470 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:48.864437 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:44:52.223574 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:44:52.223522 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a211570-77d454fb46-d25fd" Jul 14 22:45:01.146988 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:45:01.146953 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:45:01.213566 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:45:01.213524 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:45:09.825409 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:45:09.825375 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:45:21.384468 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:45:21.384385 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:45:21.487979 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:45:21.487944 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:45:38.266886 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:45:38.266854 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:45:38.369002 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:45:38.368955 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:45:48.255572 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:45:48.255520 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7a211570-77d454fb46-d25fd"] Jul 14 22:45:48.256018 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:45:48.255780 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a211570-77d454fb46-d25fd" podUID="4ea7f300-7721-41ef-a5d2-cef75b89aef5" containerName="maas-api" containerID="cri-o://0ddd633b2896bc3e9c2f69465c64f64467449fb7003b7319e868253d3a645bfa" gracePeriod=30 Jul 14 22:45:48.294773 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:45:48.294739 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:45:48.491725 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:45:48.491703 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a211570-77d454fb46-d25fd" Jul 14 22:45:48.517649 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:45:48.517582 2550 generic.go:356] "Generic (PLEG): container finished" podID="4ea7f300-7721-41ef-a5d2-cef75b89aef5" containerID="0ddd633b2896bc3e9c2f69465c64f64467449fb7003b7319e868253d3a645bfa" exitCode=0 Jul 14 22:45:48.517649 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:45:48.517619 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a211570-77d454fb46-d25fd" Jul 14 22:45:48.517649 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:45:48.517637 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a211570-77d454fb46-d25fd" event={"ID":"4ea7f300-7721-41ef-a5d2-cef75b89aef5","Type":"ContainerDied","Data":"0ddd633b2896bc3e9c2f69465c64f64467449fb7003b7319e868253d3a645bfa"} Jul 14 22:45:48.517836 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:45:48.517694 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a211570-77d454fb46-d25fd" event={"ID":"4ea7f300-7721-41ef-a5d2-cef75b89aef5","Type":"ContainerDied","Data":"c1917047db7ae7ceec042a397f98e43331271a73a61611feb3a8e82cf8867896"} Jul 14 22:45:48.517836 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:45:48.517724 2550 scope.go:117] "RemoveContainer" containerID="0ddd633b2896bc3e9c2f69465c64f64467449fb7003b7319e868253d3a645bfa" Jul 14 22:45:48.526671 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:45:48.526649 2550 scope.go:117] "RemoveContainer" containerID="0ddd633b2896bc3e9c2f69465c64f64467449fb7003b7319e868253d3a645bfa" Jul 14 22:45:48.526910 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:45:48.526887 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ddd633b2896bc3e9c2f69465c64f64467449fb7003b7319e868253d3a645bfa\": container with ID starting with 0ddd633b2896bc3e9c2f69465c64f64467449fb7003b7319e868253d3a645bfa not found: ID does not exist" containerID="0ddd633b2896bc3e9c2f69465c64f64467449fb7003b7319e868253d3a645bfa" Jul 14 22:45:48.526990 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:45:48.526922 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ddd633b2896bc3e9c2f69465c64f64467449fb7003b7319e868253d3a645bfa"} err="failed to get container status \"0ddd633b2896bc3e9c2f69465c64f64467449fb7003b7319e868253d3a645bfa\": rpc error: code = NotFound desc = could not find container \"0ddd633b2896bc3e9c2f69465c64f64467449fb7003b7319e868253d3a645bfa\": container with ID starting with 0ddd633b2896bc3e9c2f69465c64f64467449fb7003b7319e868253d3a645bfa not found: ID does not exist" Jul 14 22:45:48.580102 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:45:48.580074 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ea7f300-7721-41ef-a5d2-cef75b89aef5-maas-api-tls\") pod \"4ea7f300-7721-41ef-a5d2-cef75b89aef5\" (UID: \"4ea7f300-7721-41ef-a5d2-cef75b89aef5\") " Jul 14 22:45:48.580275 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:45:48.580150 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhpmf\" (UniqueName: \"kubernetes.io/projected/4ea7f300-7721-41ef-a5d2-cef75b89aef5-kube-api-access-mhpmf\") pod \"4ea7f300-7721-41ef-a5d2-cef75b89aef5\" (UID: \"4ea7f300-7721-41ef-a5d2-cef75b89aef5\") " Jul 14 22:45:48.580275 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:45:48.580229 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea7f300-7721-41ef-a5d2-cef75b89aef5-openshift-service-ca\") pod \"4ea7f300-7721-41ef-a5d2-cef75b89aef5\" (UID: \"4ea7f300-7721-41ef-a5d2-cef75b89aef5\") " Jul 14 22:45:48.580567 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:45:48.580531 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ea7f300-7721-41ef-a5d2-cef75b89aef5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4ea7f300-7721-41ef-a5d2-cef75b89aef5" (UID: "4ea7f300-7721-41ef-a5d2-cef75b89aef5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:45:48.582188 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:45:48.582168 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea7f300-7721-41ef-a5d2-cef75b89aef5-kube-api-access-mhpmf" (OuterVolumeSpecName: "kube-api-access-mhpmf") pod "4ea7f300-7721-41ef-a5d2-cef75b89aef5" (UID: "4ea7f300-7721-41ef-a5d2-cef75b89aef5"). InnerVolumeSpecName "kube-api-access-mhpmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:45:48.582276 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:45:48.582250 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ea7f300-7721-41ef-a5d2-cef75b89aef5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4ea7f300-7721-41ef-a5d2-cef75b89aef5" (UID: "4ea7f300-7721-41ef-a5d2-cef75b89aef5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 22:45:48.681183 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:45:48.681148 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhpmf\" (UniqueName: \"kubernetes.io/projected/4ea7f300-7721-41ef-a5d2-cef75b89aef5-kube-api-access-mhpmf\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:45:48.681183 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:45:48.681178 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea7f300-7721-41ef-a5d2-cef75b89aef5-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:45:48.681183 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:45:48.681191 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ea7f300-7721-41ef-a5d2-cef75b89aef5-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:45:48.838959 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:45:48.838930 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7a211570-77d454fb46-d25fd"] Jul 14 22:45:48.843014 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:45:48.842990 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7a211570-77d454fb46-d25fd"] Jul 14 22:45:50.194597 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:45:50.194545 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ea7f300-7721-41ef-a5d2-cef75b89aef5" path="/var/lib/kubelet/pods/4ea7f300-7721-41ef-a5d2-cef75b89aef5/volumes" Jul 14 22:45:52.822588 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:45:52.822502 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:45:57.851217 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:45:57.851179 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:46:20.083907 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:46:20.083870 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:46:20.140940 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:46:20.140909 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:46:34.712130 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:46:34.712091 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:46:35.741579 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:46:35.741525 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:46:41.987468 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:46:41.987436 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:46:42.088073 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:46:42.088035 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:46:45.193002 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:46:45.192974 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:46:45.440187 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:46:45.440157 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:46:46.585762 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:46:46.585724 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:46:46.719153 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:46:46.719119 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:46:47.495842 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:46:47.495802 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:46:47.596277 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:46:47.596242 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:46:47.659360 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:46:47.659331 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:46:49.130410 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:46:49.130375 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:46:55.432311 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:46:55.432280 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:46:55.504508 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:46:55.504467 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:46:56.148241 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:46:56.148210 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:47:03.176110 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:03.176077 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:47:04.419985 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:04.419936 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:47:13.968405 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:13.968365 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:47:14.324844 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:14.324750 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:47:35.450109 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:35.450073 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:47:41.463852 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:41.463812 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:47:41.527502 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:41.527469 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:47:41.844714 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:41.844636 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:47:42.236690 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:42.236659 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:47:42.458459 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:42.458424 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:47:45.276141 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:45.276096 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:47:45.380730 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:45.380692 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw"] Jul 14 22:47:45.381191 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:45.381077 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ea7f300-7721-41ef-a5d2-cef75b89aef5" containerName="maas-api" Jul 14 22:47:45.381191 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:45.381096 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ea7f300-7721-41ef-a5d2-cef75b89aef5" containerName="maas-api" Jul 14 22:47:45.381326 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:45.381196 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ea7f300-7721-41ef-a5d2-cef75b89aef5" containerName="maas-api" Jul 14 22:47:45.381904 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:45.381875 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:47:45.384565 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:45.384530 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw" Jul 14 22:47:45.387438 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:45.387414 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-fcf58c-serving-cert\"" Jul 14 22:47:45.400099 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:45.400072 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw"] Jul 14 22:47:45.463274 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:45.463236 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw\" (UID: \"1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw" Jul 14 22:47:45.463473 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:45.463308 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c-maas-api-tls\") pod \"maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw\" (UID: \"1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw" Jul 14 22:47:45.463473 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:45.463355 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-728zv\" (UniqueName: \"kubernetes.io/projected/1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c-kube-api-access-728zv\") pod \"maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw\" (UID: \"1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw" Jul 14 22:47:45.472149 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:45.472120 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:47:45.564324 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:45.564246 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw\" (UID: \"1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw" Jul 14 22:47:45.564488 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:45.564325 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c-maas-api-tls\") pod \"maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw\" (UID: \"1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw" Jul 14 22:47:45.564488 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:45.564364 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-728zv\" (UniqueName: \"kubernetes.io/projected/1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c-kube-api-access-728zv\") pod \"maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw\" (UID: \"1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw" Jul 14 22:47:45.564488 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:47:45.564441 2550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-fcf58c-serving-cert: secret "maas-api-e2e-inf-a-fcf58c-serving-cert" not found Jul 14 22:47:45.564713 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:47:45.564510 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c-maas-api-tls podName:1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c nodeName:}" failed. No retries permitted until 2026-07-14 22:47:46.064494569 +0000 UTC m=+2296.423640989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c-maas-api-tls") pod "maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw" (UID: "1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c") : secret "maas-api-e2e-inf-a-fcf58c-serving-cert" not found Jul 14 22:47:45.564984 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:45.564961 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw\" (UID: \"1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw" Jul 14 22:47:45.574828 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:45.574791 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-728zv\" (UniqueName: \"kubernetes.io/projected/1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c-kube-api-access-728zv\") pod \"maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw\" (UID: \"1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw" Jul 14 22:47:45.852620 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:45.852532 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:47:46.014816 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:46.014785 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:47:46.068745 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:46.068709 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c-maas-api-tls\") pod \"maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw\" (UID: \"1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw" Jul 14 22:47:46.071225 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:46.071201 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c-maas-api-tls\") pod \"maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw\" (UID: \"1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw" Jul 14 22:47:46.114897 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:46.114810 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:47:46.296960 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:46.296926 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw" Jul 14 22:47:46.427997 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:46.427969 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw"] Jul 14 22:47:46.429298 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:47:46.429268 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d02c9d6_cfba_4a5a_99e3_2f2904c07e1c.slice/crio-18576d3b6ccf372895b9c821f2e1635df299598e573e6d732304d1c1386cfe8d WatchSource:0}: Error finding container 18576d3b6ccf372895b9c821f2e1635df299598e573e6d732304d1c1386cfe8d: Status 404 returned error can't find the container with id 18576d3b6ccf372895b9c821f2e1635df299598e573e6d732304d1c1386cfe8d Jul 14 22:47:47.110126 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:47.110085 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw" event={"ID":"1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c","Type":"ContainerStarted","Data":"93729db34414a37fed280e0f061a394533dc33121899e27894d984cf4aa632f6"} Jul 14 22:47:47.110282 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:47.110145 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw" event={"ID":"1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c","Type":"ContainerStarted","Data":"18576d3b6ccf372895b9c821f2e1635df299598e573e6d732304d1c1386cfe8d"} Jul 14 22:47:47.110282 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:47.110265 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw" Jul 14 22:47:47.132913 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:47.132867 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw" podStartSLOduration=1.626817516 podStartE2EDuration="2.13285286s" totalImagesPullingTime="506.035344ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:47:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:47:47.131179786 +0000 UTC m=+2297.490326224" watchObservedRunningTime="2026-07-14 22:47:47.13285286 +0000 UTC m=+2297.491999357" Jul 14 22:47:49.840088 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:49.840045 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:47:50.162627 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:50.162590 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:47:50.502887 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:50.502808 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:47:50.677593 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:50.677545 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:47:53.120329 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:53.120302 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw" Jul 14 22:47:53.460685 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:53.460644 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:47:53.562830 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:53.562794 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd"] Jul 14 22:47:53.566170 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:53.566150 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd" Jul 14 22:47:53.569774 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:53.569752 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-cb7974-serving-cert\"" Jul 14 22:47:53.573406 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:53.573383 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:47:53.579813 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:53.579792 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd"] Jul 14 22:47:53.690269 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:53.690227 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:47:53.738490 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:53.738414 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87mq4\" (UniqueName: \"kubernetes.io/projected/d0793a99-ef68-478e-86c3-63e6ea6d7bec-kube-api-access-87mq4\") pod \"maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd\" (UID: \"d0793a99-ef68-478e-86c3-63e6ea6d7bec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd" Jul 14 22:47:53.738490 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:53.738461 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0793a99-ef68-478e-86c3-63e6ea6d7bec-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd\" (UID: \"d0793a99-ef68-478e-86c3-63e6ea6d7bec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd" Jul 14 22:47:53.738705 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:53.738525 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0793a99-ef68-478e-86c3-63e6ea6d7bec-maas-api-tls\") pod \"maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd\" (UID: \"d0793a99-ef68-478e-86c3-63e6ea6d7bec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd" Jul 14 22:47:53.785966 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:53.785940 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:47:53.839566 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:53.839518 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87mq4\" (UniqueName: \"kubernetes.io/projected/d0793a99-ef68-478e-86c3-63e6ea6d7bec-kube-api-access-87mq4\") pod \"maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd\" (UID: \"d0793a99-ef68-478e-86c3-63e6ea6d7bec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd" Jul 14 22:47:53.839766 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:53.839605 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0793a99-ef68-478e-86c3-63e6ea6d7bec-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd\" (UID: \"d0793a99-ef68-478e-86c3-63e6ea6d7bec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd" Jul 14 22:47:53.839766 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:53.839644 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0793a99-ef68-478e-86c3-63e6ea6d7bec-maas-api-tls\") pod \"maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd\" (UID: \"d0793a99-ef68-478e-86c3-63e6ea6d7bec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd" Jul 14 22:47:53.840347 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:53.840323 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0793a99-ef68-478e-86c3-63e6ea6d7bec-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd\" (UID: \"d0793a99-ef68-478e-86c3-63e6ea6d7bec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd" Jul 14 22:47:53.842381 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:53.842355 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0793a99-ef68-478e-86c3-63e6ea6d7bec-maas-api-tls\") pod \"maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd\" (UID: \"d0793a99-ef68-478e-86c3-63e6ea6d7bec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd" Jul 14 22:47:53.848989 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:53.848964 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87mq4\" (UniqueName: \"kubernetes.io/projected/d0793a99-ef68-478e-86c3-63e6ea6d7bec-kube-api-access-87mq4\") pod \"maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd\" (UID: \"d0793a99-ef68-478e-86c3-63e6ea6d7bec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd" Jul 14 22:47:53.876719 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:53.876687 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd" Jul 14 22:47:54.013175 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:54.013150 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd"] Jul 14 22:47:54.013938 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:47:54.013917 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0793a99_ef68_478e_86c3_63e6ea6d7bec.slice/crio-007790118aca738d312a5e6f11212652617c6affbe85fff7ca7f7fc86e093390 WatchSource:0}: Error finding container 007790118aca738d312a5e6f11212652617c6affbe85fff7ca7f7fc86e093390: Status 404 returned error can't find the container with id 007790118aca738d312a5e6f11212652617c6affbe85fff7ca7f7fc86e093390 Jul 14 22:47:54.145090 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:54.145048 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd" event={"ID":"d0793a99-ef68-478e-86c3-63e6ea6d7bec","Type":"ContainerStarted","Data":"007790118aca738d312a5e6f11212652617c6affbe85fff7ca7f7fc86e093390"} Jul 14 22:47:54.352875 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:54.352790 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:47:54.453863 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:54.453835 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:47:55.150824 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:55.150788 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd" event={"ID":"d0793a99-ef68-478e-86c3-63e6ea6d7bec","Type":"ContainerStarted","Data":"3d671ba67dcef5a86e58451c0bec9fdda75905c31d87c081c8ca8ec9a3defa8a"} Jul 14 22:47:55.151299 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:55.150936 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd" Jul 14 22:47:55.168926 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:55.168877 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd" podStartSLOduration=1.813272375 podStartE2EDuration="2.168862267s" totalImagesPullingTime="355.589892ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:47:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:47:55.168080609 +0000 UTC m=+2305.527227047" watchObservedRunningTime="2026-07-14 22:47:55.168862267 +0000 UTC m=+2305.528008704" Jul 14 22:47:57.887784 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:47:57.887745 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:48:01.163186 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:48:01.163158 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd" Jul 14 22:48:06.906003 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:48:06.905942 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:48:07.007724 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:48:07.007688 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:48:14.184397 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:48:14.184359 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:48:15.294667 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:48:15.294630 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:48:15.397919 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:48:15.397890 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:48:40.520417 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:48:40.520331 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:49:12.044927 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:12.044895 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:49:12.940739 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:12.940711 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:49:13.746885 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:13.746853 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:49:13.836782 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:13.836742 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:49:23.662062 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:23.662026 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw"] Jul 14 22:49:23.662535 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:23.662226 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw" podUID="1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c" containerName="maas-api" containerID="cri-o://93729db34414a37fed280e0f061a394533dc33121899e27894d984cf4aa632f6" gracePeriod=30 Jul 14 22:49:23.685744 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:23.685717 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:49:23.908177 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:23.908154 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw" Jul 14 22:49:24.016133 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:24.016039 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c-openshift-service-ca\") pod \"1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c\" (UID: \"1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c\") " Jul 14 22:49:24.016133 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:24.016092 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-728zv\" (UniqueName: \"kubernetes.io/projected/1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c-kube-api-access-728zv\") pod \"1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c\" (UID: \"1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c\") " Jul 14 22:49:24.016133 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:24.016122 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c-maas-api-tls\") pod \"1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c\" (UID: \"1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c\") " Jul 14 22:49:24.016427 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:24.016406 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c" (UID: "1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:49:24.018256 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:24.018223 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c-kube-api-access-728zv" (OuterVolumeSpecName: "kube-api-access-728zv") pod "1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c" (UID: "1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c"). InnerVolumeSpecName "kube-api-access-728zv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:49:24.018256 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:24.018227 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c" (UID: "1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 22:49:24.117621 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:24.117585 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-728zv\" (UniqueName: \"kubernetes.io/projected/1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c-kube-api-access-728zv\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:49:24.117621 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:24.117615 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:49:24.117621 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:24.117625 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:49:24.597400 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:24.597364 2550 generic.go:356] "Generic (PLEG): container finished" podID="1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c" containerID="93729db34414a37fed280e0f061a394533dc33121899e27894d984cf4aa632f6" exitCode=0 Jul 14 22:49:24.597594 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:24.597421 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw" Jul 14 22:49:24.597594 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:24.597453 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw" event={"ID":"1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c","Type":"ContainerDied","Data":"93729db34414a37fed280e0f061a394533dc33121899e27894d984cf4aa632f6"} Jul 14 22:49:24.597594 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:24.597504 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw" event={"ID":"1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c","Type":"ContainerDied","Data":"18576d3b6ccf372895b9c821f2e1635df299598e573e6d732304d1c1386cfe8d"} Jul 14 22:49:24.597594 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:24.597525 2550 scope.go:117] "RemoveContainer" containerID="93729db34414a37fed280e0f061a394533dc33121899e27894d984cf4aa632f6" Jul 14 22:49:24.605960 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:24.605940 2550 scope.go:117] "RemoveContainer" containerID="93729db34414a37fed280e0f061a394533dc33121899e27894d984cf4aa632f6" Jul 14 22:49:24.606227 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:49:24.606209 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93729db34414a37fed280e0f061a394533dc33121899e27894d984cf4aa632f6\": container with ID starting with 93729db34414a37fed280e0f061a394533dc33121899e27894d984cf4aa632f6 not found: ID does not exist" containerID="93729db34414a37fed280e0f061a394533dc33121899e27894d984cf4aa632f6" Jul 14 22:49:24.606276 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:24.606244 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93729db34414a37fed280e0f061a394533dc33121899e27894d984cf4aa632f6"} err="failed to get container status \"93729db34414a37fed280e0f061a394533dc33121899e27894d984cf4aa632f6\": rpc error: code = NotFound desc = could not find container \"93729db34414a37fed280e0f061a394533dc33121899e27894d984cf4aa632f6\": container with ID starting with 93729db34414a37fed280e0f061a394533dc33121899e27894d984cf4aa632f6 not found: ID does not exist" Jul 14 22:49:24.616637 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:24.616615 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw"] Jul 14 22:49:24.623121 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:24.623102 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fcf58c-7dfd5d8569-jdmlw"] Jul 14 22:49:26.195597 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:26.195546 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c" path="/var/lib/kubelet/pods/1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c/volumes" Jul 14 22:49:28.175973 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:28.175933 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:49:30.215878 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:30.215853 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zj4sb_0884baa5-9206-4196-a7c9-b5686623311b/ovn-acl-logging/0.log" Jul 14 22:49:30.223097 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:30.223078 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zj4sb_0884baa5-9206-4196-a7c9-b5686623311b/ovn-acl-logging/0.log" Jul 14 22:49:33.873410 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:33.873369 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:49:34.629063 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:34.629024 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:49:34.730938 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:34.730898 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:49:38.909297 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:38.909255 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:49:39.143114 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:39.143081 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:49:39.278267 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:39.278230 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cb7974-tv2d8"] Jul 14 22:49:39.278643 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:39.278627 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c" containerName="maas-api" Jul 14 22:49:39.278736 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:39.278645 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c" containerName="maas-api" Jul 14 22:49:39.278792 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:39.278756 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d02c9d6-cfba-4a5a-99e3-2f2904c07e1c" containerName="maas-api" Jul 14 22:49:39.281489 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:39.281469 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cb7974-tv2d8" Jul 14 22:49:39.284233 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:39.284217 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mj4xg\"" Jul 14 22:49:39.290324 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:39.290302 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cb7974-tv2d8"] Jul 14 22:49:39.335730 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:39.335697 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5733856-6a7b-4ccf-9289-fdd9e2497bdb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cb7974-tv2d8\" (UID: \"b5733856-6a7b-4ccf-9289-fdd9e2497bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cb7974-tv2d8" Jul 14 22:49:39.436899 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:39.436859 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5733856-6a7b-4ccf-9289-fdd9e2497bdb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cb7974-tv2d8\" (UID: \"b5733856-6a7b-4ccf-9289-fdd9e2497bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cb7974-tv2d8" Jul 14 22:49:39.437487 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:39.437465 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5733856-6a7b-4ccf-9289-fdd9e2497bdb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cb7974-tv2d8\" (UID: \"b5733856-6a7b-4ccf-9289-fdd9e2497bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cb7974-tv2d8" Jul 14 22:49:39.593464 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:39.593363 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cb7974-tv2d8" Jul 14 22:49:39.724175 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:39.724152 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cb7974-tv2d8"] Jul 14 22:49:39.725806 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:49:39.725781 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5733856_6a7b_4ccf_9289_fdd9e2497bdb.slice/crio-8beecc5f4ecdbce8272b5064209c588b63d122a77f760f224385212bb580fa7a WatchSource:0}: Error finding container 8beecc5f4ecdbce8272b5064209c588b63d122a77f760f224385212bb580fa7a: Status 404 returned error can't find the container with id 8beecc5f4ecdbce8272b5064209c588b63d122a77f760f224385212bb580fa7a Jul 14 22:49:39.727591 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:39.727572 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 22:49:40.142414 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:40.142365 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:49:40.241896 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:40.241868 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:49:40.683629 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:40.683591 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cb7974-tv2d8" event={"ID":"b5733856-6a7b-4ccf-9289-fdd9e2497bdb","Type":"ContainerStarted","Data":"8beecc5f4ecdbce8272b5064209c588b63d122a77f760f224385212bb580fa7a"} Jul 14 22:49:41.020296 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:41.020215 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:49:42.701977 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:42.701936 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cb7974-tv2d8" event={"ID":"b5733856-6a7b-4ccf-9289-fdd9e2497bdb","Type":"ContainerStarted","Data":"828c055ca55f0614649bb467e1f1bdd3b2467ce930624191d40dfa389b3850d4"} Jul 14 22:49:42.720266 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:42.720221 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cb7974-tv2d8" podStartSLOduration=1.399189972 podStartE2EDuration="3.720206894s" totalImagesPullingTime="2.321016922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:49:42.718987056 +0000 UTC m=+2413.078133491" watchObservedRunningTime="2026-07-14 22:49:42.720206894 +0000 UTC m=+2413.079353331" Jul 14 22:49:43.707789 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:43.707753 2550 generic.go:356] "Generic (PLEG): container finished" podID="b5733856-6a7b-4ccf-9289-fdd9e2497bdb" containerID="828c055ca55f0614649bb467e1f1bdd3b2467ce930624191d40dfa389b3850d4" exitCode=0 Jul 14 22:49:43.708259 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:43.707862 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cb7974-tv2d8" event={"ID":"b5733856-6a7b-4ccf-9289-fdd9e2497bdb","Type":"ContainerDied","Data":"828c055ca55f0614649bb467e1f1bdd3b2467ce930624191d40dfa389b3850d4"} Jul 14 22:49:44.837898 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:44.837871 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cb7974-tv2d8" Jul 14 22:49:44.874041 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:44.874017 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5733856-6a7b-4ccf-9289-fdd9e2497bdb-maas-api-service-ca\") pod \"b5733856-6a7b-4ccf-9289-fdd9e2497bdb\" (UID: \"b5733856-6a7b-4ccf-9289-fdd9e2497bdb\") " Jul 14 22:49:44.874337 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:44.874316 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5733856-6a7b-4ccf-9289-fdd9e2497bdb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b5733856-6a7b-4ccf-9289-fdd9e2497bdb" (UID: "b5733856-6a7b-4ccf-9289-fdd9e2497bdb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:49:44.975417 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:44.975339 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5733856-6a7b-4ccf-9289-fdd9e2497bdb-maas-api-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:49:45.718790 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:45.718755 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cb7974-tv2d8" event={"ID":"b5733856-6a7b-4ccf-9289-fdd9e2497bdb","Type":"ContainerDied","Data":"8beecc5f4ecdbce8272b5064209c588b63d122a77f760f224385212bb580fa7a"} Jul 14 22:49:45.718790 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:45.718786 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cb7974-tv2d8" Jul 14 22:49:45.719003 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:45.718805 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8beecc5f4ecdbce8272b5064209c588b63d122a77f760f224385212bb580fa7a" Jul 14 22:49:49.750917 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:49.750880 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd"] Jul 14 22:49:49.751364 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:49.751181 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd" podUID="d0793a99-ef68-478e-86c3-63e6ea6d7bec" containerName="maas-api" containerID="cri-o://3d671ba67dcef5a86e58451c0bec9fdda75905c31d87c081c8ca8ec9a3defa8a" gracePeriod=30 Jul 14 22:49:49.783477 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:49.783447 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:49:50.192220 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:50.192197 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd" Jul 14 22:49:50.216788 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:50.216765 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0793a99-ef68-478e-86c3-63e6ea6d7bec-openshift-service-ca\") pod \"d0793a99-ef68-478e-86c3-63e6ea6d7bec\" (UID: \"d0793a99-ef68-478e-86c3-63e6ea6d7bec\") " Jul 14 22:49:50.216910 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:50.216802 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0793a99-ef68-478e-86c3-63e6ea6d7bec-maas-api-tls\") pod \"d0793a99-ef68-478e-86c3-63e6ea6d7bec\" (UID: \"d0793a99-ef68-478e-86c3-63e6ea6d7bec\") " Jul 14 22:49:50.216910 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:50.216873 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87mq4\" (UniqueName: \"kubernetes.io/projected/d0793a99-ef68-478e-86c3-63e6ea6d7bec-kube-api-access-87mq4\") pod \"d0793a99-ef68-478e-86c3-63e6ea6d7bec\" (UID: \"d0793a99-ef68-478e-86c3-63e6ea6d7bec\") " Jul 14 22:49:50.217139 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:50.217111 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0793a99-ef68-478e-86c3-63e6ea6d7bec-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d0793a99-ef68-478e-86c3-63e6ea6d7bec" (UID: "d0793a99-ef68-478e-86c3-63e6ea6d7bec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:49:50.218831 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:50.218809 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0793a99-ef68-478e-86c3-63e6ea6d7bec-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d0793a99-ef68-478e-86c3-63e6ea6d7bec" (UID: "d0793a99-ef68-478e-86c3-63e6ea6d7bec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 22:49:50.218938 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:50.218915 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0793a99-ef68-478e-86c3-63e6ea6d7bec-kube-api-access-87mq4" (OuterVolumeSpecName: "kube-api-access-87mq4") pod "d0793a99-ef68-478e-86c3-63e6ea6d7bec" (UID: "d0793a99-ef68-478e-86c3-63e6ea6d7bec"). InnerVolumeSpecName "kube-api-access-87mq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:49:50.318441 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:50.318358 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0793a99-ef68-478e-86c3-63e6ea6d7bec-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:49:50.318441 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:50.318383 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0793a99-ef68-478e-86c3-63e6ea6d7bec-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:49:50.318441 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:50.318393 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87mq4\" (UniqueName: \"kubernetes.io/projected/d0793a99-ef68-478e-86c3-63e6ea6d7bec-kube-api-access-87mq4\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:49:50.747745 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:50.747711 2550 generic.go:356] "Generic (PLEG): container finished" podID="d0793a99-ef68-478e-86c3-63e6ea6d7bec" containerID="3d671ba67dcef5a86e58451c0bec9fdda75905c31d87c081c8ca8ec9a3defa8a" exitCode=0 Jul 14 22:49:50.747945 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:50.747765 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd" Jul 14 22:49:50.747945 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:50.747798 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd" event={"ID":"d0793a99-ef68-478e-86c3-63e6ea6d7bec","Type":"ContainerDied","Data":"3d671ba67dcef5a86e58451c0bec9fdda75905c31d87c081c8ca8ec9a3defa8a"} Jul 14 22:49:50.747945 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:50.747855 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd" event={"ID":"d0793a99-ef68-478e-86c3-63e6ea6d7bec","Type":"ContainerDied","Data":"007790118aca738d312a5e6f11212652617c6affbe85fff7ca7f7fc86e093390"} Jul 14 22:49:50.747945 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:50.747878 2550 scope.go:117] "RemoveContainer" containerID="3d671ba67dcef5a86e58451c0bec9fdda75905c31d87c081c8ca8ec9a3defa8a" Jul 14 22:49:50.757896 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:50.757733 2550 scope.go:117] "RemoveContainer" containerID="3d671ba67dcef5a86e58451c0bec9fdda75905c31d87c081c8ca8ec9a3defa8a" Jul 14 22:49:50.758126 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:49:50.757969 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d671ba67dcef5a86e58451c0bec9fdda75905c31d87c081c8ca8ec9a3defa8a\": container with ID starting with 3d671ba67dcef5a86e58451c0bec9fdda75905c31d87c081c8ca8ec9a3defa8a not found: ID does not exist" containerID="3d671ba67dcef5a86e58451c0bec9fdda75905c31d87c081c8ca8ec9a3defa8a" Jul 14 22:49:50.758126 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:50.757994 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d671ba67dcef5a86e58451c0bec9fdda75905c31d87c081c8ca8ec9a3defa8a"} err="failed to get container status \"3d671ba67dcef5a86e58451c0bec9fdda75905c31d87c081c8ca8ec9a3defa8a\": rpc error: code = NotFound desc = could not find container \"3d671ba67dcef5a86e58451c0bec9fdda75905c31d87c081c8ca8ec9a3defa8a\": container with ID starting with 3d671ba67dcef5a86e58451c0bec9fdda75905c31d87c081c8ca8ec9a3defa8a not found: ID does not exist" Jul 14 22:49:50.770867 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:50.770839 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd"] Jul 14 22:49:50.772813 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:50.772794 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cb7974-6cb8cc9f5-fr2vd"] Jul 14 22:49:52.195280 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:52.195249 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0793a99-ef68-478e-86c3-63e6ea6d7bec" path="/var/lib/kubelet/pods/d0793a99-ef68-478e-86c3-63e6ea6d7bec/volumes" Jul 14 22:49:54.365173 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:49:54.365143 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:01.045807 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:01.045771 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:01.146287 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:01.146255 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:05.122613 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:05.122576 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:05.369744 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:05.369710 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:05.711224 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:05.711177 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:06.119427 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:06.119348 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:06.333371 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:06.333334 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:09.200013 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:09.199982 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:09.291905 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:09.291877 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:09.402184 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:09.402157 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf"] Jul 14 22:50:09.402506 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:09.402494 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0793a99-ef68-478e-86c3-63e6ea6d7bec" containerName="maas-api" Jul 14 22:50:09.402613 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:09.402507 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0793a99-ef68-478e-86c3-63e6ea6d7bec" containerName="maas-api" Jul 14 22:50:09.402613 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:09.402533 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5733856-6a7b-4ccf-9289-fdd9e2497bdb" containerName="revoke-keys" Jul 14 22:50:09.402613 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:09.402539 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5733856-6a7b-4ccf-9289-fdd9e2497bdb" containerName="revoke-keys" Jul 14 22:50:09.402729 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:09.402622 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5733856-6a7b-4ccf-9289-fdd9e2497bdb" containerName="revoke-keys" Jul 14 22:50:09.402729 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:09.402633 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0793a99-ef68-478e-86c3-63e6ea6d7bec" containerName="maas-api" Jul 14 22:50:09.405591 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:09.405575 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf" Jul 14 22:50:09.408874 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:09.408857 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-971ed6-serving-cert\"" Jul 14 22:50:09.422619 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:09.422595 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf"] Jul 14 22:50:09.475150 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:09.475061 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:09.493430 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:09.493408 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8311dd4-1875-4dde-9d7c-5b9a9db675c1-maas-api-tls\") pod \"maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf\" (UID: \"d8311dd4-1875-4dde-9d7c-5b9a9db675c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf" Jul 14 22:50:09.493599 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:09.493462 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmh2d\" (UniqueName: \"kubernetes.io/projected/d8311dd4-1875-4dde-9d7c-5b9a9db675c1-kube-api-access-dmh2d\") pod \"maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf\" (UID: \"d8311dd4-1875-4dde-9d7c-5b9a9db675c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf" Jul 14 22:50:09.493599 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:09.493545 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8311dd4-1875-4dde-9d7c-5b9a9db675c1-openshift-service-ca\") pod \"maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf\" (UID: \"d8311dd4-1875-4dde-9d7c-5b9a9db675c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf" Jul 14 22:50:09.594318 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:09.594279 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8311dd4-1875-4dde-9d7c-5b9a9db675c1-maas-api-tls\") pod \"maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf\" (UID: \"d8311dd4-1875-4dde-9d7c-5b9a9db675c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf" Jul 14 22:50:09.594516 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:09.594337 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmh2d\" (UniqueName: \"kubernetes.io/projected/d8311dd4-1875-4dde-9d7c-5b9a9db675c1-kube-api-access-dmh2d\") pod \"maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf\" (UID: \"d8311dd4-1875-4dde-9d7c-5b9a9db675c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf" Jul 14 22:50:09.594516 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:09.594370 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8311dd4-1875-4dde-9d7c-5b9a9db675c1-openshift-service-ca\") pod \"maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf\" (UID: \"d8311dd4-1875-4dde-9d7c-5b9a9db675c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf" Jul 14 22:50:09.594516 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:50:09.594453 2550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-971ed6-serving-cert: secret "maas-api-e2e-api-a-971ed6-serving-cert" not found Jul 14 22:50:09.594770 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:50:09.594571 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8311dd4-1875-4dde-9d7c-5b9a9db675c1-maas-api-tls podName:d8311dd4-1875-4dde-9d7c-5b9a9db675c1 nodeName:}" failed. No retries permitted until 2026-07-14 22:50:10.094531507 +0000 UTC m=+2440.453677927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d8311dd4-1875-4dde-9d7c-5b9a9db675c1-maas-api-tls") pod "maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf" (UID: "d8311dd4-1875-4dde-9d7c-5b9a9db675c1") : secret "maas-api-e2e-api-a-971ed6-serving-cert" not found Jul 14 22:50:09.595129 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:09.595107 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8311dd4-1875-4dde-9d7c-5b9a9db675c1-openshift-service-ca\") pod \"maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf\" (UID: \"d8311dd4-1875-4dde-9d7c-5b9a9db675c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf" Jul 14 22:50:09.612018 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:09.611998 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmh2d\" (UniqueName: \"kubernetes.io/projected/d8311dd4-1875-4dde-9d7c-5b9a9db675c1-kube-api-access-dmh2d\") pod \"maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf\" (UID: \"d8311dd4-1875-4dde-9d7c-5b9a9db675c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf" Jul 14 22:50:09.930801 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:09.930762 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:10.027718 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:10.027687 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:10.098417 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:10.098384 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8311dd4-1875-4dde-9d7c-5b9a9db675c1-maas-api-tls\") pod \"maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf\" (UID: \"d8311dd4-1875-4dde-9d7c-5b9a9db675c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf" Jul 14 22:50:10.100932 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:10.100906 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8311dd4-1875-4dde-9d7c-5b9a9db675c1-maas-api-tls\") pod \"maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf\" (UID: \"d8311dd4-1875-4dde-9d7c-5b9a9db675c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf" Jul 14 22:50:10.315889 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:10.315808 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf" Jul 14 22:50:10.654597 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:10.654575 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf"] Jul 14 22:50:10.656201 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:50:10.656174 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8311dd4_1875_4dde_9d7c_5b9a9db675c1.slice/crio-ec664b1a01388caa78b589c40031b167318917c432069d7c00a66992ca06766b WatchSource:0}: Error finding container ec664b1a01388caa78b589c40031b167318917c432069d7c00a66992ca06766b: Status 404 returned error can't find the container with id ec664b1a01388caa78b589c40031b167318917c432069d7c00a66992ca06766b Jul 14 22:50:10.847216 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:10.847175 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf" event={"ID":"d8311dd4-1875-4dde-9d7c-5b9a9db675c1","Type":"ContainerStarted","Data":"ec664b1a01388caa78b589c40031b167318917c432069d7c00a66992ca06766b"} Jul 14 22:50:11.853704 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:11.853673 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf" event={"ID":"d8311dd4-1875-4dde-9d7c-5b9a9db675c1","Type":"ContainerStarted","Data":"0bc8e4f7c37d5e712b2f090aa95660001f787fe8221f01596320c3ac05a436e5"} Jul 14 22:50:11.854091 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:11.853791 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf" Jul 14 22:50:11.886181 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:11.886129 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf" podStartSLOduration=2.393156285 podStartE2EDuration="2.886112806s" totalImagesPullingTime="492.956521ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:50:11.884130019 +0000 UTC m=+2442.243276456" watchObservedRunningTime="2026-07-14 22:50:11.886112806 +0000 UTC m=+2442.245259245" Jul 14 22:50:13.747286 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:13.747258 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:14.284269 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:14.284243 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:14.747856 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:14.747823 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:14.909622 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:14.909595 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:17.863316 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:17.863288 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf" Jul 14 22:50:18.322238 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:18.322193 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:18.420240 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:18.420209 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:18.489455 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:18.489424 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7215d1-85b848694d-bxzcs"] Jul 14 22:50:18.493541 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:18.493519 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7215d1-85b848694d-bxzcs" Jul 14 22:50:18.499062 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:18.499039 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-7215d1-serving-cert\"" Jul 14 22:50:18.507013 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:18.506990 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7215d1-85b848694d-bxzcs"] Jul 14 22:50:18.538426 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:18.538400 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:18.639807 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:18.639724 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:18.672607 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:18.672570 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fa6f47c-b185-422f-8112-0821ae5c9966-maas-api-tls\") pod \"maas-api-e2e-api-b-7215d1-85b848694d-bxzcs\" (UID: \"2fa6f47c-b185-422f-8112-0821ae5c9966\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7215d1-85b848694d-bxzcs" Jul 14 22:50:18.672771 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:18.672657 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fa6f47c-b185-422f-8112-0821ae5c9966-openshift-service-ca\") pod \"maas-api-e2e-api-b-7215d1-85b848694d-bxzcs\" (UID: \"2fa6f47c-b185-422f-8112-0821ae5c9966\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7215d1-85b848694d-bxzcs" Jul 14 22:50:18.672771 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:18.672692 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62zdr\" (UniqueName: \"kubernetes.io/projected/2fa6f47c-b185-422f-8112-0821ae5c9966-kube-api-access-62zdr\") pod \"maas-api-e2e-api-b-7215d1-85b848694d-bxzcs\" (UID: \"2fa6f47c-b185-422f-8112-0821ae5c9966\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7215d1-85b848694d-bxzcs" Jul 14 22:50:18.773413 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:18.773381 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fa6f47c-b185-422f-8112-0821ae5c9966-openshift-service-ca\") pod \"maas-api-e2e-api-b-7215d1-85b848694d-bxzcs\" (UID: \"2fa6f47c-b185-422f-8112-0821ae5c9966\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7215d1-85b848694d-bxzcs" Jul 14 22:50:18.773628 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:18.773418 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62zdr\" (UniqueName: \"kubernetes.io/projected/2fa6f47c-b185-422f-8112-0821ae5c9966-kube-api-access-62zdr\") pod \"maas-api-e2e-api-b-7215d1-85b848694d-bxzcs\" (UID: \"2fa6f47c-b185-422f-8112-0821ae5c9966\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7215d1-85b848694d-bxzcs" Jul 14 22:50:18.773628 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:18.773543 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fa6f47c-b185-422f-8112-0821ae5c9966-maas-api-tls\") pod \"maas-api-e2e-api-b-7215d1-85b848694d-bxzcs\" (UID: \"2fa6f47c-b185-422f-8112-0821ae5c9966\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7215d1-85b848694d-bxzcs" Jul 14 22:50:18.774101 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:18.774075 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fa6f47c-b185-422f-8112-0821ae5c9966-openshift-service-ca\") pod \"maas-api-e2e-api-b-7215d1-85b848694d-bxzcs\" (UID: \"2fa6f47c-b185-422f-8112-0821ae5c9966\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7215d1-85b848694d-bxzcs" Jul 14 22:50:18.775951 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:18.775930 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fa6f47c-b185-422f-8112-0821ae5c9966-maas-api-tls\") pod \"maas-api-e2e-api-b-7215d1-85b848694d-bxzcs\" (UID: \"2fa6f47c-b185-422f-8112-0821ae5c9966\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7215d1-85b848694d-bxzcs" Jul 14 22:50:18.788607 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:18.788581 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62zdr\" (UniqueName: \"kubernetes.io/projected/2fa6f47c-b185-422f-8112-0821ae5c9966-kube-api-access-62zdr\") pod \"maas-api-e2e-api-b-7215d1-85b848694d-bxzcs\" (UID: \"2fa6f47c-b185-422f-8112-0821ae5c9966\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7215d1-85b848694d-bxzcs" Jul 14 22:50:18.805311 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:18.805290 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7215d1-85b848694d-bxzcs" Jul 14 22:50:18.945921 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:18.945892 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7215d1-85b848694d-bxzcs"] Jul 14 22:50:18.947729 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:50:18.947699 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa6f47c_b185_422f_8112_0821ae5c9966.slice/crio-8837b2f9186e270a0f1c4b1ff35eed1c7617c6a2847e9b80f329d4f22c99a949 WatchSource:0}: Error finding container 8837b2f9186e270a0f1c4b1ff35eed1c7617c6a2847e9b80f329d4f22c99a949: Status 404 returned error can't find the container with id 8837b2f9186e270a0f1c4b1ff35eed1c7617c6a2847e9b80f329d4f22c99a949 Jul 14 22:50:19.209404 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:19.209323 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:19.306364 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:19.306333 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:19.894847 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:19.894812 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7215d1-85b848694d-bxzcs" event={"ID":"2fa6f47c-b185-422f-8112-0821ae5c9966","Type":"ContainerStarted","Data":"b85c7e66b2d1e71447fd9f40232f83cd048c5862b1a09600842da03e6d43cb1b"} Jul 14 22:50:19.895055 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:19.894876 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7215d1-85b848694d-bxzcs" Jul 14 22:50:19.895055 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:19.894896 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7215d1-85b848694d-bxzcs" event={"ID":"2fa6f47c-b185-422f-8112-0821ae5c9966","Type":"ContainerStarted","Data":"8837b2f9186e270a0f1c4b1ff35eed1c7617c6a2847e9b80f329d4f22c99a949"} Jul 14 22:50:19.911919 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:19.911877 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7215d1-85b848694d-bxzcs" podStartSLOduration=1.491810367 podStartE2EDuration="1.911864698s" totalImagesPullingTime="420.054331ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:50:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:50:19.911439451 +0000 UTC m=+2450.270585899" watchObservedRunningTime="2026-07-14 22:50:19.911864698 +0000 UTC m=+2450.271011134" Jul 14 22:50:22.287703 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:22.287668 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:22.388036 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:22.387997 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:22.818051 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:22.818015 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:25.907152 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:25.907127 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7215d1-85b848694d-bxzcs" Jul 14 22:50:30.862485 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:30.862451 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:30.968631 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:30.968531 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:31.079441 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:31.079407 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-971ed6-2f758"] Jul 14 22:50:31.084133 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:31.084113 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-971ed6-2f758" Jul 14 22:50:31.086777 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:31.086742 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mj4xg\"" Jul 14 22:50:31.089937 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:31.089913 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-971ed6-2f758"] Jul 14 22:50:31.168644 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:31.168615 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d81d81d-e70c-4251-9420-056f247d175b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-971ed6-2f758\" (UID: \"5d81d81d-e70c-4251-9420-056f247d175b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-971ed6-2f758" Jul 14 22:50:31.269283 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:31.269240 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d81d81d-e70c-4251-9420-056f247d175b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-971ed6-2f758\" (UID: \"5d81d81d-e70c-4251-9420-056f247d175b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-971ed6-2f758" Jul 14 22:50:31.269886 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:31.269864 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d81d81d-e70c-4251-9420-056f247d175b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-971ed6-2f758\" (UID: \"5d81d81d-e70c-4251-9420-056f247d175b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-971ed6-2f758" Jul 14 22:50:31.395422 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:31.395393 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-971ed6-2f758" Jul 14 22:50:31.528775 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:31.528750 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-971ed6-2f758"] Jul 14 22:50:31.530420 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:50:31.530393 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d81d81d_e70c_4251_9420_056f247d175b.slice/crio-58414f63040a88b6ca9f205e9df458e591c49c6e5c2d94d7b49028d621d0f14d WatchSource:0}: Error finding container 58414f63040a88b6ca9f205e9df458e591c49c6e5c2d94d7b49028d621d0f14d: Status 404 returned error can't find the container with id 58414f63040a88b6ca9f205e9df458e591c49c6e5c2d94d7b49028d621d0f14d Jul 14 22:50:31.955301 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:31.955268 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-971ed6-2f758" event={"ID":"5d81d81d-e70c-4251-9420-056f247d175b","Type":"ContainerStarted","Data":"58414f63040a88b6ca9f205e9df458e591c49c6e5c2d94d7b49028d621d0f14d"} Jul 14 22:50:32.962019 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:32.961987 2550 generic.go:356] "Generic (PLEG): container finished" podID="5d81d81d-e70c-4251-9420-056f247d175b" containerID="db72bd7954f679e57b9cfb20adb1656a1769328bcd062087c1aba27b3f80cab1" exitCode=0 Jul 14 22:50:32.962377 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:32.962072 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-971ed6-2f758" event={"ID":"5d81d81d-e70c-4251-9420-056f247d175b","Type":"ContainerDied","Data":"db72bd7954f679e57b9cfb20adb1656a1769328bcd062087c1aba27b3f80cab1"} Jul 14 22:50:34.097362 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:34.097337 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-971ed6-2f758" Jul 14 22:50:34.189540 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:34.189514 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d81d81d-e70c-4251-9420-056f247d175b-maas-api-service-ca\") pod \"5d81d81d-e70c-4251-9420-056f247d175b\" (UID: \"5d81d81d-e70c-4251-9420-056f247d175b\") " Jul 14 22:50:34.189928 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:34.189901 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d81d81d-e70c-4251-9420-056f247d175b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5d81d81d-e70c-4251-9420-056f247d175b" (UID: "5d81d81d-e70c-4251-9420-056f247d175b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:50:34.290637 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:34.290545 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d81d81d-e70c-4251-9420-056f247d175b-maas-api-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:50:34.973622 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:34.973590 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-971ed6-2f758" event={"ID":"5d81d81d-e70c-4251-9420-056f247d175b","Type":"ContainerDied","Data":"58414f63040a88b6ca9f205e9df458e591c49c6e5c2d94d7b49028d621d0f14d"} Jul 14 22:50:34.973772 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:34.973638 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58414f63040a88b6ca9f205e9df458e591c49c6e5c2d94d7b49028d621d0f14d" Jul 14 22:50:34.973772 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:34.973596 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-971ed6-2f758" Jul 14 22:50:40.194925 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:40.194883 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:40.295348 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:40.295305 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:41.700268 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:41.700235 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:41.801394 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:41.799262 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:41.958140 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:41.958049 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf"] Jul 14 22:50:41.958284 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:41.958245 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf" podUID="d8311dd4-1875-4dde-9d7c-5b9a9db675c1" containerName="maas-api" containerID="cri-o://0bc8e4f7c37d5e712b2f090aa95660001f787fe8221f01596320c3ac05a436e5" gracePeriod=30 Jul 14 22:50:41.978767 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:41.978743 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:42.218785 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:42.218725 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf" Jul 14 22:50:42.264813 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:42.262839 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8311dd4-1875-4dde-9d7c-5b9a9db675c1-maas-api-tls\") pod \"d8311dd4-1875-4dde-9d7c-5b9a9db675c1\" (UID: \"d8311dd4-1875-4dde-9d7c-5b9a9db675c1\") " Jul 14 22:50:42.264813 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:42.262891 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8311dd4-1875-4dde-9d7c-5b9a9db675c1-openshift-service-ca\") pod \"d8311dd4-1875-4dde-9d7c-5b9a9db675c1\" (UID: \"d8311dd4-1875-4dde-9d7c-5b9a9db675c1\") " Jul 14 22:50:42.264813 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:42.262968 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmh2d\" (UniqueName: \"kubernetes.io/projected/d8311dd4-1875-4dde-9d7c-5b9a9db675c1-kube-api-access-dmh2d\") pod \"d8311dd4-1875-4dde-9d7c-5b9a9db675c1\" (UID: \"d8311dd4-1875-4dde-9d7c-5b9a9db675c1\") " Jul 14 22:50:42.264813 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:42.264293 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8311dd4-1875-4dde-9d7c-5b9a9db675c1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d8311dd4-1875-4dde-9d7c-5b9a9db675c1" (UID: "d8311dd4-1875-4dde-9d7c-5b9a9db675c1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:50:42.266890 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:42.266862 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8311dd4-1875-4dde-9d7c-5b9a9db675c1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d8311dd4-1875-4dde-9d7c-5b9a9db675c1" (UID: "d8311dd4-1875-4dde-9d7c-5b9a9db675c1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 22:50:42.267103 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:42.267083 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8311dd4-1875-4dde-9d7c-5b9a9db675c1-kube-api-access-dmh2d" (OuterVolumeSpecName: "kube-api-access-dmh2d") pod "d8311dd4-1875-4dde-9d7c-5b9a9db675c1" (UID: "d8311dd4-1875-4dde-9d7c-5b9a9db675c1"). InnerVolumeSpecName "kube-api-access-dmh2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:50:42.364244 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:42.364211 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8311dd4-1875-4dde-9d7c-5b9a9db675c1-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:50:42.364244 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:42.364238 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8311dd4-1875-4dde-9d7c-5b9a9db675c1-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:50:42.364244 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:42.364248 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmh2d\" (UniqueName: \"kubernetes.io/projected/d8311dd4-1875-4dde-9d7c-5b9a9db675c1-kube-api-access-dmh2d\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:50:43.014134 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:43.014100 2550 generic.go:356] "Generic (PLEG): container finished" podID="d8311dd4-1875-4dde-9d7c-5b9a9db675c1" containerID="0bc8e4f7c37d5e712b2f090aa95660001f787fe8221f01596320c3ac05a436e5" exitCode=0 Jul 14 22:50:43.014530 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:43.014155 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf" Jul 14 22:50:43.014530 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:43.014190 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf" event={"ID":"d8311dd4-1875-4dde-9d7c-5b9a9db675c1","Type":"ContainerDied","Data":"0bc8e4f7c37d5e712b2f090aa95660001f787fe8221f01596320c3ac05a436e5"} Jul 14 22:50:43.014530 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:43.014267 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf" event={"ID":"d8311dd4-1875-4dde-9d7c-5b9a9db675c1","Type":"ContainerDied","Data":"ec664b1a01388caa78b589c40031b167318917c432069d7c00a66992ca06766b"} Jul 14 22:50:43.014530 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:43.014291 2550 scope.go:117] "RemoveContainer" containerID="0bc8e4f7c37d5e712b2f090aa95660001f787fe8221f01596320c3ac05a436e5" Jul 14 22:50:43.023726 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:43.023711 2550 scope.go:117] "RemoveContainer" containerID="0bc8e4f7c37d5e712b2f090aa95660001f787fe8221f01596320c3ac05a436e5" Jul 14 22:50:43.023947 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:50:43.023931 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bc8e4f7c37d5e712b2f090aa95660001f787fe8221f01596320c3ac05a436e5\": container with ID starting with 0bc8e4f7c37d5e712b2f090aa95660001f787fe8221f01596320c3ac05a436e5 not found: ID does not exist" containerID="0bc8e4f7c37d5e712b2f090aa95660001f787fe8221f01596320c3ac05a436e5" Jul 14 22:50:43.023994 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:43.023955 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bc8e4f7c37d5e712b2f090aa95660001f787fe8221f01596320c3ac05a436e5"} err="failed to get container status \"0bc8e4f7c37d5e712b2f090aa95660001f787fe8221f01596320c3ac05a436e5\": rpc error: code = NotFound desc = could not find container \"0bc8e4f7c37d5e712b2f090aa95660001f787fe8221f01596320c3ac05a436e5\": container with ID starting with 0bc8e4f7c37d5e712b2f090aa95660001f787fe8221f01596320c3ac05a436e5 not found: ID does not exist" Jul 14 22:50:43.036571 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:43.036536 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf"] Jul 14 22:50:43.041065 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:43.041045 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-971ed6-5fb7d84d79-btvvf"] Jul 14 22:50:44.195238 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:44.195205 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8311dd4-1875-4dde-9d7c-5b9a9db675c1" path="/var/lib/kubelet/pods/d8311dd4-1875-4dde-9d7c-5b9a9db675c1/volumes" Jul 14 22:50:46.208531 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:46.208500 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:52.275203 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:52.275146 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:52.376542 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:52.376506 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:56.946206 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:56.946174 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:57.192158 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:57.192126 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:57.327296 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:57.327222 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7215d1-nhknm"] Jul 14 22:50:57.327570 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:57.327537 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8311dd4-1875-4dde-9d7c-5b9a9db675c1" containerName="maas-api" Jul 14 22:50:57.327570 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:57.327563 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8311dd4-1875-4dde-9d7c-5b9a9db675c1" containerName="maas-api" Jul 14 22:50:57.327656 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:57.327574 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d81d81d-e70c-4251-9420-056f247d175b" containerName="revoke-keys" Jul 14 22:50:57.327656 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:57.327579 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d81d81d-e70c-4251-9420-056f247d175b" containerName="revoke-keys" Jul 14 22:50:57.327656 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:57.327636 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d81d81d-e70c-4251-9420-056f247d175b" containerName="revoke-keys" Jul 14 22:50:57.327656 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:57.327646 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8311dd4-1875-4dde-9d7c-5b9a9db675c1" containerName="maas-api" Jul 14 22:50:57.332083 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:57.332063 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7215d1-nhknm" Jul 14 22:50:57.334841 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:57.334818 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mj4xg\"" Jul 14 22:50:57.341210 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:57.341179 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7215d1-nhknm"] Jul 14 22:50:57.394988 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:57.394957 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f5bce7a-aa7e-42ad-90a2-3476475017f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7215d1-nhknm\" (UID: \"9f5bce7a-aa7e-42ad-90a2-3476475017f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7215d1-nhknm" Jul 14 22:50:57.496297 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:57.496263 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f5bce7a-aa7e-42ad-90a2-3476475017f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7215d1-nhknm\" (UID: \"9f5bce7a-aa7e-42ad-90a2-3476475017f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7215d1-nhknm" Jul 14 22:50:57.496898 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:57.496880 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f5bce7a-aa7e-42ad-90a2-3476475017f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7215d1-nhknm\" (UID: \"9f5bce7a-aa7e-42ad-90a2-3476475017f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7215d1-nhknm" Jul 14 22:50:57.644594 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:57.644492 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7215d1-nhknm" Jul 14 22:50:57.774040 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:57.774016 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7215d1-nhknm"] Jul 14 22:50:57.775641 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:50:57.775612 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f5bce7a_aa7e_42ad_90a2_3476475017f2.slice/crio-27f16cea6bb3d3b9cba61a5e111f8516c32ea4ad244bbca49dc85c61be7c863a WatchSource:0}: Error finding container 27f16cea6bb3d3b9cba61a5e111f8516c32ea4ad244bbca49dc85c61be7c863a: Status 404 returned error can't find the container with id 27f16cea6bb3d3b9cba61a5e111f8516c32ea4ad244bbca49dc85c61be7c863a Jul 14 22:50:57.998584 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:57.998545 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:58.090460 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:58.090432 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7215d1-nhknm" event={"ID":"9f5bce7a-aa7e-42ad-90a2-3476475017f2","Type":"ContainerStarted","Data":"b968aa36a4d15c50f34b1419d0ecadc01636dbcb09f37f3454cbbcaceb540c62"} Jul 14 22:50:58.090572 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:58.090491 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7215d1-nhknm" event={"ID":"9f5bce7a-aa7e-42ad-90a2-3476475017f2","Type":"ContainerStarted","Data":"27f16cea6bb3d3b9cba61a5e111f8516c32ea4ad244bbca49dc85c61be7c863a"} Jul 14 22:50:58.098260 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:58.098240 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:50:58.107000 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:58.106959 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7215d1-nhknm" podStartSLOduration=1.106944978 podStartE2EDuration="1.106944978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:50:58.105124858 +0000 UTC m=+2488.464271296" watchObservedRunningTime="2026-07-14 22:50:58.106944978 +0000 UTC m=+2488.466091415" Jul 14 22:50:59.096451 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:59.096421 2550 generic.go:356] "Generic (PLEG): container finished" podID="9f5bce7a-aa7e-42ad-90a2-3476475017f2" containerID="b968aa36a4d15c50f34b1419d0ecadc01636dbcb09f37f3454cbbcaceb540c62" exitCode=0 Jul 14 22:50:59.096868 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:50:59.096474 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7215d1-nhknm" event={"ID":"9f5bce7a-aa7e-42ad-90a2-3476475017f2","Type":"ContainerDied","Data":"b968aa36a4d15c50f34b1419d0ecadc01636dbcb09f37f3454cbbcaceb540c62"} Jul 14 22:51:00.227914 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:00.227888 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7215d1-nhknm" Jul 14 22:51:00.317999 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:00.317969 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f5bce7a-aa7e-42ad-90a2-3476475017f2-maas-api-service-ca\") pod \"9f5bce7a-aa7e-42ad-90a2-3476475017f2\" (UID: \"9f5bce7a-aa7e-42ad-90a2-3476475017f2\") " Jul 14 22:51:00.318253 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:00.318232 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f5bce7a-aa7e-42ad-90a2-3476475017f2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9f5bce7a-aa7e-42ad-90a2-3476475017f2" (UID: "9f5bce7a-aa7e-42ad-90a2-3476475017f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:51:00.419502 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:00.419478 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f5bce7a-aa7e-42ad-90a2-3476475017f2-maas-api-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:51:01.107414 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:01.107383 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7215d1-nhknm" Jul 14 22:51:01.107414 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:01.107401 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7215d1-nhknm" event={"ID":"9f5bce7a-aa7e-42ad-90a2-3476475017f2","Type":"ContainerDied","Data":"27f16cea6bb3d3b9cba61a5e111f8516c32ea4ad244bbca49dc85c61be7c863a"} Jul 14 22:51:01.107650 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:01.107477 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27f16cea6bb3d3b9cba61a5e111f8516c32ea4ad244bbca49dc85c61be7c863a" Jul 14 22:51:05.443916 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:05.443884 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:51:07.856186 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:07.856155 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7215d1-85b848694d-bxzcs"] Jul 14 22:51:07.856649 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:07.856402 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7215d1-85b848694d-bxzcs" podUID="2fa6f47c-b185-422f-8112-0821ae5c9966" containerName="maas-api" containerID="cri-o://b85c7e66b2d1e71447fd9f40232f83cd048c5862b1a09600842da03e6d43cb1b" gracePeriod=30 Jul 14 22:51:07.883282 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:07.883253 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:51:08.107927 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:08.107871 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7215d1-85b848694d-bxzcs" Jul 14 22:51:08.144394 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:08.144360 2550 generic.go:356] "Generic (PLEG): container finished" podID="2fa6f47c-b185-422f-8112-0821ae5c9966" containerID="b85c7e66b2d1e71447fd9f40232f83cd048c5862b1a09600842da03e6d43cb1b" exitCode=0 Jul 14 22:51:08.144595 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:08.144396 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7215d1-85b848694d-bxzcs" Jul 14 22:51:08.144595 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:08.144448 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7215d1-85b848694d-bxzcs" event={"ID":"2fa6f47c-b185-422f-8112-0821ae5c9966","Type":"ContainerDied","Data":"b85c7e66b2d1e71447fd9f40232f83cd048c5862b1a09600842da03e6d43cb1b"} Jul 14 22:51:08.144595 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:08.144517 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7215d1-85b848694d-bxzcs" event={"ID":"2fa6f47c-b185-422f-8112-0821ae5c9966","Type":"ContainerDied","Data":"8837b2f9186e270a0f1c4b1ff35eed1c7617c6a2847e9b80f329d4f22c99a949"} Jul 14 22:51:08.144595 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:08.144540 2550 scope.go:117] "RemoveContainer" containerID="b85c7e66b2d1e71447fd9f40232f83cd048c5862b1a09600842da03e6d43cb1b" Jul 14 22:51:08.153606 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:08.153584 2550 scope.go:117] "RemoveContainer" containerID="b85c7e66b2d1e71447fd9f40232f83cd048c5862b1a09600842da03e6d43cb1b" Jul 14 22:51:08.153858 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:51:08.153827 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b85c7e66b2d1e71447fd9f40232f83cd048c5862b1a09600842da03e6d43cb1b\": container with ID starting with b85c7e66b2d1e71447fd9f40232f83cd048c5862b1a09600842da03e6d43cb1b not found: ID does not exist" containerID="b85c7e66b2d1e71447fd9f40232f83cd048c5862b1a09600842da03e6d43cb1b" Jul 14 22:51:08.153907 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:08.153866 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b85c7e66b2d1e71447fd9f40232f83cd048c5862b1a09600842da03e6d43cb1b"} err="failed to get container status \"b85c7e66b2d1e71447fd9f40232f83cd048c5862b1a09600842da03e6d43cb1b\": rpc error: code = NotFound desc = could not find container \"b85c7e66b2d1e71447fd9f40232f83cd048c5862b1a09600842da03e6d43cb1b\": container with ID starting with b85c7e66b2d1e71447fd9f40232f83cd048c5862b1a09600842da03e6d43cb1b not found: ID does not exist" Jul 14 22:51:08.195055 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:08.195032 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62zdr\" (UniqueName: \"kubernetes.io/projected/2fa6f47c-b185-422f-8112-0821ae5c9966-kube-api-access-62zdr\") pod \"2fa6f47c-b185-422f-8112-0821ae5c9966\" (UID: \"2fa6f47c-b185-422f-8112-0821ae5c9966\") " Jul 14 22:51:08.195148 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:08.195106 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fa6f47c-b185-422f-8112-0821ae5c9966-maas-api-tls\") pod \"2fa6f47c-b185-422f-8112-0821ae5c9966\" (UID: \"2fa6f47c-b185-422f-8112-0821ae5c9966\") " Jul 14 22:51:08.195216 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:08.195160 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fa6f47c-b185-422f-8112-0821ae5c9966-openshift-service-ca\") pod \"2fa6f47c-b185-422f-8112-0821ae5c9966\" (UID: \"2fa6f47c-b185-422f-8112-0821ae5c9966\") " Jul 14 22:51:08.195523 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:08.195500 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fa6f47c-b185-422f-8112-0821ae5c9966-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2fa6f47c-b185-422f-8112-0821ae5c9966" (UID: "2fa6f47c-b185-422f-8112-0821ae5c9966"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:51:08.197059 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:08.197027 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa6f47c-b185-422f-8112-0821ae5c9966-kube-api-access-62zdr" (OuterVolumeSpecName: "kube-api-access-62zdr") pod "2fa6f47c-b185-422f-8112-0821ae5c9966" (UID: "2fa6f47c-b185-422f-8112-0821ae5c9966"). InnerVolumeSpecName "kube-api-access-62zdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:51:08.197059 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:08.197038 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fa6f47c-b185-422f-8112-0821ae5c9966-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2fa6f47c-b185-422f-8112-0821ae5c9966" (UID: "2fa6f47c-b185-422f-8112-0821ae5c9966"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 22:51:08.296448 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:08.296412 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62zdr\" (UniqueName: \"kubernetes.io/projected/2fa6f47c-b185-422f-8112-0821ae5c9966-kube-api-access-62zdr\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:51:08.296448 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:08.296442 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fa6f47c-b185-422f-8112-0821ae5c9966-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:51:08.296448 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:08.296451 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fa6f47c-b185-422f-8112-0821ae5c9966-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:51:08.467545 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:08.467503 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7215d1-85b848694d-bxzcs"] Jul 14 22:51:08.470585 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:08.470544 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7215d1-85b848694d-bxzcs"] Jul 14 22:51:10.194851 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:10.194819 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fa6f47c-b185-422f-8112-0821ae5c9966" path="/var/lib/kubelet/pods/2fa6f47c-b185-422f-8112-0821ae5c9966/volumes" Jul 14 22:51:12.422892 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:12.422860 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:51:18.704785 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:18.704751 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:51:18.804848 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:18.804816 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:51:23.159474 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:23.159405 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:51:23.409863 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:23.409839 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:51:23.797011 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:23.796928 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:51:24.136533 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:24.136450 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:51:24.266867 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:24.266836 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:51:27.566425 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:27.566394 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:51:27.659069 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:27.659039 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t"] Jul 14 22:51:27.659413 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:27.659396 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fa6f47c-b185-422f-8112-0821ae5c9966" containerName="maas-api" Jul 14 22:51:27.659413 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:27.659412 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fa6f47c-b185-422f-8112-0821ae5c9966" containerName="maas-api" Jul 14 22:51:27.659598 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:27.659427 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f5bce7a-aa7e-42ad-90a2-3476475017f2" containerName="revoke-keys" Jul 14 22:51:27.659598 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:27.659433 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f5bce7a-aa7e-42ad-90a2-3476475017f2" containerName="revoke-keys" Jul 14 22:51:27.659598 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:27.659514 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fa6f47c-b185-422f-8112-0821ae5c9966" containerName="maas-api" Jul 14 22:51:27.659598 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:27.659529 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f5bce7a-aa7e-42ad-90a2-3476475017f2" containerName="revoke-keys" Jul 14 22:51:27.662517 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:27.662490 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:51:27.664271 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:27.664254 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t" Jul 14 22:51:27.666862 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:27.666841 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-7ab9a6-serving-cert\"" Jul 14 22:51:27.673546 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:27.673522 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t"] Jul 14 22:51:27.740308 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:27.740267 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:51:27.756496 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:27.756470 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5c055d9-c060-41ba-a4ac-c14278cdb1d5-maas-api-tls\") pod \"maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t\" (UID: \"b5c055d9-c060-41ba-a4ac-c14278cdb1d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t" Jul 14 22:51:27.756638 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:27.756500 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfvpv\" (UniqueName: \"kubernetes.io/projected/b5c055d9-c060-41ba-a4ac-c14278cdb1d5-kube-api-access-dfvpv\") pod \"maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t\" (UID: \"b5c055d9-c060-41ba-a4ac-c14278cdb1d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t" Jul 14 22:51:27.756638 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:27.756531 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c055d9-c060-41ba-a4ac-c14278cdb1d5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t\" (UID: \"b5c055d9-c060-41ba-a4ac-c14278cdb1d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t" Jul 14 22:51:27.843167 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:27.843066 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:51:27.857545 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:27.857519 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5c055d9-c060-41ba-a4ac-c14278cdb1d5-maas-api-tls\") pod \"maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t\" (UID: \"b5c055d9-c060-41ba-a4ac-c14278cdb1d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t" Jul 14 22:51:27.857545 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:27.857570 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfvpv\" (UniqueName: \"kubernetes.io/projected/b5c055d9-c060-41ba-a4ac-c14278cdb1d5-kube-api-access-dfvpv\") pod \"maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t\" (UID: \"b5c055d9-c060-41ba-a4ac-c14278cdb1d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t" Jul 14 22:51:27.857769 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:27.857749 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c055d9-c060-41ba-a4ac-c14278cdb1d5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t\" (UID: \"b5c055d9-c060-41ba-a4ac-c14278cdb1d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t" Jul 14 22:51:27.858317 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:27.858294 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c055d9-c060-41ba-a4ac-c14278cdb1d5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t\" (UID: \"b5c055d9-c060-41ba-a4ac-c14278cdb1d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t" Jul 14 22:51:27.859851 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:27.859825 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5c055d9-c060-41ba-a4ac-c14278cdb1d5-maas-api-tls\") pod \"maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t\" (UID: \"b5c055d9-c060-41ba-a4ac-c14278cdb1d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t" Jul 14 22:51:27.867132 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:27.867107 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfvpv\" (UniqueName: \"kubernetes.io/projected/b5c055d9-c060-41ba-a4ac-c14278cdb1d5-kube-api-access-dfvpv\") pod \"maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t\" (UID: \"b5c055d9-c060-41ba-a4ac-c14278cdb1d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t" Jul 14 22:51:27.976038 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:27.976006 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t" Jul 14 22:51:28.106794 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:28.106770 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t"] Jul 14 22:51:28.108669 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:51:28.108639 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5c055d9_c060_41ba_a4ac_c14278cdb1d5.slice/crio-a0d646969f7953f8f4dfe9b7eb8961799995ddb5b75a70f7b51b9bcce969de19 WatchSource:0}: Error finding container a0d646969f7953f8f4dfe9b7eb8961799995ddb5b75a70f7b51b9bcce969de19: Status 404 returned error can't find the container with id a0d646969f7953f8f4dfe9b7eb8961799995ddb5b75a70f7b51b9bcce969de19 Jul 14 22:51:28.242149 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:28.242117 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:51:28.244268 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:28.244240 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t" event={"ID":"b5c055d9-c060-41ba-a4ac-c14278cdb1d5","Type":"ContainerStarted","Data":"a0d646969f7953f8f4dfe9b7eb8961799995ddb5b75a70f7b51b9bcce969de19"} Jul 14 22:51:28.343038 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:28.342999 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:51:29.250990 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:29.250956 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t" event={"ID":"b5c055d9-c060-41ba-a4ac-c14278cdb1d5","Type":"ContainerStarted","Data":"d5e7c89d0a876e0236d1694a3d799e9eba1c8f9791439993e05975649bf0a4ac"} Jul 14 22:51:29.251361 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:29.251066 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t" Jul 14 22:51:29.270332 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:29.270286 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t" podStartSLOduration=1.752542254 podStartE2EDuration="2.270272493s" totalImagesPullingTime="517.730239ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:51:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:51:29.269489696 +0000 UTC m=+2519.628636134" watchObservedRunningTime="2026-07-14 22:51:29.270272493 +0000 UTC m=+2519.629418931" Jul 14 22:51:32.122809 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:32.122775 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:51:32.484287 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:32.484252 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:51:32.821269 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:32.821191 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:51:32.934106 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:32.934070 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:51:35.262066 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:35.262040 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t" Jul 14 22:51:35.643228 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:35.643149 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:51:35.725991 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:35.725956 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-546296-684bd79c45-sf4gn"] Jul 14 22:51:35.727117 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:35.727090 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:51:35.730899 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:35.730882 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-546296-684bd79c45-sf4gn" Jul 14 22:51:35.733263 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:35.733245 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-546296-serving-cert\"" Jul 14 22:51:35.739447 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:35.739423 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-546296-684bd79c45-sf4gn"] Jul 14 22:51:35.806128 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:35.806096 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:51:35.826283 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:35.826249 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/312722d1-879d-434c-8447-e4ae4bf42adf-openshift-service-ca\") pod \"maas-api-e2e-shared-b-546296-684bd79c45-sf4gn\" (UID: \"312722d1-879d-434c-8447-e4ae4bf42adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-546296-684bd79c45-sf4gn" Jul 14 22:51:35.826435 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:35.826314 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/312722d1-879d-434c-8447-e4ae4bf42adf-maas-api-tls\") pod \"maas-api-e2e-shared-b-546296-684bd79c45-sf4gn\" (UID: \"312722d1-879d-434c-8447-e4ae4bf42adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-546296-684bd79c45-sf4gn" Jul 14 22:51:35.826435 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:35.826356 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f8rh\" (UniqueName: \"kubernetes.io/projected/312722d1-879d-434c-8447-e4ae4bf42adf-kube-api-access-8f8rh\") pod \"maas-api-e2e-shared-b-546296-684bd79c45-sf4gn\" (UID: \"312722d1-879d-434c-8447-e4ae4bf42adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-546296-684bd79c45-sf4gn" Jul 14 22:51:35.926961 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:35.926927 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/312722d1-879d-434c-8447-e4ae4bf42adf-openshift-service-ca\") pod \"maas-api-e2e-shared-b-546296-684bd79c45-sf4gn\" (UID: \"312722d1-879d-434c-8447-e4ae4bf42adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-546296-684bd79c45-sf4gn" Jul 14 22:51:35.927138 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:35.926992 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/312722d1-879d-434c-8447-e4ae4bf42adf-maas-api-tls\") pod \"maas-api-e2e-shared-b-546296-684bd79c45-sf4gn\" (UID: \"312722d1-879d-434c-8447-e4ae4bf42adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-546296-684bd79c45-sf4gn" Jul 14 22:51:35.927138 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:35.927103 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f8rh\" (UniqueName: \"kubernetes.io/projected/312722d1-879d-434c-8447-e4ae4bf42adf-kube-api-access-8f8rh\") pod \"maas-api-e2e-shared-b-546296-684bd79c45-sf4gn\" (UID: \"312722d1-879d-434c-8447-e4ae4bf42adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-546296-684bd79c45-sf4gn" Jul 14 22:51:35.927262 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:51:35.927104 2550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-546296-serving-cert: secret "maas-api-e2e-shared-b-546296-serving-cert" not found Jul 14 22:51:35.927315 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:51:35.927278 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/312722d1-879d-434c-8447-e4ae4bf42adf-maas-api-tls podName:312722d1-879d-434c-8447-e4ae4bf42adf nodeName:}" failed. No retries permitted until 2026-07-14 22:51:36.427253914 +0000 UTC m=+2526.786400335 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/312722d1-879d-434c-8447-e4ae4bf42adf-maas-api-tls") pod "maas-api-e2e-shared-b-546296-684bd79c45-sf4gn" (UID: "312722d1-879d-434c-8447-e4ae4bf42adf") : secret "maas-api-e2e-shared-b-546296-serving-cert" not found Jul 14 22:51:35.927540 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:35.927519 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/312722d1-879d-434c-8447-e4ae4bf42adf-openshift-service-ca\") pod \"maas-api-e2e-shared-b-546296-684bd79c45-sf4gn\" (UID: \"312722d1-879d-434c-8447-e4ae4bf42adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-546296-684bd79c45-sf4gn" Jul 14 22:51:35.936264 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:35.936239 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f8rh\" (UniqueName: \"kubernetes.io/projected/312722d1-879d-434c-8447-e4ae4bf42adf-kube-api-access-8f8rh\") pod \"maas-api-e2e-shared-b-546296-684bd79c45-sf4gn\" (UID: \"312722d1-879d-434c-8447-e4ae4bf42adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-546296-684bd79c45-sf4gn" Jul 14 22:51:36.091456 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:36.091424 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:51:36.333207 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:36.331674 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:51:36.430186 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:36.430153 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:51:36.432128 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:36.432108 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/312722d1-879d-434c-8447-e4ae4bf42adf-maas-api-tls\") pod \"maas-api-e2e-shared-b-546296-684bd79c45-sf4gn\" (UID: \"312722d1-879d-434c-8447-e4ae4bf42adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-546296-684bd79c45-sf4gn" Jul 14 22:51:36.434537 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:36.434508 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/312722d1-879d-434c-8447-e4ae4bf42adf-maas-api-tls\") pod \"maas-api-e2e-shared-b-546296-684bd79c45-sf4gn\" (UID: \"312722d1-879d-434c-8447-e4ae4bf42adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-546296-684bd79c45-sf4gn" Jul 14 22:51:36.643218 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:36.643136 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-546296-684bd79c45-sf4gn" Jul 14 22:51:36.772960 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:36.772935 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-546296-684bd79c45-sf4gn"] Jul 14 22:51:36.775116 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:51:36.775088 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod312722d1_879d_434c_8447_e4ae4bf42adf.slice/crio-6e89769124fb1fd4c1fe4b27a0608dabe1c1e86ddeac6e1f72adc36462bb5d5c WatchSource:0}: Error finding container 6e89769124fb1fd4c1fe4b27a0608dabe1c1e86ddeac6e1f72adc36462bb5d5c: Status 404 returned error can't find the container with id 6e89769124fb1fd4c1fe4b27a0608dabe1c1e86ddeac6e1f72adc36462bb5d5c Jul 14 22:51:37.295092 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:37.295056 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-546296-684bd79c45-sf4gn" event={"ID":"312722d1-879d-434c-8447-e4ae4bf42adf","Type":"ContainerStarted","Data":"779a1a9c851c7b800a072f38dc2d2a3d7086eb311c13a8aaf0bb0ce242729cc1"} Jul 14 22:51:37.295206 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:37.295124 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-546296-684bd79c45-sf4gn" Jul 14 22:51:37.295206 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:37.295142 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-546296-684bd79c45-sf4gn" event={"ID":"312722d1-879d-434c-8447-e4ae4bf42adf","Type":"ContainerStarted","Data":"6e89769124fb1fd4c1fe4b27a0608dabe1c1e86ddeac6e1f72adc36462bb5d5c"} Jul 14 22:51:37.338506 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:37.338463 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-546296-684bd79c45-sf4gn" podStartSLOduration=1.898913246 podStartE2EDuration="2.338449097s" totalImagesPullingTime="439.535851ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:51:37.33658518 +0000 UTC m=+2527.695731619" watchObservedRunningTime="2026-07-14 22:51:37.338449097 +0000 UTC m=+2527.697595534" Jul 14 22:51:40.177677 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:40.177641 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:51:43.305433 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:43.305404 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-546296-684bd79c45-sf4gn" Jul 14 22:51:50.101350 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:50.101307 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:51:50.202129 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:50.202100 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:51:57.248157 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:57.248116 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:51:57.345895 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:51:57.345861 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:52:09.683253 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:52:09.683210 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:52:39.314083 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:52:39.314043 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:53:44.619886 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:53:44.619849 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:54:15.400410 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:15.400378 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:54:30.247024 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:30.246956 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zj4sb_0884baa5-9206-4196-a7c9-b5686623311b/ovn-acl-logging/0.log" Jul 14 22:54:30.255346 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:30.255322 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zj4sb_0884baa5-9206-4196-a7c9-b5686623311b/ovn-acl-logging/0.log" Jul 14 22:54:36.482678 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:36.482640 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:54:37.425818 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:37.425769 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:54:48.987644 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:48.987615 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7"] Jul 14 22:54:48.991057 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:48.991042 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7" Jul 14 22:54:48.996749 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:48.995216 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7ab9a6\"/\"openshift-service-ca.crt\"" Jul 14 22:54:48.996749 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:48.995287 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7ab9a6\"/\"default-dockercfg-dlfns\"" Jul 14 22:54:48.996749 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:48.996202 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7ab9a6\"/\"kube-root-ca.crt\"" Jul 14 22:54:48.997486 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:48.997394 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7ab9a6\"/\"sub-test-model-7ab9a6-kserve-self-signed-certs\"" Jul 14 22:54:49.005763 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:49.005736 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7"] Jul 14 22:54:49.006682 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:49.006661 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:54:49.094768 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:49.094738 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12ded869-889c-4103-a4d8-3d9ba8f161fd-tls-certs\") pod \"sub-test-model-7ab9a6-kserve-55465748dc-kldc7\" (UID: \"12ded869-889c-4103-a4d8-3d9ba8f161fd\") " pod="ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7" Jul 14 22:54:49.094903 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:49.094782 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12ded869-889c-4103-a4d8-3d9ba8f161fd-tmp-dir\") pod \"sub-test-model-7ab9a6-kserve-55465748dc-kldc7\" (UID: \"12ded869-889c-4103-a4d8-3d9ba8f161fd\") " pod="ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7" Jul 14 22:54:49.094903 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:49.094852 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12ded869-889c-4103-a4d8-3d9ba8f161fd-model-cache\") pod \"sub-test-model-7ab9a6-kserve-55465748dc-kldc7\" (UID: \"12ded869-889c-4103-a4d8-3d9ba8f161fd\") " pod="ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7" Jul 14 22:54:49.094903 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:49.094877 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12ded869-889c-4103-a4d8-3d9ba8f161fd-home\") pod \"sub-test-model-7ab9a6-kserve-55465748dc-kldc7\" (UID: \"12ded869-889c-4103-a4d8-3d9ba8f161fd\") " pod="ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7" Jul 14 22:54:49.094903 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:49.094897 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12ded869-889c-4103-a4d8-3d9ba8f161fd-dshm\") pod \"sub-test-model-7ab9a6-kserve-55465748dc-kldc7\" (UID: \"12ded869-889c-4103-a4d8-3d9ba8f161fd\") " pod="ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7" Jul 14 22:54:49.095052 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:49.094925 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7v68\" (UniqueName: \"kubernetes.io/projected/12ded869-889c-4103-a4d8-3d9ba8f161fd-kube-api-access-s7v68\") pod \"sub-test-model-7ab9a6-kserve-55465748dc-kldc7\" (UID: \"12ded869-889c-4103-a4d8-3d9ba8f161fd\") " pod="ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7" Jul 14 22:54:49.195809 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:49.195773 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12ded869-889c-4103-a4d8-3d9ba8f161fd-model-cache\") pod \"sub-test-model-7ab9a6-kserve-55465748dc-kldc7\" (UID: \"12ded869-889c-4103-a4d8-3d9ba8f161fd\") " pod="ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7" Jul 14 22:54:49.197407 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:49.196042 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12ded869-889c-4103-a4d8-3d9ba8f161fd-home\") pod \"sub-test-model-7ab9a6-kserve-55465748dc-kldc7\" (UID: \"12ded869-889c-4103-a4d8-3d9ba8f161fd\") " pod="ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7" Jul 14 22:54:49.197407 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:49.196085 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12ded869-889c-4103-a4d8-3d9ba8f161fd-dshm\") pod \"sub-test-model-7ab9a6-kserve-55465748dc-kldc7\" (UID: \"12ded869-889c-4103-a4d8-3d9ba8f161fd\") " pod="ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7" Jul 14 22:54:49.197407 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:49.196115 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7v68\" (UniqueName: \"kubernetes.io/projected/12ded869-889c-4103-a4d8-3d9ba8f161fd-kube-api-access-s7v68\") pod \"sub-test-model-7ab9a6-kserve-55465748dc-kldc7\" (UID: \"12ded869-889c-4103-a4d8-3d9ba8f161fd\") " pod="ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7" Jul 14 22:54:49.197407 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:49.196188 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12ded869-889c-4103-a4d8-3d9ba8f161fd-tls-certs\") pod \"sub-test-model-7ab9a6-kserve-55465748dc-kldc7\" (UID: \"12ded869-889c-4103-a4d8-3d9ba8f161fd\") " pod="ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7" Jul 14 22:54:49.197407 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:49.196233 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12ded869-889c-4103-a4d8-3d9ba8f161fd-tmp-dir\") pod \"sub-test-model-7ab9a6-kserve-55465748dc-kldc7\" (UID: \"12ded869-889c-4103-a4d8-3d9ba8f161fd\") " pod="ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7" Jul 14 22:54:49.197407 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:49.196686 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12ded869-889c-4103-a4d8-3d9ba8f161fd-tmp-dir\") pod \"sub-test-model-7ab9a6-kserve-55465748dc-kldc7\" (UID: \"12ded869-889c-4103-a4d8-3d9ba8f161fd\") " pod="ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7" Jul 14 22:54:49.197407 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:49.196909 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12ded869-889c-4103-a4d8-3d9ba8f161fd-model-cache\") pod \"sub-test-model-7ab9a6-kserve-55465748dc-kldc7\" (UID: \"12ded869-889c-4103-a4d8-3d9ba8f161fd\") " pod="ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7" Jul 14 22:54:49.197407 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:49.197115 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12ded869-889c-4103-a4d8-3d9ba8f161fd-home\") pod \"sub-test-model-7ab9a6-kserve-55465748dc-kldc7\" (UID: \"12ded869-889c-4103-a4d8-3d9ba8f161fd\") " pod="ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7" Jul 14 22:54:49.200325 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:49.200282 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12ded869-889c-4103-a4d8-3d9ba8f161fd-dshm\") pod \"sub-test-model-7ab9a6-kserve-55465748dc-kldc7\" (UID: \"12ded869-889c-4103-a4d8-3d9ba8f161fd\") " pod="ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7" Jul 14 22:54:49.202739 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:49.200574 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12ded869-889c-4103-a4d8-3d9ba8f161fd-tls-certs\") pod \"sub-test-model-7ab9a6-kserve-55465748dc-kldc7\" (UID: \"12ded869-889c-4103-a4d8-3d9ba8f161fd\") " pod="ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7" Jul 14 22:54:49.205691 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:49.205670 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7v68\" (UniqueName: \"kubernetes.io/projected/12ded869-889c-4103-a4d8-3d9ba8f161fd-kube-api-access-s7v68\") pod \"sub-test-model-7ab9a6-kserve-55465748dc-kldc7\" (UID: \"12ded869-889c-4103-a4d8-3d9ba8f161fd\") " pod="ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7" Jul 14 22:54:49.307079 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:49.306832 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7" Jul 14 22:54:49.440786 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:49.440759 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7"] Jul 14 22:54:49.442363 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:54:49.442337 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12ded869_889c_4103_a4d8_3d9ba8f161fd.slice/crio-e2a152c3cf73865c65a63785b89dcb13418d905fc8ea6e114ac0522617d1fb88 WatchSource:0}: Error finding container e2a152c3cf73865c65a63785b89dcb13418d905fc8ea6e114ac0522617d1fb88: Status 404 returned error can't find the container with id e2a152c3cf73865c65a63785b89dcb13418d905fc8ea6e114ac0522617d1fb88 Jul 14 22:54:50.267837 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:50.267805 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7" event={"ID":"12ded869-889c-4103-a4d8-3d9ba8f161fd","Type":"ContainerStarted","Data":"52c21a8a1a2651ced4db8e33c3be9652d1df110abcdaca8cfaa28f1afc435c7c"} Jul 14 22:54:50.268221 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:50.267846 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7" event={"ID":"12ded869-889c-4103-a4d8-3d9ba8f161fd","Type":"ContainerStarted","Data":"e2a152c3cf73865c65a63785b89dcb13418d905fc8ea6e114ac0522617d1fb88"} Jul 14 22:54:50.285598 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:50.285530 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7" podStartSLOduration=2.2855168790000002 podStartE2EDuration="2.285516879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:54:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:54:50.283891805 +0000 UTC m=+2720.643038243" watchObservedRunningTime="2026-07-14 22:54:50.285516879 +0000 UTC m=+2720.644663316" Jul 14 22:54:59.306951 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:59.306917 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7" Jul 14 22:54:59.307315 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:59.306971 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7" Jul 14 22:54:59.319508 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:59.319480 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7" Jul 14 22:54:59.330812 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:54:59.330789 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7" Jul 14 22:55:06.122629 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:55:06.122593 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:55:23.253437 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:55:23.253403 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:55:23.352826 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:55:23.352779 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:55:58.884929 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:55:58.884843 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7"] Jul 14 22:55:58.904129 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:55:58.904101 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:55:59.714378 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:55:59.714347 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:56:13.905241 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:13.905187 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7" podUID="12ded869-889c-4103-a4d8-3d9ba8f161fd" containerName="main" containerID="cri-o://52c21a8a1a2651ced4db8e33c3be9652d1df110abcdaca8cfaa28f1afc435c7c" gracePeriod=45 Jul 14 22:56:14.150969 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:14.150947 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7" Jul 14 22:56:14.244167 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:14.244138 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12ded869-889c-4103-a4d8-3d9ba8f161fd-tls-certs\") pod \"12ded869-889c-4103-a4d8-3d9ba8f161fd\" (UID: \"12ded869-889c-4103-a4d8-3d9ba8f161fd\") " Jul 14 22:56:14.244356 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:14.244175 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12ded869-889c-4103-a4d8-3d9ba8f161fd-dshm\") pod \"12ded869-889c-4103-a4d8-3d9ba8f161fd\" (UID: \"12ded869-889c-4103-a4d8-3d9ba8f161fd\") " Jul 14 22:56:14.244356 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:14.244214 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12ded869-889c-4103-a4d8-3d9ba8f161fd-tmp-dir\") pod \"12ded869-889c-4103-a4d8-3d9ba8f161fd\" (UID: \"12ded869-889c-4103-a4d8-3d9ba8f161fd\") " Jul 14 22:56:14.244356 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:14.244233 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12ded869-889c-4103-a4d8-3d9ba8f161fd-model-cache\") pod \"12ded869-889c-4103-a4d8-3d9ba8f161fd\" (UID: \"12ded869-889c-4103-a4d8-3d9ba8f161fd\") " Jul 14 22:56:14.244356 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:14.244263 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7v68\" (UniqueName: \"kubernetes.io/projected/12ded869-889c-4103-a4d8-3d9ba8f161fd-kube-api-access-s7v68\") pod \"12ded869-889c-4103-a4d8-3d9ba8f161fd\" (UID: \"12ded869-889c-4103-a4d8-3d9ba8f161fd\") " Jul 14 22:56:14.244356 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:14.244285 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12ded869-889c-4103-a4d8-3d9ba8f161fd-home\") pod \"12ded869-889c-4103-a4d8-3d9ba8f161fd\" (UID: \"12ded869-889c-4103-a4d8-3d9ba8f161fd\") " Jul 14 22:56:14.244679 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:14.244527 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ded869-889c-4103-a4d8-3d9ba8f161fd-model-cache" (OuterVolumeSpecName: "model-cache") pod "12ded869-889c-4103-a4d8-3d9ba8f161fd" (UID: "12ded869-889c-4103-a4d8-3d9ba8f161fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:56:14.244679 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:14.244541 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ded869-889c-4103-a4d8-3d9ba8f161fd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "12ded869-889c-4103-a4d8-3d9ba8f161fd" (UID: "12ded869-889c-4103-a4d8-3d9ba8f161fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:56:14.244679 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:14.244645 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ded869-889c-4103-a4d8-3d9ba8f161fd-home" (OuterVolumeSpecName: "home") pod "12ded869-889c-4103-a4d8-3d9ba8f161fd" (UID: "12ded869-889c-4103-a4d8-3d9ba8f161fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:56:14.246433 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:14.246407 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12ded869-889c-4103-a4d8-3d9ba8f161fd-kube-api-access-s7v68" (OuterVolumeSpecName: "kube-api-access-s7v68") pod "12ded869-889c-4103-a4d8-3d9ba8f161fd" (UID: "12ded869-889c-4103-a4d8-3d9ba8f161fd"). InnerVolumeSpecName "kube-api-access-s7v68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:56:14.246529 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:14.246444 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12ded869-889c-4103-a4d8-3d9ba8f161fd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "12ded869-889c-4103-a4d8-3d9ba8f161fd" (UID: "12ded869-889c-4103-a4d8-3d9ba8f161fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 22:56:14.246529 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:14.246447 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ded869-889c-4103-a4d8-3d9ba8f161fd-dshm" (OuterVolumeSpecName: "dshm") pod "12ded869-889c-4103-a4d8-3d9ba8f161fd" (UID: "12ded869-889c-4103-a4d8-3d9ba8f161fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 22:56:14.345829 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:14.345799 2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12ded869-889c-4103-a4d8-3d9ba8f161fd-tls-certs\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:56:14.345829 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:14.345828 2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12ded869-889c-4103-a4d8-3d9ba8f161fd-dshm\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:56:14.346014 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:14.345840 2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12ded869-889c-4103-a4d8-3d9ba8f161fd-tmp-dir\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:56:14.346014 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:14.345848 2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12ded869-889c-4103-a4d8-3d9ba8f161fd-model-cache\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:56:14.346014 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:14.345855 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7v68\" (UniqueName: \"kubernetes.io/projected/12ded869-889c-4103-a4d8-3d9ba8f161fd-kube-api-access-s7v68\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:56:14.346014 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:14.345863 2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12ded869-889c-4103-a4d8-3d9ba8f161fd-home\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:56:14.705130 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:14.705094 2550 generic.go:356] "Generic (PLEG): container finished" podID="12ded869-889c-4103-a4d8-3d9ba8f161fd" containerID="52c21a8a1a2651ced4db8e33c3be9652d1df110abcdaca8cfaa28f1afc435c7c" exitCode=0 Jul 14 22:56:14.705321 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:14.705142 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7" event={"ID":"12ded869-889c-4103-a4d8-3d9ba8f161fd","Type":"ContainerDied","Data":"52c21a8a1a2651ced4db8e33c3be9652d1df110abcdaca8cfaa28f1afc435c7c"} Jul 14 22:56:14.705321 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:14.705150 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7" Jul 14 22:56:14.705321 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:14.705173 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7" event={"ID":"12ded869-889c-4103-a4d8-3d9ba8f161fd","Type":"ContainerDied","Data":"e2a152c3cf73865c65a63785b89dcb13418d905fc8ea6e114ac0522617d1fb88"} Jul 14 22:56:14.705321 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:14.705189 2550 scope.go:117] "RemoveContainer" containerID="52c21a8a1a2651ced4db8e33c3be9652d1df110abcdaca8cfaa28f1afc435c7c" Jul 14 22:56:14.715025 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:14.715008 2550 scope.go:117] "RemoveContainer" containerID="52c21a8a1a2651ced4db8e33c3be9652d1df110abcdaca8cfaa28f1afc435c7c" Jul 14 22:56:14.715307 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:56:14.715281 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52c21a8a1a2651ced4db8e33c3be9652d1df110abcdaca8cfaa28f1afc435c7c\": container with ID starting with 52c21a8a1a2651ced4db8e33c3be9652d1df110abcdaca8cfaa28f1afc435c7c not found: ID does not exist" containerID="52c21a8a1a2651ced4db8e33c3be9652d1df110abcdaca8cfaa28f1afc435c7c" Jul 14 22:56:14.715412 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:14.715318 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52c21a8a1a2651ced4db8e33c3be9652d1df110abcdaca8cfaa28f1afc435c7c"} err="failed to get container status \"52c21a8a1a2651ced4db8e33c3be9652d1df110abcdaca8cfaa28f1afc435c7c\": rpc error: code = NotFound desc = could not find container \"52c21a8a1a2651ced4db8e33c3be9652d1df110abcdaca8cfaa28f1afc435c7c\": container with ID starting with 52c21a8a1a2651ced4db8e33c3be9652d1df110abcdaca8cfaa28f1afc435c7c not found: ID does not exist" Jul 14 22:56:14.728299 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:14.728259 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7"] Jul 14 22:56:14.734503 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:14.734479 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-7ab9a6/sub-test-model-7ab9a6-kserve-55465748dc-kldc7"] Jul 14 22:56:16.196801 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:16.196768 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12ded869-889c-4103-a4d8-3d9ba8f161fd" path="/var/lib/kubelet/pods/12ded869-889c-4103-a4d8-3d9ba8f161fd/volumes" Jul 14 22:56:17.267615 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:17.267580 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:56:19.716978 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:19.716946 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:56:39.720086 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:39.720052 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:56:50.926882 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:50.926840 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:56:51.027960 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:51.027928 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:56:58.150208 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:58.150173 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:56:58.249185 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:56:58.249147 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:57:29.449870 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:57:29.449776 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:57:31.035395 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:57:31.035362 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:57:49.630352 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:57:49.630276 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7ab9a6-v2nnd"] Jul 14 22:57:49.630721 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:57:49.630622 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12ded869-889c-4103-a4d8-3d9ba8f161fd" containerName="main" Jul 14 22:57:49.630721 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:57:49.630634 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="12ded869-889c-4103-a4d8-3d9ba8f161fd" containerName="main" Jul 14 22:57:49.630721 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:57:49.630710 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="12ded869-889c-4103-a4d8-3d9ba8f161fd" containerName="main" Jul 14 22:57:49.633530 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:57:49.633512 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7ab9a6-v2nnd" Jul 14 22:57:49.636238 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:57:49.636217 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mj4xg\"" Jul 14 22:57:49.641984 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:57:49.641961 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7ab9a6-v2nnd"] Jul 14 22:57:49.677675 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:57:49.677643 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bde58c9-9bdd-4fed-b157-db3abbcc5152-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-7ab9a6-v2nnd\" (UID: \"5bde58c9-9bdd-4fed-b157-db3abbcc5152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7ab9a6-v2nnd" Jul 14 22:57:49.778482 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:57:49.778437 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bde58c9-9bdd-4fed-b157-db3abbcc5152-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-7ab9a6-v2nnd\" (UID: \"5bde58c9-9bdd-4fed-b157-db3abbcc5152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7ab9a6-v2nnd" Jul 14 22:57:49.779061 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:57:49.779037 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bde58c9-9bdd-4fed-b157-db3abbcc5152-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-7ab9a6-v2nnd\" (UID: \"5bde58c9-9bdd-4fed-b157-db3abbcc5152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7ab9a6-v2nnd" Jul 14 22:57:49.945459 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:57:49.945424 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7ab9a6-v2nnd" Jul 14 22:57:50.073576 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:57:50.073540 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7ab9a6-v2nnd"] Jul 14 22:57:50.075094 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:57:50.075057 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bde58c9_9bdd_4fed_b157_db3abbcc5152.slice/crio-9041cad7f18cc4d947ad40e29fe60fa0fbf22959d2b637e4c93780d225701ccc WatchSource:0}: Error finding container 9041cad7f18cc4d947ad40e29fe60fa0fbf22959d2b637e4c93780d225701ccc: Status 404 returned error can't find the container with id 9041cad7f18cc4d947ad40e29fe60fa0fbf22959d2b637e4c93780d225701ccc Jul 14 22:57:50.207492 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:57:50.207406 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7ab9a6-v2nnd" event={"ID":"5bde58c9-9bdd-4fed-b157-db3abbcc5152","Type":"ContainerStarted","Data":"9041cad7f18cc4d947ad40e29fe60fa0fbf22959d2b637e4c93780d225701ccc"} Jul 14 22:57:50.268115 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:57:50.268084 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:57:50.370530 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:57:50.370498 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:57:51.213959 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:57:51.213925 2550 generic.go:356] "Generic (PLEG): container finished" podID="5bde58c9-9bdd-4fed-b157-db3abbcc5152" containerID="2ea4f93f6215c69d992f979b7334858dc43867297002af13626cb20905f9ca06" exitCode=0 Jul 14 22:57:51.214381 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:57:51.214009 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7ab9a6-v2nnd" event={"ID":"5bde58c9-9bdd-4fed-b157-db3abbcc5152","Type":"ContainerDied","Data":"2ea4f93f6215c69d992f979b7334858dc43867297002af13626cb20905f9ca06"} Jul 14 22:57:52.351857 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:57:52.351832 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7ab9a6-v2nnd" Jul 14 22:57:52.397010 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:57:52.396982 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bde58c9-9bdd-4fed-b157-db3abbcc5152-maas-api-service-ca\") pod \"5bde58c9-9bdd-4fed-b157-db3abbcc5152\" (UID: \"5bde58c9-9bdd-4fed-b157-db3abbcc5152\") " Jul 14 22:57:52.397281 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:57:52.397261 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bde58c9-9bdd-4fed-b157-db3abbcc5152-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5bde58c9-9bdd-4fed-b157-db3abbcc5152" (UID: "5bde58c9-9bdd-4fed-b157-db3abbcc5152"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:57:52.498005 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:57:52.497945 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bde58c9-9bdd-4fed-b157-db3abbcc5152-maas-api-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:57:52.779040 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:57:52.778954 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:57:53.225084 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:57:53.225057 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7ab9a6-v2nnd" Jul 14 22:57:53.225084 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:57:53.225078 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7ab9a6-v2nnd" event={"ID":"5bde58c9-9bdd-4fed-b157-db3abbcc5152","Type":"ContainerDied","Data":"9041cad7f18cc4d947ad40e29fe60fa0fbf22959d2b637e4c93780d225701ccc"} Jul 14 22:57:53.225343 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:57:53.225119 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9041cad7f18cc4d947ad40e29fe60fa0fbf22959d2b637e4c93780d225701ccc" Jul 14 22:58:00.133000 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:00.132970 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t"] Jul 14 22:58:00.133480 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:00.133219 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t" podUID="b5c055d9-c060-41ba-a4ac-c14278cdb1d5" containerName="maas-api" containerID="cri-o://d5e7c89d0a876e0236d1694a3d799e9eba1c8f9791439993e05975649bf0a4ac" gracePeriod=30 Jul 14 22:58:00.160972 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:00.160926 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:58:00.257617 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:00.257581 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t" podUID="b5c055d9-c060-41ba-a4ac-c14278cdb1d5" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.50:8443/health\": dial tcp 10.134.0.50:8443: connect: connection refused" Jul 14 22:58:00.260630 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:00.260602 2550 generic.go:356] "Generic (PLEG): container finished" podID="b5c055d9-c060-41ba-a4ac-c14278cdb1d5" containerID="d5e7c89d0a876e0236d1694a3d799e9eba1c8f9791439993e05975649bf0a4ac" exitCode=0 Jul 14 22:58:00.260775 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:00.260730 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t" event={"ID":"b5c055d9-c060-41ba-a4ac-c14278cdb1d5","Type":"ContainerDied","Data":"d5e7c89d0a876e0236d1694a3d799e9eba1c8f9791439993e05975649bf0a4ac"} Jul 14 22:58:00.381434 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:00.381411 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t" Jul 14 22:58:00.456175 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:00.456142 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfvpv\" (UniqueName: \"kubernetes.io/projected/b5c055d9-c060-41ba-a4ac-c14278cdb1d5-kube-api-access-dfvpv\") pod \"b5c055d9-c060-41ba-a4ac-c14278cdb1d5\" (UID: \"b5c055d9-c060-41ba-a4ac-c14278cdb1d5\") " Jul 14 22:58:00.456370 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:00.456187 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c055d9-c060-41ba-a4ac-c14278cdb1d5-openshift-service-ca\") pod \"b5c055d9-c060-41ba-a4ac-c14278cdb1d5\" (UID: \"b5c055d9-c060-41ba-a4ac-c14278cdb1d5\") " Jul 14 22:58:00.456370 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:00.456234 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5c055d9-c060-41ba-a4ac-c14278cdb1d5-maas-api-tls\") pod \"b5c055d9-c060-41ba-a4ac-c14278cdb1d5\" (UID: \"b5c055d9-c060-41ba-a4ac-c14278cdb1d5\") " Jul 14 22:58:00.456622 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:00.456590 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5c055d9-c060-41ba-a4ac-c14278cdb1d5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b5c055d9-c060-41ba-a4ac-c14278cdb1d5" (UID: "b5c055d9-c060-41ba-a4ac-c14278cdb1d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:58:00.458250 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:00.458229 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5c055d9-c060-41ba-a4ac-c14278cdb1d5-kube-api-access-dfvpv" (OuterVolumeSpecName: "kube-api-access-dfvpv") pod "b5c055d9-c060-41ba-a4ac-c14278cdb1d5" (UID: "b5c055d9-c060-41ba-a4ac-c14278cdb1d5"). InnerVolumeSpecName "kube-api-access-dfvpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:58:00.458327 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:00.458259 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5c055d9-c060-41ba-a4ac-c14278cdb1d5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b5c055d9-c060-41ba-a4ac-c14278cdb1d5" (UID: "b5c055d9-c060-41ba-a4ac-c14278cdb1d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 22:58:00.557124 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:00.557089 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfvpv\" (UniqueName: \"kubernetes.io/projected/b5c055d9-c060-41ba-a4ac-c14278cdb1d5-kube-api-access-dfvpv\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:58:00.557124 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:00.557117 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c055d9-c060-41ba-a4ac-c14278cdb1d5-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:58:00.557124 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:00.557127 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5c055d9-c060-41ba-a4ac-c14278cdb1d5-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:58:01.267249 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:01.267224 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t" Jul 14 22:58:01.267660 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:01.267265 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t" event={"ID":"b5c055d9-c060-41ba-a4ac-c14278cdb1d5","Type":"ContainerDied","Data":"a0d646969f7953f8f4dfe9b7eb8961799995ddb5b75a70f7b51b9bcce969de19"} Jul 14 22:58:01.267660 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:01.267324 2550 scope.go:117] "RemoveContainer" containerID="d5e7c89d0a876e0236d1694a3d799e9eba1c8f9791439993e05975649bf0a4ac" Jul 14 22:58:01.296912 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:01.296883 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t"] Jul 14 22:58:01.301278 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:01.301251 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7ab9a6-d8fb65c6b-6qj9t"] Jul 14 22:58:02.195518 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:02.195487 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5c055d9-c060-41ba-a4ac-c14278cdb1d5" path="/var/lib/kubelet/pods/b5c055d9-c060-41ba-a4ac-c14278cdb1d5/volumes" Jul 14 22:58:04.733263 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:04.733230 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:58:10.911642 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:10.911610 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:58:10.972089 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:10.972059 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:58:15.505270 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:15.505237 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:58:15.745532 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:15.745495 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:58:15.891764 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:15.891685 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-546296-dthck"] Jul 14 22:58:15.892034 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:15.892021 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5c055d9-c060-41ba-a4ac-c14278cdb1d5" containerName="maas-api" Jul 14 22:58:15.892079 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:15.892036 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5c055d9-c060-41ba-a4ac-c14278cdb1d5" containerName="maas-api" Jul 14 22:58:15.892079 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:15.892053 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bde58c9-9bdd-4fed-b157-db3abbcc5152" containerName="revoke-keys" Jul 14 22:58:15.892079 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:15.892061 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bde58c9-9bdd-4fed-b157-db3abbcc5152" containerName="revoke-keys" Jul 14 22:58:15.892176 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:15.892123 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bde58c9-9bdd-4fed-b157-db3abbcc5152" containerName="revoke-keys" Jul 14 22:58:15.892176 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:15.892137 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5c055d9-c060-41ba-a4ac-c14278cdb1d5" containerName="maas-api" Jul 14 22:58:15.895136 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:15.895120 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-546296-dthck" Jul 14 22:58:15.897714 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:15.897683 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mj4xg\"" Jul 14 22:58:15.902434 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:15.902402 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-546296-dthck"] Jul 14 22:58:15.994501 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:15.994465 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/324c1f0d-b9a5-4f0a-8fa8-a069f95dc62c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-546296-dthck\" (UID: \"324c1f0d-b9a5-4f0a-8fa8-a069f95dc62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-546296-dthck" Jul 14 22:58:16.095783 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:16.095729 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/324c1f0d-b9a5-4f0a-8fa8-a069f95dc62c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-546296-dthck\" (UID: \"324c1f0d-b9a5-4f0a-8fa8-a069f95dc62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-546296-dthck" Jul 14 22:58:16.096395 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:16.096376 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/324c1f0d-b9a5-4f0a-8fa8-a069f95dc62c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-546296-dthck\" (UID: \"324c1f0d-b9a5-4f0a-8fa8-a069f95dc62c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-546296-dthck" Jul 14 22:58:16.206908 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:16.206880 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-546296-dthck" Jul 14 22:58:16.338099 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:16.338075 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-546296-dthck"] Jul 14 22:58:16.340119 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:58:16.340088 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod324c1f0d_b9a5_4f0a_8fa8_a069f95dc62c.slice/crio-d563973e794ec59253ebed262f5763c7cb1380121b5e7cfdff3e8c382c8d5a26 WatchSource:0}: Error finding container d563973e794ec59253ebed262f5763c7cb1380121b5e7cfdff3e8c382c8d5a26: Status 404 returned error can't find the container with id d563973e794ec59253ebed262f5763c7cb1380121b5e7cfdff3e8c382c8d5a26 Jul 14 22:58:16.554186 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:16.554094 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:58:16.760496 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:16.760455 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:58:17.352073 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:17.351994 2550 generic.go:356] "Generic (PLEG): container finished" podID="324c1f0d-b9a5-4f0a-8fa8-a069f95dc62c" containerID="832812b3b2a388b2cb713a250a502683c490b14f9e9f25807819c0a36456a54c" exitCode=0 Jul 14 22:58:17.352209 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:17.352077 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-546296-dthck" event={"ID":"324c1f0d-b9a5-4f0a-8fa8-a069f95dc62c","Type":"ContainerDied","Data":"832812b3b2a388b2cb713a250a502683c490b14f9e9f25807819c0a36456a54c"} Jul 14 22:58:17.352209 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:17.352134 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-546296-dthck" event={"ID":"324c1f0d-b9a5-4f0a-8fa8-a069f95dc62c","Type":"ContainerStarted","Data":"d563973e794ec59253ebed262f5763c7cb1380121b5e7cfdff3e8c382c8d5a26"} Jul 14 22:58:18.494880 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:18.494843 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-546296-dthck" Jul 14 22:58:18.619269 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:18.619184 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/324c1f0d-b9a5-4f0a-8fa8-a069f95dc62c-maas-api-service-ca\") pod \"324c1f0d-b9a5-4f0a-8fa8-a069f95dc62c\" (UID: \"324c1f0d-b9a5-4f0a-8fa8-a069f95dc62c\") " Jul 14 22:58:18.619515 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:18.619494 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/324c1f0d-b9a5-4f0a-8fa8-a069f95dc62c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "324c1f0d-b9a5-4f0a-8fa8-a069f95dc62c" (UID: "324c1f0d-b9a5-4f0a-8fa8-a069f95dc62c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:58:18.719834 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:18.719796 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/324c1f0d-b9a5-4f0a-8fa8-a069f95dc62c-maas-api-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:58:19.363609 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:19.363580 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-546296-dthck" Jul 14 22:58:19.363757 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:19.363608 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-546296-dthck" event={"ID":"324c1f0d-b9a5-4f0a-8fa8-a069f95dc62c","Type":"ContainerDied","Data":"d563973e794ec59253ebed262f5763c7cb1380121b5e7cfdff3e8c382c8d5a26"} Jul 14 22:58:19.363757 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:19.363655 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d563973e794ec59253ebed262f5763c7cb1380121b5e7cfdff3e8c382c8d5a26" Jul 14 22:58:26.434331 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:26.434297 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-546296-684bd79c45-sf4gn"] Jul 14 22:58:26.434804 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:26.434479 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-546296-684bd79c45-sf4gn" podUID="312722d1-879d-434c-8447-e4ae4bf42adf" containerName="maas-api" containerID="cri-o://779a1a9c851c7b800a072f38dc2d2a3d7086eb311c13a8aaf0bb0ce242729cc1" gracePeriod=30 Jul 14 22:58:26.456804 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:26.456781 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:58:26.682810 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:26.682788 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-546296-684bd79c45-sf4gn" Jul 14 22:58:26.786478 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:26.786381 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/312722d1-879d-434c-8447-e4ae4bf42adf-maas-api-tls\") pod \"312722d1-879d-434c-8447-e4ae4bf42adf\" (UID: \"312722d1-879d-434c-8447-e4ae4bf42adf\") " Jul 14 22:58:26.786478 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:26.786432 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/312722d1-879d-434c-8447-e4ae4bf42adf-openshift-service-ca\") pod \"312722d1-879d-434c-8447-e4ae4bf42adf\" (UID: \"312722d1-879d-434c-8447-e4ae4bf42adf\") " Jul 14 22:58:26.786478 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:26.786451 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8f8rh\" (UniqueName: \"kubernetes.io/projected/312722d1-879d-434c-8447-e4ae4bf42adf-kube-api-access-8f8rh\") pod \"312722d1-879d-434c-8447-e4ae4bf42adf\" (UID: \"312722d1-879d-434c-8447-e4ae4bf42adf\") " Jul 14 22:58:26.786849 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:26.786824 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/312722d1-879d-434c-8447-e4ae4bf42adf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "312722d1-879d-434c-8447-e4ae4bf42adf" (UID: "312722d1-879d-434c-8447-e4ae4bf42adf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 22:58:26.788585 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:26.788536 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/312722d1-879d-434c-8447-e4ae4bf42adf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "312722d1-879d-434c-8447-e4ae4bf42adf" (UID: "312722d1-879d-434c-8447-e4ae4bf42adf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 22:58:26.788679 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:26.788628 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/312722d1-879d-434c-8447-e4ae4bf42adf-kube-api-access-8f8rh" (OuterVolumeSpecName: "kube-api-access-8f8rh") pod "312722d1-879d-434c-8447-e4ae4bf42adf" (UID: "312722d1-879d-434c-8447-e4ae4bf42adf"). InnerVolumeSpecName "kube-api-access-8f8rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 22:58:26.888013 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:26.887973 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/312722d1-879d-434c-8447-e4ae4bf42adf-maas-api-tls\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:58:26.888013 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:26.888006 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/312722d1-879d-434c-8447-e4ae4bf42adf-openshift-service-ca\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:58:26.888013 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:26.888015 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f8rh\" (UniqueName: \"kubernetes.io/projected/312722d1-879d-434c-8447-e4ae4bf42adf-kube-api-access-8f8rh\") on node \"ip-10-0-133-118.ec2.internal\" DevicePath \"\"" Jul 14 22:58:27.405819 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:27.405780 2550 generic.go:356] "Generic (PLEG): container finished" podID="312722d1-879d-434c-8447-e4ae4bf42adf" containerID="779a1a9c851c7b800a072f38dc2d2a3d7086eb311c13a8aaf0bb0ce242729cc1" exitCode=0 Jul 14 22:58:27.405986 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:27.405826 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-546296-684bd79c45-sf4gn" Jul 14 22:58:27.405986 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:27.405854 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-546296-684bd79c45-sf4gn" event={"ID":"312722d1-879d-434c-8447-e4ae4bf42adf","Type":"ContainerDied","Data":"779a1a9c851c7b800a072f38dc2d2a3d7086eb311c13a8aaf0bb0ce242729cc1"} Jul 14 22:58:27.405986 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:27.405905 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-546296-684bd79c45-sf4gn" event={"ID":"312722d1-879d-434c-8447-e4ae4bf42adf","Type":"ContainerDied","Data":"6e89769124fb1fd4c1fe4b27a0608dabe1c1e86ddeac6e1f72adc36462bb5d5c"} Jul 14 22:58:27.405986 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:27.405934 2550 scope.go:117] "RemoveContainer" containerID="779a1a9c851c7b800a072f38dc2d2a3d7086eb311c13a8aaf0bb0ce242729cc1" Jul 14 22:58:27.415357 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:27.415336 2550 scope.go:117] "RemoveContainer" containerID="779a1a9c851c7b800a072f38dc2d2a3d7086eb311c13a8aaf0bb0ce242729cc1" Jul 14 22:58:27.415622 ip-10-0-133-118 kubenswrapper[2550]: E0714 22:58:27.415602 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"779a1a9c851c7b800a072f38dc2d2a3d7086eb311c13a8aaf0bb0ce242729cc1\": container with ID starting with 779a1a9c851c7b800a072f38dc2d2a3d7086eb311c13a8aaf0bb0ce242729cc1 not found: ID does not exist" containerID="779a1a9c851c7b800a072f38dc2d2a3d7086eb311c13a8aaf0bb0ce242729cc1" Jul 14 22:58:27.415670 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:27.415631 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"779a1a9c851c7b800a072f38dc2d2a3d7086eb311c13a8aaf0bb0ce242729cc1"} err="failed to get container status \"779a1a9c851c7b800a072f38dc2d2a3d7086eb311c13a8aaf0bb0ce242729cc1\": rpc error: code = NotFound desc = could not find container \"779a1a9c851c7b800a072f38dc2d2a3d7086eb311c13a8aaf0bb0ce242729cc1\": container with ID starting with 779a1a9c851c7b800a072f38dc2d2a3d7086eb311c13a8aaf0bb0ce242729cc1 not found: ID does not exist" Jul 14 22:58:27.428433 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:27.428410 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-546296-684bd79c45-sf4gn"] Jul 14 22:58:27.433214 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:27.433183 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-546296-684bd79c45-sf4gn"] Jul 14 22:58:28.195450 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:28.195414 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="312722d1-879d-434c-8447-e4ae4bf42adf" path="/var/lib/kubelet/pods/312722d1-879d-434c-8447-e4ae4bf42adf/volumes" Jul 14 22:58:30.982123 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:30.982092 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:58:37.102617 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:37.102541 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:58:37.203613 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:37.203544 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:58:41.712683 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:41.712648 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:58:41.968267 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:41.968183 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:58:48.325200 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:48.325175 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86c5998494-g2p5d_5ccad00f-8074-4fb0-bfa9-c89d07edede6/copy-manifests/0.log" Jul 14 22:58:48.343794 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:48.343767 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86c5998494-g2p5d_5ccad00f-8074-4fb0-bfa9-c89d07edede6/manager/0.log" Jul 14 22:58:49.008977 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:49.008947 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-d559f64cb-bsjmz_4b3ca406-334e-47c4-86d9-bd27fffc3a08/maas-api/0.log" Jul 14 22:58:49.612749 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:49.612659 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-971ed6-2f758_5d81d81d-e70c-4251-9420-056f247d175b/revoke-keys/0.log" Jul 14 22:58:49.732901 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:49.732877 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-7215d1-nhknm_9f5bce7a-aa7e-42ad-90a2-3476475017f2/revoke-keys/0.log" Jul 14 22:58:50.098769 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:50.098736 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-cb7974-tv2d8_b5733856-6a7b-4ccf-9289-fdd9e2497bdb/revoke-keys/0.log" Jul 14 22:58:50.597839 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:50.597808 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-7ab9a6-v2nnd_5bde58c9-9bdd-4fed-b157-db3abbcc5152/revoke-keys/0.log" Jul 14 22:58:50.721962 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:50.721937 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-546296-dthck_324c1f0d-b9a5-4f0a-8fa8-a069f95dc62c/revoke-keys/0.log" Jul 14 22:58:51.685000 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:51.684971 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6_17b5cd52-9924-4249-ad92-3d80b85ac197/util/0.log" Jul 14 22:58:51.692449 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:51.692418 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6_17b5cd52-9924-4249-ad92-3d80b85ac197/pull/0.log" Jul 14 22:58:51.701485 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:51.701461 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6_17b5cd52-9924-4249-ad92-3d80b85ac197/extract/0.log" Jul 14 22:58:51.828094 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:51.828069 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx_34d9904f-1e3b-478a-bde9-9a1967ee85b5/util/0.log" Jul 14 22:58:51.835206 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:51.835184 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx_34d9904f-1e3b-478a-bde9-9a1967ee85b5/pull/0.log" Jul 14 22:58:51.842506 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:51.842483 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx_34d9904f-1e3b-478a-bde9-9a1967ee85b5/extract/0.log" Jul 14 22:58:51.963442 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:51.963362 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m_2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf/extract/0.log" Jul 14 22:58:51.970179 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:51.970158 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m_2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf/util/0.log" Jul 14 22:58:51.977004 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:51.976968 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m_2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf/pull/0.log" Jul 14 22:58:52.101689 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:52.101661 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c_fa6b4ac5-8086-4b39-98cb-11db6e876fd0/util/0.log" Jul 14 22:58:52.108621 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:52.108600 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c_fa6b4ac5-8086-4b39-98cb-11db6e876fd0/pull/0.log" Jul 14 22:58:52.115768 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:52.115744 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c_fa6b4ac5-8086-4b39-98cb-11db6e876fd0/extract/0.log" Jul 14 22:58:53.165267 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:53.165234 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-g86p5_d4311840-f902-47c1-bdb2-a106214e546e/manager/0.log" Jul 14 22:58:53.674133 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:53.674104 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-484t2_140164c5-c024-4fb4-91da-9ff07e0c9406/discovery/0.log" Jul 14 22:58:54.907726 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:54.907698 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-n78xj_f451ea84-ed76-4751-87d0-b341564aec23/main/0.log" Jul 14 22:58:55.271104 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:55.271074 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-jhjbl_787afbc7-1a70-4126-a7fb-c337b468ea7b/main/0.log" Jul 14 22:58:59.537352 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:59.537323 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:58:59.639278 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:58:59.639246 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" limit=1024 Jul 14 22:59:02.239958 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:02.239920 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6g6bp_ce5ffbb6-99ac-4817-acd9-f6cd4577f724/global-pull-secret-syncer/0.log" Jul 14 22:59:02.383727 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:02.383699 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gmmnx_3a228cf3-be18-4e92-bc1b-0045d1b2c823/konnectivity-agent/0.log" Jul 14 22:59:02.516367 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:02.516284 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-118.ec2.internal_da6abbb4b54c9a3e9a35bc6fff12e4c3/haproxy/0.log" Jul 14 22:59:06.285168 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:06.285132 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6_17b5cd52-9924-4249-ad92-3d80b85ac197/extract/0.log" Jul 14 22:59:06.307231 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:06.307206 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6_17b5cd52-9924-4249-ad92-3d80b85ac197/util/0.log" Jul 14 22:59:06.335673 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:06.335647 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759dgvs6_17b5cd52-9924-4249-ad92-3d80b85ac197/pull/0.log" Jul 14 22:59:06.392644 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:06.392618 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx_34d9904f-1e3b-478a-bde9-9a1967ee85b5/extract/0.log" Jul 14 22:59:06.453528 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:06.453500 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx_34d9904f-1e3b-478a-bde9-9a1967ee85b5/util/0.log" Jul 14 22:59:06.529969 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:06.529936 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rdx_34d9904f-1e3b-478a-bde9-9a1967ee85b5/pull/0.log" Jul 14 22:59:06.602227 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:06.602148 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m_2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf/extract/0.log" Jul 14 22:59:06.627559 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:06.627532 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m_2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf/util/0.log" Jul 14 22:59:06.658452 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:06.658426 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jng8m_2c7d18a6-3c83-4a6d-9bb0-9b34eccdd1bf/pull/0.log" Jul 14 22:59:06.687776 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:06.687750 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c_fa6b4ac5-8086-4b39-98cb-11db6e876fd0/extract/0.log" Jul 14 22:59:06.712799 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:06.712779 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c_fa6b4ac5-8086-4b39-98cb-11db6e876fd0/util/0.log" Jul 14 22:59:06.737881 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:06.737855 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1b2j2c_fa6b4ac5-8086-4b39-98cb-11db6e876fd0/pull/0.log" Jul 14 22:59:07.680044 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:07.680014 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-g86p5_d4311840-f902-47c1-bdb2-a106214e546e/manager/0.log" Jul 14 22:59:09.479220 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:09.479185 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-462d8_af11146f-0b62-492b-b73a-440e2c69a95b/kube-state-metrics/0.log" Jul 14 22:59:09.497112 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:09.497092 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-462d8_af11146f-0b62-492b-b73a-440e2c69a95b/kube-rbac-proxy-main/0.log" Jul 14 22:59:09.526616 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:09.526594 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-462d8_af11146f-0b62-492b-b73a-440e2c69a95b/kube-rbac-proxy-self/0.log" Jul 14 22:59:09.556887 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:09.556866 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6977c64d5f-44v55_079986c7-2809-4036-9fe3-3587d5902d62/metrics-server/0.log" Jul 14 22:59:09.587282 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:09.587263 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-r856d_5cc10e6c-47f8-4b27-9dd4-0554cc6ee905/monitoring-plugin/0.log" Jul 14 22:59:09.746034 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:09.745951 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9bglr_1d845a84-9484-41d0-9373-27ede38c8f5f/node-exporter/0.log" Jul 14 22:59:09.770863 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:09.770839 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9bglr_1d845a84-9484-41d0-9373-27ede38c8f5f/kube-rbac-proxy/0.log" Jul 14 22:59:09.795851 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:09.795833 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9bglr_1d845a84-9484-41d0-9373-27ede38c8f5f/init-textfile/0.log" Jul 14 22:59:09.912728 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:09.912702 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-t66mm_c770b0db-7fa6-4090-826a-412b9d542753/kube-rbac-proxy-main/0.log" Jul 14 22:59:09.942110 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:09.942083 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-t66mm_c770b0db-7fa6-4090-826a-412b9d542753/kube-rbac-proxy-self/0.log" Jul 14 22:59:09.978109 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:09.978082 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-t66mm_c770b0db-7fa6-4090-826a-412b9d542753/openshift-state-metrics/0.log" Jul 14 22:59:12.719695 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:12.719656 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9nq2f/perf-node-gather-daemonset-bvt9v"] Jul 14 22:59:12.720172 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:12.720153 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="312722d1-879d-434c-8447-e4ae4bf42adf" containerName="maas-api" Jul 14 22:59:12.720246 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:12.720176 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="312722d1-879d-434c-8447-e4ae4bf42adf" containerName="maas-api" Jul 14 22:59:12.720246 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:12.720200 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="324c1f0d-b9a5-4f0a-8fa8-a069f95dc62c" containerName="revoke-keys" Jul 14 22:59:12.720246 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:12.720209 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="324c1f0d-b9a5-4f0a-8fa8-a069f95dc62c" containerName="revoke-keys" Jul 14 22:59:12.720398 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:12.720302 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="312722d1-879d-434c-8447-e4ae4bf42adf" containerName="maas-api" Jul 14 22:59:12.720398 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:12.720316 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="324c1f0d-b9a5-4f0a-8fa8-a069f95dc62c" containerName="revoke-keys" Jul 14 22:59:12.723493 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:12.723459 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-bvt9v" Jul 14 22:59:12.725804 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:12.725783 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9nq2f\"/\"kube-root-ca.crt\"" Jul 14 22:59:12.726818 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:12.726799 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9nq2f\"/\"openshift-service-ca.crt\"" Jul 14 22:59:12.726920 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:12.726845 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9nq2f\"/\"default-dockercfg-zqhpv\"" Jul 14 22:59:12.730773 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:12.730614 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9nq2f/perf-node-gather-daemonset-bvt9v"] Jul 14 22:59:12.771319 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:12.771295 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33d888d1-2052-404c-acf0-f3f93ddcc211-lib-modules\") pod \"perf-node-gather-daemonset-bvt9v\" (UID: \"33d888d1-2052-404c-acf0-f3f93ddcc211\") " pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-bvt9v" Jul 14 22:59:12.771441 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:12.771339 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/33d888d1-2052-404c-acf0-f3f93ddcc211-podres\") pod \"perf-node-gather-daemonset-bvt9v\" (UID: \"33d888d1-2052-404c-acf0-f3f93ddcc211\") " pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-bvt9v" Jul 14 22:59:12.771441 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:12.771361 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n2z9\" (UniqueName: \"kubernetes.io/projected/33d888d1-2052-404c-acf0-f3f93ddcc211-kube-api-access-2n2z9\") pod \"perf-node-gather-daemonset-bvt9v\" (UID: \"33d888d1-2052-404c-acf0-f3f93ddcc211\") " pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-bvt9v" Jul 14 22:59:12.771516 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:12.771439 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33d888d1-2052-404c-acf0-f3f93ddcc211-sys\") pod \"perf-node-gather-daemonset-bvt9v\" (UID: \"33d888d1-2052-404c-acf0-f3f93ddcc211\") " pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-bvt9v" Jul 14 22:59:12.771516 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:12.771455 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/33d888d1-2052-404c-acf0-f3f93ddcc211-proc\") pod \"perf-node-gather-daemonset-bvt9v\" (UID: \"33d888d1-2052-404c-acf0-f3f93ddcc211\") " pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-bvt9v" Jul 14 22:59:12.872527 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:12.872495 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/33d888d1-2052-404c-acf0-f3f93ddcc211-podres\") pod \"perf-node-gather-daemonset-bvt9v\" (UID: \"33d888d1-2052-404c-acf0-f3f93ddcc211\") " pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-bvt9v" Jul 14 22:59:12.872708 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:12.872532 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n2z9\" (UniqueName: \"kubernetes.io/projected/33d888d1-2052-404c-acf0-f3f93ddcc211-kube-api-access-2n2z9\") pod \"perf-node-gather-daemonset-bvt9v\" (UID: \"33d888d1-2052-404c-acf0-f3f93ddcc211\") " pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-bvt9v" Jul 14 22:59:12.872708 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:12.872602 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33d888d1-2052-404c-acf0-f3f93ddcc211-sys\") pod \"perf-node-gather-daemonset-bvt9v\" (UID: \"33d888d1-2052-404c-acf0-f3f93ddcc211\") " pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-bvt9v" Jul 14 22:59:12.872708 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:12.872620 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/33d888d1-2052-404c-acf0-f3f93ddcc211-proc\") pod \"perf-node-gather-daemonset-bvt9v\" (UID: \"33d888d1-2052-404c-acf0-f3f93ddcc211\") " pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-bvt9v" Jul 14 22:59:12.872708 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:12.872654 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33d888d1-2052-404c-acf0-f3f93ddcc211-lib-modules\") pod \"perf-node-gather-daemonset-bvt9v\" (UID: \"33d888d1-2052-404c-acf0-f3f93ddcc211\") " pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-bvt9v" Jul 14 22:59:12.872708 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:12.872675 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/33d888d1-2052-404c-acf0-f3f93ddcc211-podres\") pod \"perf-node-gather-daemonset-bvt9v\" (UID: \"33d888d1-2052-404c-acf0-f3f93ddcc211\") " pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-bvt9v" Jul 14 22:59:12.872911 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:12.872725 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33d888d1-2052-404c-acf0-f3f93ddcc211-sys\") pod \"perf-node-gather-daemonset-bvt9v\" (UID: \"33d888d1-2052-404c-acf0-f3f93ddcc211\") " pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-bvt9v" Jul 14 22:59:12.872911 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:12.872738 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/33d888d1-2052-404c-acf0-f3f93ddcc211-proc\") pod \"perf-node-gather-daemonset-bvt9v\" (UID: \"33d888d1-2052-404c-acf0-f3f93ddcc211\") " pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-bvt9v" Jul 14 22:59:12.872911 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:12.872766 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33d888d1-2052-404c-acf0-f3f93ddcc211-lib-modules\") pod \"perf-node-gather-daemonset-bvt9v\" (UID: \"33d888d1-2052-404c-acf0-f3f93ddcc211\") " pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-bvt9v" Jul 14 22:59:12.880969 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:12.880947 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n2z9\" (UniqueName: \"kubernetes.io/projected/33d888d1-2052-404c-acf0-f3f93ddcc211-kube-api-access-2n2z9\") pod \"perf-node-gather-daemonset-bvt9v\" (UID: \"33d888d1-2052-404c-acf0-f3f93ddcc211\") " pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-bvt9v" Jul 14 22:59:13.033795 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:13.033712 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-bvt9v" Jul 14 22:59:13.071279 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:13.071252 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7d5d545ff5-9t7ht_9cf8274b-6860-426f-98e9-a2af0eb5d8ed/console/0.log" Jul 14 22:59:13.141328 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:13.141287 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-jghjl_1337dea6-26b7-4707-a955-1fc73f898523/download-server/0.log" Jul 14 22:59:13.166568 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:13.166533 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9nq2f/perf-node-gather-daemonset-bvt9v"] Jul 14 22:59:13.168134 ip-10-0-133-118 kubenswrapper[2550]: W0714 22:59:13.168108 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod33d888d1_2052_404c_acf0_f3f93ddcc211.slice/crio-db1e7098d23e2337b67bf904b315b2436b4a4d6b176e20502d1507a73cd41c40 WatchSource:0}: Error finding container db1e7098d23e2337b67bf904b315b2436b4a4d6b176e20502d1507a73cd41c40: Status 404 returned error can't find the container with id db1e7098d23e2337b67bf904b315b2436b4a4d6b176e20502d1507a73cd41c40 Jul 14 22:59:13.643143 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:13.643113 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-bvt9v" event={"ID":"33d888d1-2052-404c-acf0-f3f93ddcc211","Type":"ContainerStarted","Data":"ea40ec216c386e6e18c4268ef490b841ec74bad025146c222034e499f43ebddd"} Jul 14 22:59:13.643335 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:13.643170 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-bvt9v" event={"ID":"33d888d1-2052-404c-acf0-f3f93ddcc211","Type":"ContainerStarted","Data":"db1e7098d23e2337b67bf904b315b2436b4a4d6b176e20502d1507a73cd41c40"} Jul 14 22:59:13.643335 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:13.643209 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-bvt9v" Jul 14 22:59:13.659469 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:13.659417 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-bvt9v" podStartSLOduration=1.659401748 podStartE2EDuration="1.659401748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:59:13.659166197 +0000 UTC m=+2984.018312637" watchObservedRunningTime="2026-07-14 22:59:13.659401748 +0000 UTC m=+2984.018548184" Jul 14 22:59:14.982357 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:14.982322 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dktnm_7d5e9f87-1d85-41a5-af29-5e7586fe23f9/dns/0.log" Jul 14 22:59:15.005272 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:15.005245 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dktnm_7d5e9f87-1d85-41a5-af29-5e7586fe23f9/kube-rbac-proxy/0.log" Jul 14 22:59:15.031190 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:15.031164 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6tt4w_1159189b-384f-4107-807e-250f243409df/dns-node-resolver/0.log" Jul 14 22:59:15.636240 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:15.636207 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-794ffd4775-6nwdp_f1f23e43-cf64-40e8-9dd3-4ac6ad774dc3/registry/0.log" Jul 14 22:59:15.687868 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:15.687837 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dj68k_8c265d22-e8f9-4516-aaf7-b274420c1b23/node-ca/0.log" Jul 14 22:59:16.920507 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:16.920477 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-484t2_140164c5-c024-4fb4-91da-9ff07e0c9406/discovery/0.log" Jul 14 22:59:17.868313 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:17.868265 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2jnrx_ae93f9b7-e76d-4f57-b149-b726ed58218a/serve-healthcheck-canary/0.log" Jul 14 22:59:18.488947 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:18.488919 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b4h5c_9aa0f4fd-943b-4367-9e82-1190d2d4e1c8/kube-rbac-proxy/0.log" Jul 14 22:59:18.510435 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:18.510413 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b4h5c_9aa0f4fd-943b-4367-9e82-1190d2d4e1c8/exporter/0.log" Jul 14 22:59:18.533377 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:18.533355 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b4h5c_9aa0f4fd-943b-4367-9e82-1190d2d4e1c8/extractor/0.log" Jul 14 22:59:19.658099 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:19.658074 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9nq2f/perf-node-gather-daemonset-bvt9v" Jul 14 22:59:21.478486 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:21.478416 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86c5998494-g2p5d_5ccad00f-8074-4fb0-bfa9-c89d07edede6/manager/0.log" Jul 14 22:59:21.496867 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:21.496847 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86c5998494-g2p5d_5ccad00f-8074-4fb0-bfa9-c89d07edede6/copy-manifests/0.log" Jul 14 22:59:22.894129 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:22.894050 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-484t2_140164c5-c024-4fb4-91da-9ff07e0c9406/discovery/0.log" Jul 14 22:59:22.994995 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:22.994968 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-66545d98-lq6fq_2312297c-7f42-47c9-b0cf-a1474ae6e6d4/manager/0.log" Jul 14 22:59:23.021486 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:23.021462 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-ndz9d_3d25f945-dabe-42ac-af03-f36dfbd0e213/openshift-lws-operator/0.log" Jul 14 22:59:24.572426 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:24.572390 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-484t2_140164c5-c024-4fb4-91da-9ff07e0c9406/discovery/0.log" Jul 14 22:59:29.722620 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:29.722595 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l9qrh_d5f7e8bb-6df3-4c30-84c7-4e30f116545a/kube-multus-additional-cni-plugins/0.log" Jul 14 22:59:29.743614 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:29.743589 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l9qrh_d5f7e8bb-6df3-4c30-84c7-4e30f116545a/egress-router-binary-copy/0.log" Jul 14 22:59:29.764345 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:29.764323 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l9qrh_d5f7e8bb-6df3-4c30-84c7-4e30f116545a/cni-plugins/0.log" Jul 14 22:59:29.784889 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:29.784864 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l9qrh_d5f7e8bb-6df3-4c30-84c7-4e30f116545a/bond-cni-plugin/0.log" Jul 14 22:59:29.806007 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:29.805982 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l9qrh_d5f7e8bb-6df3-4c30-84c7-4e30f116545a/routeoverride-cni/0.log" Jul 14 22:59:29.827687 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:29.827670 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l9qrh_d5f7e8bb-6df3-4c30-84c7-4e30f116545a/whereabouts-cni-bincopy/0.log" Jul 14 22:59:29.848446 ip-10-0-133-118 kubenswrapper[2550]: I0714 22:59:29.848414 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l9qrh_d5f7e8bb-6df3-4c30-84c7-4e30f116545a/whereabouts-cni/0.log"