Jul 10 18:56:40.529893 ip-10-0-138-213 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 10 18:56:40.529907 ip-10-0-138-213 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 10 18:56:40.529917 ip-10-0-138-213 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 10 18:56:40.530245 ip-10-0-138-213 systemd[1]: Failed to start Kubernetes Kubelet. Jul 10 18:56:51.797883 ip-10-0-138-213 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 10 18:56:51.797900 ip-10-0-138-213 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot ecd09261da914151bbe95ed837c9806b -- Jul 10 18:58:28.540773 ip-10-0-138-213 systemd[1]: Starting Kubernetes Kubelet... Jul 10 18:58:28.999627 ip-10-0-138-213 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 18:58:28.999627 ip-10-0-138-213 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 18:58:28.999627 ip-10-0-138-213 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 18:58:28.999627 ip-10-0-138-213 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 18:58:28.999627 ip-10-0-138-213 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 18:58:28.999627 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:28.996330 2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 18:58:29.005188 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002403 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 18:58:29.005188 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002417 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 18:58:29.005188 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002420 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 18:58:29.005188 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002424 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 18:58:29.005188 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002427 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 18:58:29.005188 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002429 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 18:58:29.005188 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002432 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 18:58:29.005188 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002436 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 18:58:29.005188 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002451 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 18:58:29.005188 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002454 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 18:58:29.005188 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002457 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 18:58:29.005188 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002460 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 18:58:29.005188 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002463 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 18:58:29.005188 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002471 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 18:58:29.005188 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002474 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 18:58:29.005188 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002477 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 18:58:29.005188 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002480 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 18:58:29.005188 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002483 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 18:58:29.005188 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002485 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 18:58:29.005188 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002488 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 18:58:29.008949 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002491 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 18:58:29.008949 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002493 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 18:58:29.008949 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002496 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 18:58:29.008949 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002498 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 18:58:29.008949 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002501 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 18:58:29.008949 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002504 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 18:58:29.008949 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002507 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 18:58:29.008949 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002509 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 18:58:29.008949 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002512 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 18:58:29.008949 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002515 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 18:58:29.008949 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002518 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 18:58:29.008949 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002520 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 18:58:29.008949 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002523 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 18:58:29.008949 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002525 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 18:58:29.008949 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002528 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 18:58:29.008949 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002531 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 18:58:29.008949 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002533 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 18:58:29.008949 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002537 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 18:58:29.008949 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002540 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 18:58:29.008949 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002543 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 18:58:29.011588 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002546 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 18:58:29.011588 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002548 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 18:58:29.011588 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002551 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 18:58:29.011588 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002553 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 18:58:29.011588 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002556 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 18:58:29.011588 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002558 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 18:58:29.011588 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002561 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 18:58:29.011588 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002563 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 18:58:29.011588 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002565 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 18:58:29.011588 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002568 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 18:58:29.011588 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002570 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 18:58:29.011588 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002573 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 18:58:29.011588 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002575 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 18:58:29.011588 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002578 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 18:58:29.011588 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002580 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 18:58:29.011588 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002583 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 18:58:29.011588 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002585 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 18:58:29.011588 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002590 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 18:58:29.011588 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002594 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 18:58:29.014995 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002597 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 18:58:29.014995 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002601 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 18:58:29.014995 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002604 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 18:58:29.014995 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002608 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 18:58:29.014995 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002611 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 18:58:29.014995 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002613 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 18:58:29.014995 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002616 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 18:58:29.014995 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002618 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 18:58:29.014995 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002622 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 18:58:29.014995 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002625 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 18:58:29.014995 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002629 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 18:58:29.014995 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002634 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 18:58:29.014995 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002639 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 18:58:29.014995 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002642 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 18:58:29.014995 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002645 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 18:58:29.014995 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002647 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 18:58:29.014995 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002650 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 18:58:29.014995 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002652 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 18:58:29.017686 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002656 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 18:58:29.017686 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002659 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 18:58:29.017686 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002662 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 18:58:29.017686 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002664 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 18:58:29.017686 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002667 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 18:58:29.017686 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002670 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 18:58:29.017686 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002672 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 18:58:29.017686 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002675 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 18:58:29.017686 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002678 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 18:58:29.017686 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002680 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 18:58:29.017686 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002683 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 18:58:29.017686 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002686 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 18:58:29.017686 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002688 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 18:58:29.017686 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002691 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 18:58:29.017686 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002693 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 18:58:29.017686 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002696 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 18:58:29.017686 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002698 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 10 18:58:29.017686 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002701 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 18:58:29.017686 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002704 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 18:58:29.017686 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002706 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 18:58:29.021549 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002709 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 18:58:29.021549 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.002712 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 18:58:29.021549 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003433 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 18:58:29.021549 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003453 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 18:58:29.021549 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003457 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 18:58:29.021549 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003459 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 18:58:29.021549 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003462 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 18:58:29.021549 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003466 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 18:58:29.021549 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003469 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 18:58:29.021549 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003471 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 18:58:29.021549 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003474 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 18:58:29.021549 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003477 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 10 18:58:29.021549 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003480 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 18:58:29.021549 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003483 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 18:58:29.021549 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003486 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 18:58:29.021549 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003488 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 18:58:29.021549 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003491 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 18:58:29.021549 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003495 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 18:58:29.021549 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003499 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 18:58:29.021549 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003502 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 18:58:29.024933 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003505 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 18:58:29.024933 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003508 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 18:58:29.024933 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003511 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 18:58:29.024933 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003513 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 18:58:29.024933 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003516 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 18:58:29.024933 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003519 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 18:58:29.024933 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003523 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 18:58:29.024933 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003526 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 18:58:29.024933 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003529 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 18:58:29.024933 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003532 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 18:58:29.024933 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003535 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 18:58:29.024933 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003538 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 18:58:29.024933 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003543 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 18:58:29.024933 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003547 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 18:58:29.024933 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003550 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 18:58:29.024933 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003552 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 18:58:29.024933 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003555 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 18:58:29.024933 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003558 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 18:58:29.028850 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003561 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 18:58:29.028850 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003564 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 18:58:29.028850 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003566 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 18:58:29.028850 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003570 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 18:58:29.028850 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003573 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 18:58:29.028850 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003577 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 18:58:29.028850 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003580 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 18:58:29.028850 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003582 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 18:58:29.028850 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003585 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 18:58:29.028850 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003588 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 18:58:29.028850 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003592 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 18:58:29.028850 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003594 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 18:58:29.028850 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003597 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 18:58:29.028850 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003599 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 18:58:29.028850 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003602 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 18:58:29.028850 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003605 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 18:58:29.028850 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003607 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 18:58:29.028850 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003610 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 18:58:29.028850 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003613 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 18:58:29.028850 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003615 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 18:58:29.031591 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003619 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 18:58:29.031591 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003621 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 18:58:29.031591 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003624 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 18:58:29.031591 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003627 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 18:58:29.031591 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003629 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 18:58:29.031591 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003632 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 18:58:29.031591 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003635 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 18:58:29.031591 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003638 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 18:58:29.031591 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003640 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 18:58:29.031591 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003643 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 18:58:29.031591 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003646 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 18:58:29.031591 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003650 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 18:58:29.031591 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003652 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 18:58:29.031591 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003655 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 18:58:29.031591 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003657 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 18:58:29.031591 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003661 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 18:58:29.031591 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003663 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 18:58:29.031591 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003666 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 18:58:29.031591 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003668 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 18:58:29.031591 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003671 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 18:58:29.035495 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003673 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 18:58:29.035495 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003676 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 18:58:29.035495 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003679 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 18:58:29.035495 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003681 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 18:58:29.035495 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003684 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 18:58:29.035495 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003687 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 18:58:29.035495 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003690 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 18:58:29.035495 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003692 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 18:58:29.035495 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003695 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 18:58:29.035495 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003697 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 18:58:29.035495 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003700 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 18:58:29.035495 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003702 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 18:58:29.035495 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003705 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 18:58:29.035495 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003710 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 18:58:29.035495 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003713 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 18:58:29.035495 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003716 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 18:58:29.035495 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003719 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 18:58:29.035495 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003721 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 18:58:29.035495 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003724 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 18:58:29.035495 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003726 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 18:58:29.038605 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003729 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 18:58:29.038605 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003731 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 18:58:29.038605 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.003734 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 18:58:29.038605 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004741 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 18:58:29.038605 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004749 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 18:58:29.038605 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004752 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 18:58:29.038605 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004756 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 18:58:29.038605 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004764 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 18:58:29.038605 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004767 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 18:58:29.038605 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004770 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 18:58:29.038605 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004773 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 18:58:29.038605 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004775 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 18:58:29.038605 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004779 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 18:58:29.038605 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004783 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 18:58:29.038605 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004786 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 18:58:29.038605 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004789 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 18:58:29.038605 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004792 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 18:58:29.038605 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004795 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 18:58:29.038605 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004798 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 18:58:29.042077 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004800 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 18:58:29.042077 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004803 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 18:58:29.042077 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004806 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 18:58:29.042077 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004809 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 18:58:29.042077 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004811 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 18:58:29.042077 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004814 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 18:58:29.042077 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004819 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 18:58:29.042077 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004822 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 18:58:29.042077 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004825 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 18:58:29.042077 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004827 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 18:58:29.042077 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004830 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 18:58:29.042077 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004832 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 18:58:29.042077 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004835 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 18:58:29.042077 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004837 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 18:58:29.042077 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004840 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 18:58:29.042077 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004844 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 18:58:29.042077 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004847 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 18:58:29.042077 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004850 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 18:58:29.042077 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004852 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 18:58:29.042077 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004855 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 18:58:29.044746 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004857 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 18:58:29.044746 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004865 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 18:58:29.044746 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004868 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 18:58:29.044746 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004871 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 18:58:29.044746 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004874 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 18:58:29.044746 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004876 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 18:58:29.044746 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004879 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 18:58:29.044746 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004882 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 18:58:29.044746 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004884 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 18:58:29.044746 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004886 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 18:58:29.044746 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004889 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 18:58:29.044746 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004893 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 18:58:29.044746 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004896 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 18:58:29.044746 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004899 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 18:58:29.044746 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004902 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 18:58:29.044746 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004905 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 18:58:29.044746 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004908 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 18:58:29.044746 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004911 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 18:58:29.044746 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004914 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 18:58:29.044746 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004917 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 18:58:29.048407 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004919 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 18:58:29.048407 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004922 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 18:58:29.048407 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004924 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 18:58:29.048407 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004926 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 18:58:29.048407 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004929 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 18:58:29.048407 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004932 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 18:58:29.048407 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004934 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 18:58:29.048407 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004938 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 18:58:29.048407 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004941 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 18:58:29.048407 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004943 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 18:58:29.048407 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004946 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 18:58:29.048407 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004948 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 18:58:29.048407 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004951 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 18:58:29.048407 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004953 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 18:58:29.048407 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004962 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 18:58:29.048407 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004964 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 18:58:29.048407 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004967 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 18:58:29.048407 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004970 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 18:58:29.048407 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004972 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 18:58:29.073088 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004975 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 18:58:29.073088 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004978 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 18:58:29.073088 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004980 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 18:58:29.073088 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004983 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 10 18:58:29.073088 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004985 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 18:58:29.073088 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004988 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 18:58:29.073088 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004990 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 18:58:29.073088 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004992 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 18:58:29.073088 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004995 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 18:58:29.073088 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.004997 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 18:58:29.073088 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005000 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 18:58:29.073088 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005005 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 18:58:29.073088 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005009 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 18:58:29.073088 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005012 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 18:58:29.073088 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005020 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 18:58:29.073088 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005023 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 18:58:29.073088 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005025 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 18:58:29.073088 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005028 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 18:58:29.073088 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005030 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 18:58:29.073088 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005033 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 18:58:29.073315 ip-10-0-138-213 systemd[1]: Started Kubernetes Kubelet. Jul 10 18:58:29.094365 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005038 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 18:58:29.094365 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005040 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 18:58:29.094365 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005043 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 18:58:29.094365 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005046 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 18:58:29.094365 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005121 2560 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 18:58:29.094365 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005133 2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 18:58:29.094365 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005146 2560 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 18:58:29.094365 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005151 2560 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 18:58:29.094365 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005161 2560 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 18:58:29.094365 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005164 2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 18:58:29.094365 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005168 2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 18:58:29.094365 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005173 2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 18:58:29.094365 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005176 2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 18:58:29.094365 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005179 2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 18:58:29.094365 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005182 2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 18:58:29.094365 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005186 2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 18:58:29.094365 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005189 2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 18:58:29.094365 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005193 2560 flags.go:64] FLAG: --cgroup-root="" Jul 10 18:58:29.094365 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005196 2560 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 18:58:29.094365 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005199 2560 flags.go:64] FLAG: --client-ca-file="" Jul 10 18:58:29.094365 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005202 2560 flags.go:64] FLAG: --cloud-provider="external" Jul 10 18:58:29.094365 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005206 2560 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 18:58:29.094365 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005213 2560 flags.go:64] FLAG: --cluster-domain="" Jul 10 18:58:29.327003 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005218 2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 18:58:29.327003 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005221 2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 18:58:29.327003 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005224 2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 18:58:29.327003 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005227 2560 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 18:58:29.327003 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005231 2560 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 18:58:29.327003 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005234 2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 18:58:29.327003 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005237 2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 18:58:29.327003 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005240 2560 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 18:58:29.327003 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005243 2560 flags.go:64] FLAG: --contention-profiling="false" Jul 10 18:58:29.327003 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005247 2560 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 18:58:29.327003 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005250 2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 18:58:29.327003 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005253 2560 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 18:58:29.327003 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005256 2560 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 18:58:29.327003 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005260 2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 18:58:29.327003 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005263 2560 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 18:58:29.327003 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005266 2560 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 18:58:29.327003 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005269 2560 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 18:58:29.327003 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005272 2560 flags.go:64] FLAG: --enable-server="true" Jul 10 18:58:29.327003 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005281 2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 18:58:29.327003 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005287 2560 flags.go:64] FLAG: --event-burst="100" Jul 10 18:58:29.327003 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005290 2560 flags.go:64] FLAG: --event-qps="50" Jul 10 18:58:29.327003 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005293 2560 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 18:58:29.327003 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005296 2560 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 18:58:29.327003 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005299 2560 flags.go:64] FLAG: --eviction-hard="" Jul 10 18:58:29.327949 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005303 2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 18:58:29.327949 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005306 2560 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 18:58:29.327949 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005309 2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 18:58:29.327949 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005312 2560 flags.go:64] FLAG: --eviction-soft="" Jul 10 18:58:29.327949 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005315 2560 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 18:58:29.327949 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005318 2560 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 18:58:29.327949 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005321 2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 18:58:29.327949 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005324 2560 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 18:58:29.327949 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005328 2560 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 18:58:29.327949 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005331 2560 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 18:58:29.327949 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005334 2560 flags.go:64] FLAG: --feature-gates="" Jul 10 18:58:29.327949 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005338 2560 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 18:58:29.327949 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005341 2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 18:58:29.327949 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005344 2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 18:58:29.327949 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005347 2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 18:58:29.327949 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005351 2560 flags.go:64] FLAG: --healthz-port="10248" Jul 10 18:58:29.327949 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005354 2560 flags.go:64] FLAG: --help="false" Jul 10 18:58:29.327949 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005357 2560 flags.go:64] FLAG: --hostname-override="ip-10-0-138-213.ec2.internal" Jul 10 18:58:29.327949 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005360 2560 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 18:58:29.327949 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005363 2560 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 18:58:29.327949 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005366 2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 18:58:29.327949 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005369 2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 18:58:29.327949 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005372 2560 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 18:58:29.329183 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005375 2560 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 18:58:29.329183 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005378 2560 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 18:58:29.329183 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005380 2560 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 18:58:29.329183 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005383 2560 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 18:58:29.329183 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005392 2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 18:58:29.329183 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005395 2560 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 18:58:29.329183 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005398 2560 flags.go:64] FLAG: --kube-reserved="" Jul 10 18:58:29.329183 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005401 2560 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 18:58:29.329183 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005404 2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 18:58:29.329183 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005407 2560 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 18:58:29.329183 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005410 2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 18:58:29.329183 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005413 2560 flags.go:64] FLAG: --lock-file="" Jul 10 18:58:29.329183 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005416 2560 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 18:58:29.329183 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005419 2560 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 18:58:29.329183 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005422 2560 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 18:58:29.329183 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005428 2560 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 18:58:29.329183 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005431 2560 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 18:58:29.329183 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005435 2560 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 18:58:29.329183 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005452 2560 flags.go:64] FLAG: --logging-format="text" Jul 10 18:58:29.329183 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005455 2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 18:58:29.329183 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005459 2560 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 18:58:29.329183 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005462 2560 flags.go:64] FLAG: --manifest-url="" Jul 10 18:58:29.329183 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005464 2560 flags.go:64] FLAG: --manifest-url-header="" Jul 10 18:58:29.329183 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005469 2560 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 18:58:29.329183 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005472 2560 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 18:58:29.330369 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005476 2560 flags.go:64] FLAG: --max-pods="110" Jul 10 18:58:29.330369 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005479 2560 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 18:58:29.330369 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005482 2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 18:58:29.330369 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005485 2560 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 18:58:29.330369 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005488 2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 18:58:29.330369 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005491 2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 18:58:29.330369 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005507 2560 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 18:58:29.330369 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005511 2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 18:58:29.330369 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005518 2560 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 18:58:29.330369 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005522 2560 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 18:58:29.330369 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005525 2560 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 18:58:29.330369 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005528 2560 flags.go:64] FLAG: --pod-cidr="" Jul 10 18:58:29.330369 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005538 2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 18:58:29.330369 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005544 2560 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 18:58:29.330369 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005547 2560 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 18:58:29.330369 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005551 2560 flags.go:64] FLAG: --pods-per-core="0" Jul 10 18:58:29.330369 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005554 2560 flags.go:64] FLAG: --port="10250" Jul 10 18:58:29.330369 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005557 2560 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 18:58:29.330369 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005560 2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00e931464831f71bc" Jul 10 18:58:29.330369 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005563 2560 flags.go:64] FLAG: --qos-reserved="" Jul 10 18:58:29.330369 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005566 2560 flags.go:64] FLAG: --read-only-port="10255" Jul 10 18:58:29.330369 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005569 2560 flags.go:64] FLAG: --register-node="true" Jul 10 18:58:29.330369 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005572 2560 flags.go:64] FLAG: --register-with-taints="" Jul 10 18:58:29.330369 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005575 2560 flags.go:64] FLAG: --registry-burst="10" Jul 10 18:58:29.331180 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005580 2560 flags.go:64] FLAG: --registry-qps="5" Jul 10 18:58:29.331180 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005582 2560 flags.go:64] FLAG: --reserved-cpus="" Jul 10 18:58:29.331180 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005585 2560 flags.go:64] FLAG: --reserved-memory="" Jul 10 18:58:29.331180 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005589 2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 18:58:29.331180 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005592 2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 18:58:29.331180 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005595 2560 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 18:58:29.331180 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005598 2560 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 18:58:29.331180 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005601 2560 flags.go:64] FLAG: --runonce="false" Jul 10 18:58:29.331180 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005604 2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 18:58:29.331180 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005607 2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 18:58:29.331180 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005610 2560 flags.go:64] FLAG: --seccomp-default="false" Jul 10 18:58:29.331180 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005614 2560 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 18:58:29.331180 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005616 2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 18:58:29.331180 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005620 2560 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 18:58:29.331180 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005623 2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 18:58:29.331180 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005626 2560 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 18:58:29.331180 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005629 2560 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 18:58:29.331180 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005631 2560 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 18:58:29.331180 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005634 2560 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 18:58:29.331180 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005638 2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 18:58:29.331180 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005640 2560 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 18:58:29.331180 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005651 2560 flags.go:64] FLAG: --system-cgroups="" Jul 10 18:58:29.331180 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005654 2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 18:58:29.331180 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005659 2560 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 18:58:29.331180 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005662 2560 flags.go:64] FLAG: --tls-cert-file="" Jul 10 18:58:29.332011 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005665 2560 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 18:58:29.332011 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005672 2560 flags.go:64] FLAG: --tls-min-version="" Jul 10 18:58:29.332011 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005674 2560 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 18:58:29.332011 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005677 2560 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 18:58:29.332011 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005680 2560 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 18:58:29.332011 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005683 2560 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 18:58:29.332011 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005685 2560 flags.go:64] FLAG: --v="2" Jul 10 18:58:29.332011 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005691 2560 flags.go:64] FLAG: --version="false" Jul 10 18:58:29.332011 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005695 2560 flags.go:64] FLAG: --vmodule="" Jul 10 18:58:29.332011 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005699 2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 18:58:29.332011 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.005702 2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 18:58:29.332011 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005861 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 18:58:29.332011 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005865 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 10 18:58:29.332011 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005869 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 18:58:29.332011 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005872 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 18:58:29.332011 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005875 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 18:58:29.332011 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005877 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 18:58:29.332011 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005880 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 18:58:29.332011 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005882 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 18:58:29.332011 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005885 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 18:58:29.332011 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005887 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 18:58:29.332011 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005890 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 18:58:29.332811 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005892 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 18:58:29.332811 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005895 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 18:58:29.332811 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005897 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 18:58:29.332811 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005900 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 18:58:29.332811 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005903 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 18:58:29.332811 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005905 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 18:58:29.332811 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005909 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 18:58:29.332811 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005919 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 18:58:29.332811 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005921 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 18:58:29.332811 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005924 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 18:58:29.332811 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005927 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 18:58:29.332811 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005929 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 18:58:29.332811 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005932 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 18:58:29.332811 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005935 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 18:58:29.332811 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005937 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 18:58:29.332811 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005940 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 18:58:29.332811 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005942 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 18:58:29.332811 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005946 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 18:58:29.332811 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005949 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 18:58:29.332811 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005952 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 18:58:29.333643 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005955 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 18:58:29.333643 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005957 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 18:58:29.333643 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005960 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 18:58:29.333643 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005962 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 18:58:29.333643 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005965 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 18:58:29.333643 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005967 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 18:58:29.333643 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005970 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 18:58:29.333643 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005973 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 18:58:29.333643 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005975 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 18:58:29.333643 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005978 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 18:58:29.333643 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005980 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 18:58:29.333643 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005982 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 18:58:29.333643 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005985 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 18:58:29.333643 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005987 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 18:58:29.333643 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005990 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 18:58:29.333643 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005993 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 18:58:29.333643 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005995 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 18:58:29.333643 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.005998 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 18:58:29.333643 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006002 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 18:58:29.333643 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006004 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 18:58:29.334470 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006012 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 18:58:29.334470 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006015 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 18:58:29.334470 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006018 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 18:58:29.334470 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006021 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 18:58:29.334470 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006024 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 18:58:29.334470 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006026 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 18:58:29.334470 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006029 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 18:58:29.334470 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006033 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 18:58:29.334470 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006037 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 18:58:29.334470 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006042 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 18:58:29.334470 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006045 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 18:58:29.334470 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006048 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 18:58:29.334470 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006052 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 18:58:29.334470 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006055 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 18:58:29.334470 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006058 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 18:58:29.334470 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006060 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 18:58:29.334470 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006064 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 18:58:29.334470 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006066 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 18:58:29.334470 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006069 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 18:58:29.335255 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006072 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 18:58:29.335255 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006075 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 18:58:29.335255 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006078 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 18:58:29.335255 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006081 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 18:58:29.335255 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006083 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 18:58:29.335255 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006086 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 18:58:29.335255 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006088 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 18:58:29.335255 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006091 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 18:58:29.335255 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006093 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 18:58:29.335255 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006096 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 18:58:29.335255 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006098 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 18:58:29.335255 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006102 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 18:58:29.335255 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006105 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 18:58:29.335255 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006108 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 18:58:29.335255 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006117 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 18:58:29.335255 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006121 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 18:58:29.335255 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006124 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 18:58:29.335255 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006127 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 18:58:29.335255 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006129 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 18:58:29.336058 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006132 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 18:58:29.336058 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006134 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 18:58:29.336058 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006137 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 18:58:29.336058 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006141 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 18:58:29.336058 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006143 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 18:58:29.336058 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006146 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 18:58:29.336058 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006148 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 18:58:29.336058 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006151 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 18:58:29.336058 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006153 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 18:58:29.336058 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.006156 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 18:58:29.336058 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.006782 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 18:58:29.336058 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.013919 2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 18:58:29.336058 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.013934 2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 18:58:29.336058 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.013955 2560 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 18:58:29.336058 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.013961 2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 18:58:29.336842 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014030 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 18:58:29.336842 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014036 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 18:58:29.336842 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014039 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 18:58:29.336842 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014042 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 18:58:29.336842 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014045 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 18:58:29.336842 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014048 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 18:58:29.336842 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014051 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 18:58:29.336842 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014054 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 18:58:29.336842 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014057 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 18:58:29.336842 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014060 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 18:58:29.336842 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014063 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 18:58:29.336842 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014066 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 18:58:29.336842 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014069 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 18:58:29.336842 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014073 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 18:58:29.336842 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014076 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 18:58:29.336842 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014079 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 18:58:29.336842 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014081 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 18:58:29.336842 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014084 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 18:58:29.336842 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014086 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 18:58:29.336842 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014089 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 18:58:29.337650 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014091 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 18:58:29.337650 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014094 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 18:58:29.337650 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014096 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 18:58:29.337650 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014099 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 18:58:29.337650 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014101 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 18:58:29.337650 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014104 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 18:58:29.337650 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014107 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 18:58:29.337650 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014111 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 18:58:29.337650 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014115 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 18:58:29.337650 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014117 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 18:58:29.337650 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014121 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 18:58:29.337650 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014123 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 18:58:29.337650 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014126 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 18:58:29.337650 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014128 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 18:58:29.337650 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014131 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 18:58:29.337650 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014134 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 18:58:29.337650 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014136 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 18:58:29.337650 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014139 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 18:58:29.337650 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014141 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 18:58:29.337650 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014144 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 18:58:29.338456 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014147 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 18:58:29.338456 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014149 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 18:58:29.338456 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014153 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 18:58:29.338456 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014156 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 18:58:29.338456 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014159 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 18:58:29.338456 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014163 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 18:58:29.338456 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014166 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 18:58:29.338456 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014169 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 18:58:29.338456 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014172 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 18:58:29.338456 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014175 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 18:58:29.338456 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014177 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 18:58:29.338456 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014180 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 18:58:29.338456 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014183 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 18:58:29.338456 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014186 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 18:58:29.338456 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014188 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 18:58:29.338456 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014191 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 18:58:29.338456 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014193 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 18:58:29.338456 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014196 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 10 18:58:29.338456 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014199 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 18:58:29.339264 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014202 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 18:58:29.339264 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014204 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 18:58:29.339264 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014207 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 18:58:29.339264 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014209 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 18:58:29.339264 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014212 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 18:58:29.339264 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014214 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 18:58:29.339264 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014217 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 18:58:29.339264 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014219 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 18:58:29.339264 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014222 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 18:58:29.339264 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014224 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 18:58:29.339264 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014227 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 18:58:29.339264 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014229 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 18:58:29.339264 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014232 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 18:58:29.339264 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014234 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 18:58:29.339264 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014237 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 18:58:29.339264 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014239 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 18:58:29.339264 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014241 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 18:58:29.339264 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014244 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 18:58:29.339264 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014247 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 18:58:29.339264 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014250 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 18:58:29.340063 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014254 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 18:58:29.340063 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014257 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 18:58:29.340063 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014260 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 18:58:29.340063 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014262 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 18:58:29.340063 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014265 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 18:58:29.340063 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014268 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 18:58:29.340063 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014270 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 18:58:29.340063 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014273 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 18:58:29.340063 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014276 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 18:58:29.340063 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014278 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 18:58:29.340063 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014281 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 18:58:29.340063 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014283 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 18:58:29.340063 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014286 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 18:58:29.340063 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014288 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 18:58:29.340063 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014291 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 18:58:29.340063 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014293 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 18:58:29.340063 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014295 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 18:58:29.340063 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014298 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 18:58:29.340063 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014300 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 18:58:29.340866 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014303 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 18:58:29.340866 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.014309 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 18:58:29.340866 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014425 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 18:58:29.340866 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014429 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 18:58:29.340866 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014433 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 18:58:29.340866 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014436 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 18:58:29.340866 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014454 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 18:58:29.340866 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014457 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 18:58:29.340866 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014460 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 18:58:29.340866 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014463 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 18:58:29.340866 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014466 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 18:58:29.340866 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014469 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 18:58:29.340866 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014472 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 18:58:29.340866 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014475 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 18:58:29.340866 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014478 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 18:58:29.341712 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014481 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 18:58:29.341712 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014483 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 18:58:29.341712 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014486 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 18:58:29.341712 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014488 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 18:58:29.341712 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014493 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 18:58:29.341712 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014496 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 18:58:29.341712 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014498 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 18:58:29.341712 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014501 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 18:58:29.341712 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014503 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 18:58:29.341712 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014506 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 18:58:29.341712 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014509 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 18:58:29.341712 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014511 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 18:58:29.341712 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014514 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 18:58:29.341712 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014516 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 18:58:29.341712 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014520 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 18:58:29.341712 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014524 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 18:58:29.341712 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014526 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 18:58:29.341712 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014529 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 18:58:29.341712 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014532 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 18:58:29.341712 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014534 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 18:58:29.342510 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014537 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 18:58:29.342510 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014540 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 18:58:29.342510 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014543 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 18:58:29.342510 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014547 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 18:58:29.342510 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014550 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 18:58:29.342510 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014552 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 18:58:29.342510 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014555 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 18:58:29.342510 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014558 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 18:58:29.342510 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014560 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 18:58:29.342510 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014563 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 18:58:29.342510 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014566 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 18:58:29.342510 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014569 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 18:58:29.342510 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014571 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 18:58:29.342510 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014574 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 18:58:29.342510 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014576 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 18:58:29.342510 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014579 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 18:58:29.342510 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014583 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 18:58:29.342510 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014587 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 18:58:29.343315 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014590 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 18:58:29.343315 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014593 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 18:58:29.343315 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014596 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 18:58:29.343315 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014599 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 18:58:29.343315 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014602 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 18:58:29.343315 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014604 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 18:58:29.343315 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014607 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 18:58:29.343315 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014609 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 18:58:29.343315 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014612 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 18:58:29.343315 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014615 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 18:58:29.343315 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014618 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 18:58:29.343315 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014621 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 18:58:29.343315 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014623 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 18:58:29.343315 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014626 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 18:58:29.343315 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014628 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 18:58:29.343315 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014631 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 18:58:29.343315 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014633 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 18:58:29.343315 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014636 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 18:58:29.343315 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014638 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 18:58:29.343315 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014641 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 18:58:29.347420 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014643 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 18:58:29.347420 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014646 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 18:58:29.347420 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014649 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 18:58:29.347420 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014651 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 18:58:29.347420 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014654 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 18:58:29.347420 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014656 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 18:58:29.347420 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014659 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 18:58:29.347420 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014662 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 18:58:29.347420 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014664 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 18:58:29.347420 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014666 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 18:58:29.347420 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014671 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 18:58:29.347420 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014673 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 18:58:29.347420 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014676 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 18:58:29.347420 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014679 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 18:58:29.347420 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014681 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 18:58:29.347420 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014684 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 18:58:29.347420 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014686 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 10 18:58:29.347420 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014689 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 18:58:29.347420 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014692 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 18:58:29.347420 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014694 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 18:58:29.348326 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014697 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 18:58:29.348326 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014699 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 18:58:29.348326 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014702 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 18:58:29.348326 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014704 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 18:58:29.348326 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014706 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 18:58:29.348326 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014709 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 18:58:29.348326 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014711 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 18:58:29.348326 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:29.014714 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 18:58:29.348326 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.014719 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 18:58:29.348326 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.014841 2560 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 18:58:29.348326 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.018929 2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 18:58:29.348326 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.020467 2560 server.go:1019] "Starting client certificate rotation" Jul 10 18:58:29.348326 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.020567 2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 18:58:29.348326 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.020609 2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 18:58:29.349646 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.026513 2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 18:58:29.349646 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.027679 2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 18:58:29.349646 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.036483 2560 log.go:25] "Validated CRI v1 runtime API" Jul 10 18:58:29.349646 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.039963 2560 log.go:25] "Validated CRI v1 image API" Jul 10 18:58:29.349646 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.040719 2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 18:58:29.349646 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.043291 2560 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9f089c3a-5b54-494f-9b64-f70236586548:/dev/nvme0n1p3 b804682d-16b1-4ccc-a148-635ec84c1408:/dev/nvme0n1p4] Jul 10 18:58:29.349646 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.043312 2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 18:58:29.350939 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.048757 2560 manager.go:216] Machine: {Timestamp:2026-07-10 18:58:29.047397498 +0000 UTC m=+0.377538320 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099889 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cca524f78c0c9f722ef19a15942d5 SystemUUID:ec2cca52-4f78-c0c9-f722-ef19a15942d5 BootID:ecd09261-da91-4151-bbe9-5ed837c9806b Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:91:33:a7:f9:35 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:91:33:a7:f9:35 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:86:be:3e:a9:bc:ba Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 18:58:29.350939 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.048862 2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 18:58:29.350939 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.048984 2560 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 18:58:29.350939 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.052581 2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 18:58:29.350939 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.052613 2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-213.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 18:58:29.350939 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.053395 2560 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 18:58:29.350939 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.053405 2560 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 18:58:29.350939 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.053414 2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 18:58:29.350939 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.054218 2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 18:58:29.350939 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.055426 2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 18:58:29.350939 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.057330 2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 18:58:29.350939 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.059094 2560 state_mem.go:36] "Initialized new in-memory state store" Jul 10 18:58:29.350939 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.059217 2560 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 18:58:29.350939 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.059239 2560 kubelet.go:486] "Attempting to sync node with API server" Jul 10 18:58:29.350939 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.059250 2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 18:58:29.350939 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.059264 2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 18:58:29.350939 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.059273 2560 kubelet.go:398] "Adding apiserver pod source" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.059282 2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.059664 2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.059678 2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.063974 2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.066563 2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.066591 2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.068041 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.068062 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.068069 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.068074 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.068083 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.068088 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.068096 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.068101 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.068108 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.068114 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.068123 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.068129 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.068933 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.068940 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:29.070069 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:29.070070 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-213.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.072336 2560 server.go:1268] "Started kubelet" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.072406 2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.072830 2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.072983 2560 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.073582 2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-213.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.073879 2560 server.go:310] "Adding debug handlers to kubelet server" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.075015 2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:29.077419 2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:29.078317 2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-213.ec2.internal.18c102b638996ee2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-213.ec2.internal,UID:ip-10-0-138-213.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-213.ec2.internal,},FirstTimestamp:2026-07-10 18:58:29.072309986 +0000 UTC m=+0.402450786,LastTimestamp:2026-07-10 18:58:29.072309986 +0000 UTC m=+0.402450786,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-213.ec2.internal,}" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.082090 2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.082101 2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:29.082246 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-213.ec2.internal\" not found" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.082734 2560 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.082772 2560 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.082941 2560 reconstruct.go:97] "Volume reconstruction finished" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.082950 2560 reconciler.go:29] "Reconciler: start to sync state" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.083053 2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.083219 2560 factory.go:153] Registering CRI-O factory Jul 10 18:58:29.351793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.083234 2560 factory.go:223] Registration of the crio container factory successfully Jul 10 18:58:29.457586 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.083288 2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 18:58:29.457586 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.083298 2560 factory.go:55] Registering systemd factory Jul 10 18:58:29.457586 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.083305 2560 factory.go:223] Registration of the systemd container factory successfully Jul 10 18:58:29.457586 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.083330 2560 factory.go:103] Registering Raw factory Jul 10 18:58:29.457586 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.083343 2560 manager.go:1196] Started watching for new ooms in manager Jul 10 18:58:29.457586 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.083875 2560 manager.go:349] Starting recovery of all containers Jul 10 18:58:29.457586 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:29.085720 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 10 18:58:29.457586 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:29.085774 2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-213.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 10 18:58:29.457586 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.097143 2560 manager.go:354] Recovery completed Jul 10 18:58:29.457586 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.101337 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 18:58:29.457586 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.103234 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-213.ec2.internal" event="NodeHasSufficientMemory" Jul 10 18:58:29.457586 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.103272 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-213.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 18:58:29.457586 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.103285 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-213.ec2.internal" event="NodeHasSufficientPID" Jul 10 18:58:29.457586 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.103808 2560 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 18:58:29.457586 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.103817 2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 18:58:29.457586 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.103825 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 18:58:29.457586 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.103831 2560 state_mem.go:36] "Initialized new in-memory state store" Jul 10 18:58:29.457586 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.104546 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-213.ec2.internal" event="NodeHasSufficientMemory" Jul 10 18:58:29.457586 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.104572 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-213.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 18:58:29.457586 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.104581 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-213.ec2.internal" event="NodeHasSufficientPID" Jul 10 18:58:29.457586 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.105256 2560 policy_none.go:49] "None policy: Start" Jul 10 18:58:29.457586 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.105270 2560 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 18:58:29.457586 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.105280 2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 18:58:29.457586 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:29.105421 2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-213.ec2.internal.18c102b63a719e2f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-213.ec2.internal,UID:ip-10-0-138-213.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-213.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-213.ec2.internal,},FirstTimestamp:2026-07-10 18:58:29.103255087 +0000 UTC m=+0.433395888,LastTimestamp:2026-07-10 18:58:29.103255087 +0000 UTC m=+0.433395888,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-213.ec2.internal,}" Jul 10 18:58:29.457586 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.106399 2560 policy_none.go:47] "Start" Jul 10 18:58:29.457586 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.110181 2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qxwv6" Jul 10 18:58:29.457586 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.122164 2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qxwv6" Jul 10 18:58:29.457586 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:29.129522 2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-213.ec2.internal.18c102b63a71fb33 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-213.ec2.internal,UID:ip-10-0-138-213.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-213.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-213.ec2.internal,},FirstTimestamp:2026-07-10 18:58:29.103278899 +0000 UTC m=+0.433419700,LastTimestamp:2026-07-10 18:58:29.103278899 +0000 UTC m=+0.433419700,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-213.ec2.internal,}" Jul 10 18:58:29.457586 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.155167 2560 manager.go:337] "Starting Device Plugin manager" Jul 10 18:58:29.457586 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:29.155204 2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 18:58:29.457586 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.155216 2560 server.go:87] "Starting device plugin registration server" Jul 10 18:58:29.457586 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.155506 2560 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.155517 2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.155606 2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.155677 2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.155685 2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:29.156243 2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:29.156287 2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-213.ec2.internal\" not found" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.244508 2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.245782 2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.245828 2560 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.245857 2560 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:29.245901 2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.248414 2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.256388 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.257233 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-213.ec2.internal" event="NodeHasSufficientMemory" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.257259 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-213.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.257269 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-213.ec2.internal" event="NodeHasSufficientPID" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.257290 2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-213.ec2.internal" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.266628 2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-213.ec2.internal" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:29.266649 2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-213.ec2.internal\": node \"ip-10-0-138-213.ec2.internal\" not found" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:29.288078 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-213.ec2.internal\" not found" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.346459 2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-213.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-213.ec2.internal"] Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.346688 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.347700 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-213.ec2.internal" event="NodeHasSufficientMemory" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.347722 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-213.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.347731 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-213.ec2.internal" event="NodeHasSufficientPID" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.349220 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.349377 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-213.ec2.internal" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.349435 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.350004 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-213.ec2.internal" event="NodeHasSufficientMemory" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.350021 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-213.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.350037 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-213.ec2.internal" event="NodeHasSufficientPID" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.350227 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-213.ec2.internal" event="NodeHasSufficientMemory" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.350237 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-213.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.350245 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-213.ec2.internal" event="NodeHasSufficientPID" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.351550 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-213.ec2.internal" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.351580 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.352286 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-213.ec2.internal" event="NodeHasSufficientMemory" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.352306 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-213.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 18:58:29.458782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.352320 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-213.ec2.internal" event="NodeHasSufficientPID" Jul 10 18:58:29.819144 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:29.720731 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-213.ec2.internal\" not found" Jul 10 18:58:29.819144 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.720826 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/14b1acdcd9ea09cc31366e606e569025-config\") pod \"kube-apiserver-proxy-ip-10-0-138-213.ec2.internal\" (UID: \"14b1acdcd9ea09cc31366e606e569025\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-213.ec2.internal" Jul 10 18:58:29.819144 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.720859 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/eb6a2004a5a296385f19413f1682752d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-213.ec2.internal\" (UID: \"eb6a2004a5a296385f19413f1682752d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-213.ec2.internal" Jul 10 18:58:29.819144 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.720890 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb6a2004a5a296385f19413f1682752d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-213.ec2.internal\" (UID: \"eb6a2004a5a296385f19413f1682752d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-213.ec2.internal" Jul 10 18:58:29.825110 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:29.821415 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-213.ec2.internal\" not found" Jul 10 18:58:29.825110 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.821428 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb6a2004a5a296385f19413f1682752d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-213.ec2.internal\" (UID: \"eb6a2004a5a296385f19413f1682752d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-213.ec2.internal" Jul 10 18:58:29.825110 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.821486 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb6a2004a5a296385f19413f1682752d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-213.ec2.internal\" (UID: \"eb6a2004a5a296385f19413f1682752d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-213.ec2.internal" Jul 10 18:58:29.825110 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.821490 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/14b1acdcd9ea09cc31366e606e569025-config\") pod \"kube-apiserver-proxy-ip-10-0-138-213.ec2.internal\" (UID: \"14b1acdcd9ea09cc31366e606e569025\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-213.ec2.internal" Jul 10 18:58:29.825110 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.821550 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/eb6a2004a5a296385f19413f1682752d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-213.ec2.internal\" (UID: \"eb6a2004a5a296385f19413f1682752d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-213.ec2.internal" Jul 10 18:58:29.825110 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.821570 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/eb6a2004a5a296385f19413f1682752d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-213.ec2.internal\" (UID: \"eb6a2004a5a296385f19413f1682752d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-213.ec2.internal" Jul 10 18:58:29.825110 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:29.821520 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/14b1acdcd9ea09cc31366e606e569025-config\") pod \"kube-apiserver-proxy-ip-10-0-138-213.ec2.internal\" (UID: \"14b1acdcd9ea09cc31366e606e569025\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-213.ec2.internal" Jul 10 18:58:29.946066 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:29.923934 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-213.ec2.internal\" not found" Jul 10 18:58:30.084575 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:30.019494 2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 18:58:30.084575 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:30.019689 2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 18:58:30.084575 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:30.024651 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-213.ec2.internal\" not found" Jul 10 18:58:30.218386 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:30.125036 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-213.ec2.internal\" not found" Jul 10 18:58:30.218386 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:30.126153 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 18:53:29 +0000 UTC" deadline="2028-01-06 23:21:31.739380249 +0000 UTC" Jul 10 18:58:30.218386 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:30.126186 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13084h23m1.613198154s" Jul 10 18:58:30.229341 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:30.225970 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-213.ec2.internal\" not found" Jul 10 18:58:30.353755 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:30.296819 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 18:58:30.353755 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:30.326640 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-213.ec2.internal\" not found" Jul 10 18:58:30.473276 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:30.369476 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-213.ec2.internal" event="NodeHasSufficientMemory" Jul 10 18:58:30.473276 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:30.369539 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-213.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 18:58:30.473276 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:30.369559 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-213.ec2.internal" event="NodeHasSufficientPID" Jul 10 18:58:30.473276 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:30.371645 2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 18:58:30.473276 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:30.376026 2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 18:58:30.473276 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:30.382859 2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-213.ec2.internal" Jul 10 18:58:30.583878 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:30.583849 2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 18:58:30.592721 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:30.592698 2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-213.ec2.internal" Jul 10 18:58:30.593198 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:30.593183 2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 18:58:30.594408 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:30.594393 2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-213.ec2.internal" Jul 10 18:58:30.597641 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:30.597626 2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-213.ec2.internal" Jul 10 18:58:30.599531 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:30.599508 2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 18:58:30.601699 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:30.601685 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-213.ec2.internal" Jul 10 18:58:30.603630 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:30.603589 2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 18:58:30.603675 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:30.603639 2560 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-213.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-213.ec2.internal" Jul 10 18:58:30.607605 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:30.607585 2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 18:58:30.607676 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:30.607645 2560 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-213.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-213.ec2.internal" Jul 10 18:58:30.610962 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:30.610948 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-213.ec2.internal" Jul 10 18:58:31.060303 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.060212 2560 apiserver.go:52] "Watching apiserver" Jul 10 18:58:31.071946 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.071916 2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 18:58:31.073115 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.073093 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spzdj","openshift-cluster-node-tuning-operator/tuned-cv4cv","openshift-dns/node-resolver-l9qmm","openshift-image-registry/node-ca-x6sx7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-213.ec2.internal","openshift-multus/multus-additional-cni-plugins-z58c2","kube-system/kube-apiserver-proxy-ip-10-0-138-213.ec2.internal","openshift-multus/multus-sx2wz","openshift-multus/network-metrics-daemon-v9b95","openshift-network-diagnostics/network-check-target-bl7wj","openshift-network-operator/iptables-alerter-g5t7c","openshift-ovn-kubernetes/ovnkube-node-9l52t","kube-system/global-pull-secret-syncer-zv9t9","kube-system/konnectivity-agent-4hn8z"] Jul 10 18:58:31.074767 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.074749 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bl7wj" Jul 10 18:58:31.074861 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:31.074839 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bl7wj" podUID="4526c66d-17d8-4ed2-9782-42b9b2d9215b" Jul 10 18:58:31.075837 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.075819 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g5t7c" Jul 10 18:58:31.077480 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.077461 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l9qmm" Jul 10 18:58:31.078719 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.078701 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 18:58:31.078923 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.078909 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 18:58:31.078991 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.078969 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x6sx7" Jul 10 18:58:31.079151 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.079129 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 18:58:31.079474 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.079457 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gjr6d\"" Jul 10 18:58:31.080017 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.079994 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 18:58:31.080230 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.080215 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4hn8z" Jul 10 18:58:31.080397 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.080263 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 18:58:31.080397 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.080364 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-g4kz7\"" Jul 10 18:58:31.081385 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.081366 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 18:58:31.081668 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.081655 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z58c2" Jul 10 18:58:31.082063 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.082049 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 18:58:31.082280 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.082269 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 18:58:31.082803 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.082785 2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 18:58:31.082886 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.082874 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 18:58:31.082941 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:31.082926 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9b95" podUID="b1bf574f-a4e2-4418-8f18-95fa0c80ca1e" Jul 10 18:58:31.084454 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.084434 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lkv45\"" Jul 10 18:58:31.086050 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.084885 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:58:31.086050 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:31.085035 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9t9" podUID="e3e3a914-d7de-4771-9d68-a7b05ab56b1f" Jul 10 18:58:31.086050 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.085221 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 18:58:31.086050 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.085343 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 18:58:31.086050 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.085634 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 18:58:31.086050 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.085888 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mv4ln\"" Jul 10 18:58:31.086050 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.085973 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 18:58:31.086050 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.086022 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 18:58:31.086816 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.086625 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 18:58:31.086816 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.086800 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-csww5\"" Jul 10 18:58:31.087126 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.087102 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.087216 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.087190 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 18:58:31.088654 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.088632 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spzdj" Jul 10 18:58:31.090224 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.090207 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.090661 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.090644 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 18:58:31.090727 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.090671 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4hndr\"" Jul 10 18:58:31.090885 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.090873 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 18:58:31.091266 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.091250 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 18:58:31.091555 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.091503 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 18:58:31.091555 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.091536 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 18:58:31.091711 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.091697 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8g9sm\"" Jul 10 18:58:31.091893 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.091877 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.093136 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.093088 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 18:58:31.093686 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.093669 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jtl22\"" Jul 10 18:58:31.096156 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.096138 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 18:58:31.096255 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.096146 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 18:58:31.096255 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.096146 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 18:58:31.096255 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.096190 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 18:58:31.096255 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.096224 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 18:58:31.096448 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.096418 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tlv89\"" Jul 10 18:58:31.096589 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.096572 2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 18:58:31.096636 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.096618 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 18:58:31.121849 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.121827 2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cfrjv" Jul 10 18:58:31.128728 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.128708 2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cfrjv" Jul 10 18:58:31.137117 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.137092 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slg7w\" (UniqueName: \"kubernetes.io/projected/4526c66d-17d8-4ed2-9782-42b9b2d9215b-kube-api-access-slg7w\") pod \"network-check-target-bl7wj\" (UID: \"4526c66d-17d8-4ed2-9782-42b9b2d9215b\") " pod="openshift-network-diagnostics/network-check-target-bl7wj" Jul 10 18:58:31.137297 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.137121 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d85934ea-40c2-4589-aca7-b0e8efca569c-hosts-file\") pod \"node-resolver-l9qmm\" (UID: \"d85934ea-40c2-4589-aca7-b0e8efca569c\") " pod="openshift-dns/node-resolver-l9qmm" Jul 10 18:58:31.137297 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.137141 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd6564ad-086b-4c7b-ada7-99f16bfe03c4-host\") pod \"node-ca-x6sx7\" (UID: \"fd6564ad-086b-4c7b-ada7-99f16bfe03c4\") " pod="openshift-image-registry/node-ca-x6sx7" Jul 10 18:58:31.137297 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.137157 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fd6564ad-086b-4c7b-ada7-99f16bfe03c4-serviceca\") pod \"node-ca-x6sx7\" (UID: \"fd6564ad-086b-4c7b-ada7-99f16bfe03c4\") " pod="openshift-image-registry/node-ca-x6sx7" Jul 10 18:58:31.137297 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.137215 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a-agent-certs\") pod \"konnectivity-agent-4hn8z\" (UID: \"00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a\") " pod="kube-system/konnectivity-agent-4hn8z" Jul 10 18:58:31.137297 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.137288 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a-konnectivity-ca\") pod \"konnectivity-agent-4hn8z\" (UID: \"00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a\") " pod="kube-system/konnectivity-agent-4hn8z" Jul 10 18:58:31.137597 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.137327 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eda6e317-2a4b-4591-a5ef-0139d0e57e95-os-release\") pod \"multus-additional-cni-plugins-z58c2\" (UID: \"eda6e317-2a4b-4591-a5ef-0139d0e57e95\") " pod="openshift-multus/multus-additional-cni-plugins-z58c2" Jul 10 18:58:31.137597 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.137354 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eda6e317-2a4b-4591-a5ef-0139d0e57e95-cni-binary-copy\") pod \"multus-additional-cni-plugins-z58c2\" (UID: \"eda6e317-2a4b-4591-a5ef-0139d0e57e95\") " pod="openshift-multus/multus-additional-cni-plugins-z58c2" Jul 10 18:58:31.137597 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.137377 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/eda6e317-2a4b-4591-a5ef-0139d0e57e95-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z58c2\" (UID: \"eda6e317-2a4b-4591-a5ef-0139d0e57e95\") " pod="openshift-multus/multus-additional-cni-plugins-z58c2" Jul 10 18:58:31.137597 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.137410 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/eda6e317-2a4b-4591-a5ef-0139d0e57e95-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z58c2\" (UID: \"eda6e317-2a4b-4591-a5ef-0139d0e57e95\") " pod="openshift-multus/multus-additional-cni-plugins-z58c2" Jul 10 18:58:31.137597 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.137437 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmgrk\" (UniqueName: \"kubernetes.io/projected/eda6e317-2a4b-4591-a5ef-0139d0e57e95-kube-api-access-fmgrk\") pod \"multus-additional-cni-plugins-z58c2\" (UID: \"eda6e317-2a4b-4591-a5ef-0139d0e57e95\") " pod="openshift-multus/multus-additional-cni-plugins-z58c2" Jul 10 18:58:31.137597 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.137472 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smmwh\" (UniqueName: \"kubernetes.io/projected/b1bf574f-a4e2-4418-8f18-95fa0c80ca1e-kube-api-access-smmwh\") pod \"network-metrics-daemon-v9b95\" (UID: \"b1bf574f-a4e2-4418-8f18-95fa0c80ca1e\") " pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 18:58:31.137597 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.137489 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e3e3a914-d7de-4771-9d68-a7b05ab56b1f-kubelet-config\") pod \"global-pull-secret-syncer-zv9t9\" (UID: \"e3e3a914-d7de-4771-9d68-a7b05ab56b1f\") " pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:58:31.137597 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.137504 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/13f6e0d5-e879-45c0-b2ec-7b310c717933-host-slash\") pod \"iptables-alerter-g5t7c\" (UID: \"13f6e0d5-e879-45c0-b2ec-7b310c717933\") " pod="openshift-network-operator/iptables-alerter-g5t7c" Jul 10 18:58:31.137597 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.137524 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eda6e317-2a4b-4591-a5ef-0139d0e57e95-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z58c2\" (UID: \"eda6e317-2a4b-4591-a5ef-0139d0e57e95\") " pod="openshift-multus/multus-additional-cni-plugins-z58c2" Jul 10 18:58:31.137597 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.137554 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d85934ea-40c2-4589-aca7-b0e8efca569c-tmp-dir\") pod \"node-resolver-l9qmm\" (UID: \"d85934ea-40c2-4589-aca7-b0e8efca569c\") " pod="openshift-dns/node-resolver-l9qmm" Jul 10 18:58:31.137597 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.137576 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eda6e317-2a4b-4591-a5ef-0139d0e57e95-system-cni-dir\") pod \"multus-additional-cni-plugins-z58c2\" (UID: \"eda6e317-2a4b-4591-a5ef-0139d0e57e95\") " pod="openshift-multus/multus-additional-cni-plugins-z58c2" Jul 10 18:58:31.137597 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.137601 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eda6e317-2a4b-4591-a5ef-0139d0e57e95-cnibin\") pod \"multus-additional-cni-plugins-z58c2\" (UID: \"eda6e317-2a4b-4591-a5ef-0139d0e57e95\") " pod="openshift-multus/multus-additional-cni-plugins-z58c2" Jul 10 18:58:31.138060 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.137618 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1bf574f-a4e2-4418-8f18-95fa0c80ca1e-metrics-certs\") pod \"network-metrics-daemon-v9b95\" (UID: \"b1bf574f-a4e2-4418-8f18-95fa0c80ca1e\") " pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 18:58:31.138060 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.137644 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/13f6e0d5-e879-45c0-b2ec-7b310c717933-iptables-alerter-script\") pod \"iptables-alerter-g5t7c\" (UID: \"13f6e0d5-e879-45c0-b2ec-7b310c717933\") " pod="openshift-network-operator/iptables-alerter-g5t7c" Jul 10 18:58:31.138060 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.137687 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg6x9\" (UniqueName: \"kubernetes.io/projected/d85934ea-40c2-4589-aca7-b0e8efca569c-kube-api-access-tg6x9\") pod \"node-resolver-l9qmm\" (UID: \"d85934ea-40c2-4589-aca7-b0e8efca569c\") " pod="openshift-dns/node-resolver-l9qmm" Jul 10 18:58:31.138060 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.137703 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e3e3a914-d7de-4771-9d68-a7b05ab56b1f-dbus\") pod \"global-pull-secret-syncer-zv9t9\" (UID: \"e3e3a914-d7de-4771-9d68-a7b05ab56b1f\") " pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:58:31.138060 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.137720 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3e3a914-d7de-4771-9d68-a7b05ab56b1f-original-pull-secret\") pod \"global-pull-secret-syncer-zv9t9\" (UID: \"e3e3a914-d7de-4771-9d68-a7b05ab56b1f\") " pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:58:31.138060 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.137745 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lshlg\" (UniqueName: \"kubernetes.io/projected/13f6e0d5-e879-45c0-b2ec-7b310c717933-kube-api-access-lshlg\") pod \"iptables-alerter-g5t7c\" (UID: \"13f6e0d5-e879-45c0-b2ec-7b310c717933\") " pod="openshift-network-operator/iptables-alerter-g5t7c" Jul 10 18:58:31.138060 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.137776 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s44cc\" (UniqueName: \"kubernetes.io/projected/fd6564ad-086b-4c7b-ada7-99f16bfe03c4-kube-api-access-s44cc\") pod \"node-ca-x6sx7\" (UID: \"fd6564ad-086b-4c7b-ada7-99f16bfe03c4\") " pod="openshift-image-registry/node-ca-x6sx7" Jul 10 18:58:31.184625 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.184595 2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 18:58:31.238740 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.238709 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s44cc\" (UniqueName: \"kubernetes.io/projected/fd6564ad-086b-4c7b-ada7-99f16bfe03c4-kube-api-access-s44cc\") pod \"node-ca-x6sx7\" (UID: \"fd6564ad-086b-4c7b-ada7-99f16bfe03c4\") " pod="openshift-image-registry/node-ca-x6sx7" Jul 10 18:58:31.238912 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.238744 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/90178079-026e-41e5-94a0-d180a1940cc1-etc-systemd\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.238912 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.238763 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-host-cni-netd\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.238912 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.238782 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fd6564ad-086b-4c7b-ada7-99f16bfe03c4-serviceca\") pod \"node-ca-x6sx7\" (UID: \"fd6564ad-086b-4c7b-ada7-99f16bfe03c4\") " pod="openshift-image-registry/node-ca-x6sx7" Jul 10 18:58:31.238912 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.238798 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a-agent-certs\") pod \"konnectivity-agent-4hn8z\" (UID: \"00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a\") " pod="kube-system/konnectivity-agent-4hn8z" Jul 10 18:58:31.238912 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.238862 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/90178079-026e-41e5-94a0-d180a1940cc1-etc-sysctl-d\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.238912 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.238900 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/90178079-026e-41e5-94a0-d180a1940cc1-etc-sysctl-conf\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.239247 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.238939 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-os-release\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.239247 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.238982 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-run-openvswitch\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.239247 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239014 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-log-socket\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.239247 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239067 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smmwh\" (UniqueName: \"kubernetes.io/projected/b1bf574f-a4e2-4418-8f18-95fa0c80ca1e-kube-api-access-smmwh\") pod \"network-metrics-daemon-v9b95\" (UID: \"b1bf574f-a4e2-4418-8f18-95fa0c80ca1e\") " pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 18:58:31.239247 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239138 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e3e3a914-d7de-4771-9d68-a7b05ab56b1f-kubelet-config\") pod \"global-pull-secret-syncer-zv9t9\" (UID: \"e3e3a914-d7de-4771-9d68-a7b05ab56b1f\") " pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:58:31.239247 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239169 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/90178079-026e-41e5-94a0-d180a1940cc1-lib-modules\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.239247 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239160 2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 18:58:31.239247 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239194 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-multus-socket-dir-parent\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.239247 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239222 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-host-run-netns\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.239683 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239253 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e3e3a914-d7de-4771-9d68-a7b05ab56b1f-kubelet-config\") pod \"global-pull-secret-syncer-zv9t9\" (UID: \"e3e3a914-d7de-4771-9d68-a7b05ab56b1f\") " pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:58:31.239683 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239255 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-spzdj\" (UID: \"262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spzdj" Jul 10 18:58:31.239683 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239295 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-run-systemd\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.239683 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239301 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fd6564ad-086b-4c7b-ada7-99f16bfe03c4-serviceca\") pod \"node-ca-x6sx7\" (UID: \"fd6564ad-086b-4c7b-ada7-99f16bfe03c4\") " pod="openshift-image-registry/node-ca-x6sx7" Jul 10 18:58:31.239683 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239320 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1bf574f-a4e2-4418-8f18-95fa0c80ca1e-metrics-certs\") pod \"network-metrics-daemon-v9b95\" (UID: \"b1bf574f-a4e2-4418-8f18-95fa0c80ca1e\") " pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 18:58:31.239683 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239369 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-host-var-lib-cni-bin\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.239683 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239403 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-multus-conf-dir\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.239683 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:31.239418 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:58:31.239683 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239428 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-host-cni-bin\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.239683 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239473 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg6x9\" (UniqueName: \"kubernetes.io/projected/d85934ea-40c2-4589-aca7-b0e8efca569c-kube-api-access-tg6x9\") pod \"node-resolver-l9qmm\" (UID: \"d85934ea-40c2-4589-aca7-b0e8efca569c\") " pod="openshift-dns/node-resolver-l9qmm" Jul 10 18:58:31.239683 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:31.239509 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1bf574f-a4e2-4418-8f18-95fa0c80ca1e-metrics-certs podName:b1bf574f-a4e2-4418-8f18-95fa0c80ca1e nodeName:}" failed. No retries permitted until 2026-07-10 18:58:31.739487345 +0000 UTC m=+3.069628161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b1bf574f-a4e2-4418-8f18-95fa0c80ca1e-metrics-certs") pod "network-metrics-daemon-v9b95" (UID: "b1bf574f-a4e2-4418-8f18-95fa0c80ca1e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:58:31.239683 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239523 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90178079-026e-41e5-94a0-d180a1940cc1-etc-kubernetes\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.239683 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239544 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-system-cni-dir\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.239683 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239567 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-etc-kubernetes\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.239683 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239594 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sqh4\" (UniqueName: \"kubernetes.io/projected/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-kube-api-access-4sqh4\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.239683 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239617 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5-etc-selinux\") pod \"aws-ebs-csi-driver-node-spzdj\" (UID: \"262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spzdj" Jul 10 18:58:31.240457 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239640 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rbr2\" (UniqueName: \"kubernetes.io/projected/262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5-kube-api-access-9rbr2\") pod \"aws-ebs-csi-driver-node-spzdj\" (UID: \"262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spzdj" Jul 10 18:58:31.240457 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239682 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-host-slash\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.240457 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239697 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-host-run-multus-certs\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.240457 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239734 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5-socket-dir\") pod \"aws-ebs-csi-driver-node-spzdj\" (UID: \"262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spzdj" Jul 10 18:58:31.240457 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239760 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-var-lib-openvswitch\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.240457 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239779 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2a22956a-4e96-41de-945c-a851b587a442-env-overrides\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.240457 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239804 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2a22956a-4e96-41de-945c-a851b587a442-ovnkube-script-lib\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.240457 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239841 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldjlz\" (UniqueName: \"kubernetes.io/projected/2a22956a-4e96-41de-945c-a851b587a442-kube-api-access-ldjlz\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.240457 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239861 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eda6e317-2a4b-4591-a5ef-0139d0e57e95-cni-binary-copy\") pod \"multus-additional-cni-plugins-z58c2\" (UID: \"eda6e317-2a4b-4591-a5ef-0139d0e57e95\") " pod="openshift-multus/multus-additional-cni-plugins-z58c2" Jul 10 18:58:31.240457 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239881 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmgrk\" (UniqueName: \"kubernetes.io/projected/eda6e317-2a4b-4591-a5ef-0139d0e57e95-kube-api-access-fmgrk\") pod \"multus-additional-cni-plugins-z58c2\" (UID: \"eda6e317-2a4b-4591-a5ef-0139d0e57e95\") " pod="openshift-multus/multus-additional-cni-plugins-z58c2" Jul 10 18:58:31.240457 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239905 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/13f6e0d5-e879-45c0-b2ec-7b310c717933-host-slash\") pod \"iptables-alerter-g5t7c\" (UID: \"13f6e0d5-e879-45c0-b2ec-7b310c717933\") " pod="openshift-network-operator/iptables-alerter-g5t7c" Jul 10 18:58:31.240457 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239922 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5-device-dir\") pod \"aws-ebs-csi-driver-node-spzdj\" (UID: \"262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spzdj" Jul 10 18:58:31.240457 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239943 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d85934ea-40c2-4589-aca7-b0e8efca569c-tmp-dir\") pod \"node-resolver-l9qmm\" (UID: \"d85934ea-40c2-4589-aca7-b0e8efca569c\") " pod="openshift-dns/node-resolver-l9qmm" Jul 10 18:58:31.240457 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.239971 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eda6e317-2a4b-4591-a5ef-0139d0e57e95-system-cni-dir\") pod \"multus-additional-cni-plugins-z58c2\" (UID: \"eda6e317-2a4b-4591-a5ef-0139d0e57e95\") " pod="openshift-multus/multus-additional-cni-plugins-z58c2" Jul 10 18:58:31.240457 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240012 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eda6e317-2a4b-4591-a5ef-0139d0e57e95-cnibin\") pod \"multus-additional-cni-plugins-z58c2\" (UID: \"eda6e317-2a4b-4591-a5ef-0139d0e57e95\") " pod="openshift-multus/multus-additional-cni-plugins-z58c2" Jul 10 18:58:31.240457 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240018 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/13f6e0d5-e879-45c0-b2ec-7b310c717933-host-slash\") pod \"iptables-alerter-g5t7c\" (UID: \"13f6e0d5-e879-45c0-b2ec-7b310c717933\") " pod="openshift-network-operator/iptables-alerter-g5t7c" Jul 10 18:58:31.241349 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240038 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90178079-026e-41e5-94a0-d180a1940cc1-tmp\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.241349 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240095 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eda6e317-2a4b-4591-a5ef-0139d0e57e95-cnibin\") pod \"multus-additional-cni-plugins-z58c2\" (UID: \"eda6e317-2a4b-4591-a5ef-0139d0e57e95\") " pod="openshift-multus/multus-additional-cni-plugins-z58c2" Jul 10 18:58:31.241349 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240105 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eda6e317-2a4b-4591-a5ef-0139d0e57e95-system-cni-dir\") pod \"multus-additional-cni-plugins-z58c2\" (UID: \"eda6e317-2a4b-4591-a5ef-0139d0e57e95\") " pod="openshift-multus/multus-additional-cni-plugins-z58c2" Jul 10 18:58:31.241349 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240136 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-host-var-lib-kubelet\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.241349 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240174 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-hostroot\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.241349 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240191 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-host-run-netns\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.241349 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240228 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3e3a914-d7de-4771-9d68-a7b05ab56b1f-original-pull-secret\") pod \"global-pull-secret-syncer-zv9t9\" (UID: \"e3e3a914-d7de-4771-9d68-a7b05ab56b1f\") " pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:58:31.241349 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240256 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lshlg\" (UniqueName: \"kubernetes.io/projected/13f6e0d5-e879-45c0-b2ec-7b310c717933-kube-api-access-lshlg\") pod \"iptables-alerter-g5t7c\" (UID: \"13f6e0d5-e879-45c0-b2ec-7b310c717933\") " pod="openshift-network-operator/iptables-alerter-g5t7c" Jul 10 18:58:31.241349 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240260 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d85934ea-40c2-4589-aca7-b0e8efca569c-tmp-dir\") pod \"node-resolver-l9qmm\" (UID: \"d85934ea-40c2-4589-aca7-b0e8efca569c\") " pod="openshift-dns/node-resolver-l9qmm" Jul 10 18:58:31.241349 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240278 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.241349 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240309 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d85934ea-40c2-4589-aca7-b0e8efca569c-hosts-file\") pod \"node-resolver-l9qmm\" (UID: \"d85934ea-40c2-4589-aca7-b0e8efca569c\") " pod="openshift-dns/node-resolver-l9qmm" Jul 10 18:58:31.241349 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240336 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a-konnectivity-ca\") pod \"konnectivity-agent-4hn8z\" (UID: \"00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a\") " pod="kube-system/konnectivity-agent-4hn8z" Jul 10 18:58:31.241349 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:31.240354 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 18:58:31.241349 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240389 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eda6e317-2a4b-4591-a5ef-0139d0e57e95-cni-binary-copy\") pod \"multus-additional-cni-plugins-z58c2\" (UID: \"eda6e317-2a4b-4591-a5ef-0139d0e57e95\") " pod="openshift-multus/multus-additional-cni-plugins-z58c2" Jul 10 18:58:31.241349 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240408 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d85934ea-40c2-4589-aca7-b0e8efca569c-hosts-file\") pod \"node-resolver-l9qmm\" (UID: \"d85934ea-40c2-4589-aca7-b0e8efca569c\") " pod="openshift-dns/node-resolver-l9qmm" Jul 10 18:58:31.241349 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:31.240414 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3e3a914-d7de-4771-9d68-a7b05ab56b1f-original-pull-secret podName:e3e3a914-d7de-4771-9d68-a7b05ab56b1f nodeName:}" failed. No retries permitted until 2026-07-10 18:58:31.740397477 +0000 UTC m=+3.070538269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3e3a914-d7de-4771-9d68-a7b05ab56b1f-original-pull-secret") pod "global-pull-secret-syncer-zv9t9" (UID: "e3e3a914-d7de-4771-9d68-a7b05ab56b1f") : object "kube-system"/"original-pull-secret" not registered Jul 10 18:58:31.241349 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240437 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90178079-026e-41e5-94a0-d180a1940cc1-sys\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.242145 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240497 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eda6e317-2a4b-4591-a5ef-0139d0e57e95-os-release\") pod \"multus-additional-cni-plugins-z58c2\" (UID: \"eda6e317-2a4b-4591-a5ef-0139d0e57e95\") " pod="openshift-multus/multus-additional-cni-plugins-z58c2" Jul 10 18:58:31.242145 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240545 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/90178079-026e-41e5-94a0-d180a1940cc1-etc-tuned\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.242145 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240572 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-host-kubelet\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.242145 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240608 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eda6e317-2a4b-4591-a5ef-0139d0e57e95-os-release\") pod \"multus-additional-cni-plugins-z58c2\" (UID: \"eda6e317-2a4b-4591-a5ef-0139d0e57e95\") " pod="openshift-multus/multus-additional-cni-plugins-z58c2" Jul 10 18:58:31.242145 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240647 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-node-log\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.242145 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240704 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eda6e317-2a4b-4591-a5ef-0139d0e57e95-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z58c2\" (UID: \"eda6e317-2a4b-4591-a5ef-0139d0e57e95\") " pod="openshift-multus/multus-additional-cni-plugins-z58c2" Jul 10 18:58:31.242145 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240736 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/13f6e0d5-e879-45c0-b2ec-7b310c717933-iptables-alerter-script\") pod \"iptables-alerter-g5t7c\" (UID: \"13f6e0d5-e879-45c0-b2ec-7b310c717933\") " pod="openshift-network-operator/iptables-alerter-g5t7c" Jul 10 18:58:31.242145 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240764 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/90178079-026e-41e5-94a0-d180a1940cc1-host\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.242145 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240793 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-multus-daemon-config\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.242145 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240819 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e3e3a914-d7de-4771-9d68-a7b05ab56b1f-dbus\") pod \"global-pull-secret-syncer-zv9t9\" (UID: \"e3e3a914-d7de-4771-9d68-a7b05ab56b1f\") " pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:58:31.242145 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240846 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/90178079-026e-41e5-94a0-d180a1940cc1-etc-modprobe-d\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.242145 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240873 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/90178079-026e-41e5-94a0-d180a1940cc1-etc-sysconfig\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.242145 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240873 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eda6e317-2a4b-4591-a5ef-0139d0e57e95-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z58c2\" (UID: \"eda6e317-2a4b-4591-a5ef-0139d0e57e95\") " pod="openshift-multus/multus-additional-cni-plugins-z58c2" Jul 10 18:58:31.242145 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240900 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/90178079-026e-41e5-94a0-d180a1940cc1-run\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.242145 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240911 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a-konnectivity-ca\") pod \"konnectivity-agent-4hn8z\" (UID: \"00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a\") " pod="kube-system/konnectivity-agent-4hn8z" Jul 10 18:58:31.242145 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240926 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-multus-cni-dir\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.242145 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240952 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-host-run-k8s-cni-cncf-io\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.242758 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.240979 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-host-var-lib-cni-multus\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.242758 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.241006 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a22956a-4e96-41de-945c-a851b587a442-ovn-node-metrics-cert\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.242758 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.241032 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slg7w\" (UniqueName: \"kubernetes.io/projected/4526c66d-17d8-4ed2-9782-42b9b2d9215b-kube-api-access-slg7w\") pod \"network-check-target-bl7wj\" (UID: \"4526c66d-17d8-4ed2-9782-42b9b2d9215b\") " pod="openshift-network-diagnostics/network-check-target-bl7wj" Jul 10 18:58:31.242758 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.241068 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd6564ad-086b-4c7b-ada7-99f16bfe03c4-host\") pod \"node-ca-x6sx7\" (UID: \"fd6564ad-086b-4c7b-ada7-99f16bfe03c4\") " pod="openshift-image-registry/node-ca-x6sx7" Jul 10 18:58:31.242758 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.241076 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e3e3a914-d7de-4771-9d68-a7b05ab56b1f-dbus\") pod \"global-pull-secret-syncer-zv9t9\" (UID: \"e3e3a914-d7de-4771-9d68-a7b05ab56b1f\") " pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:58:31.242758 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.241092 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpvvl\" (UniqueName: \"kubernetes.io/projected/90178079-026e-41e5-94a0-d180a1940cc1-kube-api-access-kpvvl\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.242758 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.241132 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd6564ad-086b-4c7b-ada7-99f16bfe03c4-host\") pod \"node-ca-x6sx7\" (UID: \"fd6564ad-086b-4c7b-ada7-99f16bfe03c4\") " pod="openshift-image-registry/node-ca-x6sx7" Jul 10 18:58:31.242758 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.241163 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-cnibin\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.242758 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.241193 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-etc-openvswitch\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.242758 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.241226 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/eda6e317-2a4b-4591-a5ef-0139d0e57e95-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z58c2\" (UID: \"eda6e317-2a4b-4591-a5ef-0139d0e57e95\") " pod="openshift-multus/multus-additional-cni-plugins-z58c2" Jul 10 18:58:31.242758 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.241269 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/eda6e317-2a4b-4591-a5ef-0139d0e57e95-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z58c2\" (UID: \"eda6e317-2a4b-4591-a5ef-0139d0e57e95\") " pod="openshift-multus/multus-additional-cni-plugins-z58c2" Jul 10 18:58:31.242758 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.241316 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/13f6e0d5-e879-45c0-b2ec-7b310c717933-iptables-alerter-script\") pod \"iptables-alerter-g5t7c\" (UID: \"13f6e0d5-e879-45c0-b2ec-7b310c717933\") " pod="openshift-network-operator/iptables-alerter-g5t7c" Jul 10 18:58:31.242758 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.241364 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90178079-026e-41e5-94a0-d180a1940cc1-var-lib-kubelet\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.242758 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.241396 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-cni-binary-copy\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.242758 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.241421 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5-sys-fs\") pod \"aws-ebs-csi-driver-node-spzdj\" (UID: \"262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spzdj" Jul 10 18:58:31.242758 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.241447 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-systemd-units\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.242758 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.241501 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2a22956a-4e96-41de-945c-a851b587a442-ovnkube-config\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.243336 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.241527 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5-registration-dir\") pod \"aws-ebs-csi-driver-node-spzdj\" (UID: \"262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spzdj" Jul 10 18:58:31.243336 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.241566 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-run-ovn\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.243336 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.241612 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-host-run-ovn-kubernetes\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.243336 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.241846 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/eda6e317-2a4b-4591-a5ef-0139d0e57e95-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z58c2\" (UID: \"eda6e317-2a4b-4591-a5ef-0139d0e57e95\") " pod="openshift-multus/multus-additional-cni-plugins-z58c2" Jul 10 18:58:31.243336 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.241860 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/eda6e317-2a4b-4591-a5ef-0139d0e57e95-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z58c2\" (UID: \"eda6e317-2a4b-4591-a5ef-0139d0e57e95\") " pod="openshift-multus/multus-additional-cni-plugins-z58c2" Jul 10 18:58:31.243535 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.243518 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a-agent-certs\") pod \"konnectivity-agent-4hn8z\" (UID: \"00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a\") " pod="kube-system/konnectivity-agent-4hn8z" Jul 10 18:58:31.251580 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:31.250300 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 18:58:31.251580 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:31.250341 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 18:58:31.251580 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:31.250369 2560 projected.go:196] Error preparing data for projected volume kube-api-access-slg7w for pod openshift-network-diagnostics/network-check-target-bl7wj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:58:31.251580 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:31.250458 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4526c66d-17d8-4ed2-9782-42b9b2d9215b-kube-api-access-slg7w podName:4526c66d-17d8-4ed2-9782-42b9b2d9215b nodeName:}" failed. No retries permitted until 2026-07-10 18:58:31.750425332 +0000 UTC m=+3.080566144 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-slg7w" (UniqueName: "kubernetes.io/projected/4526c66d-17d8-4ed2-9782-42b9b2d9215b-kube-api-access-slg7w") pod "network-check-target-bl7wj" (UID: "4526c66d-17d8-4ed2-9782-42b9b2d9215b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:58:31.251894 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.251810 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg6x9\" (UniqueName: \"kubernetes.io/projected/d85934ea-40c2-4589-aca7-b0e8efca569c-kube-api-access-tg6x9\") pod \"node-resolver-l9qmm\" (UID: \"d85934ea-40c2-4589-aca7-b0e8efca569c\") " pod="openshift-dns/node-resolver-l9qmm" Jul 10 18:58:31.252073 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.252050 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmgrk\" (UniqueName: \"kubernetes.io/projected/eda6e317-2a4b-4591-a5ef-0139d0e57e95-kube-api-access-fmgrk\") pod \"multus-additional-cni-plugins-z58c2\" (UID: \"eda6e317-2a4b-4591-a5ef-0139d0e57e95\") " pod="openshift-multus/multus-additional-cni-plugins-z58c2" Jul 10 18:58:31.252628 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.252606 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s44cc\" (UniqueName: \"kubernetes.io/projected/fd6564ad-086b-4c7b-ada7-99f16bfe03c4-kube-api-access-s44cc\") pod \"node-ca-x6sx7\" (UID: \"fd6564ad-086b-4c7b-ada7-99f16bfe03c4\") " pod="openshift-image-registry/node-ca-x6sx7" Jul 10 18:58:31.253437 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.253415 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lshlg\" (UniqueName: \"kubernetes.io/projected/13f6e0d5-e879-45c0-b2ec-7b310c717933-kube-api-access-lshlg\") pod \"iptables-alerter-g5t7c\" (UID: \"13f6e0d5-e879-45c0-b2ec-7b310c717933\") " pod="openshift-network-operator/iptables-alerter-g5t7c" Jul 10 18:58:31.253996 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.253973 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smmwh\" (UniqueName: \"kubernetes.io/projected/b1bf574f-a4e2-4418-8f18-95fa0c80ca1e-kube-api-access-smmwh\") pod \"network-metrics-daemon-v9b95\" (UID: \"b1bf574f-a4e2-4418-8f18-95fa0c80ca1e\") " pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 18:58:31.265281 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:31.265248 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb6a2004a5a296385f19413f1682752d.slice/crio-cd02a4487e18e67dc18ac4fb04809b45d1e82f7099a184509ab78dc56bc9e511 WatchSource:0}: Error finding container cd02a4487e18e67dc18ac4fb04809b45d1e82f7099a184509ab78dc56bc9e511: Status 404 returned error can't find the container with id cd02a4487e18e67dc18ac4fb04809b45d1e82f7099a184509ab78dc56bc9e511 Jul 10 18:58:31.265783 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:31.265765 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b1acdcd9ea09cc31366e606e569025.slice/crio-4f6c33f909bf3dda1866cdc80bb778500e3c93fab33daabc55415d6f91b20c4d WatchSource:0}: Error finding container 4f6c33f909bf3dda1866cdc80bb778500e3c93fab33daabc55415d6f91b20c4d: Status 404 returned error can't find the container with id 4f6c33f909bf3dda1866cdc80bb778500e3c93fab33daabc55415d6f91b20c4d Jul 10 18:58:31.272413 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.272394 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 18:58:31.314761 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.314662 2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 18:58:31.342042 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342006 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpvvl\" (UniqueName: \"kubernetes.io/projected/90178079-026e-41e5-94a0-d180a1940cc1-kube-api-access-kpvvl\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.342042 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342046 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-cnibin\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.342301 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342069 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-etc-openvswitch\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.342301 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342094 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90178079-026e-41e5-94a0-d180a1940cc1-var-lib-kubelet\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.342301 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342137 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-cni-binary-copy\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.342301 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342155 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90178079-026e-41e5-94a0-d180a1940cc1-var-lib-kubelet\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.342301 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342157 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-cnibin\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.342301 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342182 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5-sys-fs\") pod \"aws-ebs-csi-driver-node-spzdj\" (UID: \"262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spzdj" Jul 10 18:58:31.342301 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342200 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-etc-openvswitch\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.342301 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342209 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-systemd-units\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.342301 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342237 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2a22956a-4e96-41de-945c-a851b587a442-ovnkube-config\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.342301 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342257 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5-sys-fs\") pod \"aws-ebs-csi-driver-node-spzdj\" (UID: \"262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spzdj" Jul 10 18:58:31.342301 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342267 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5-registration-dir\") pod \"aws-ebs-csi-driver-node-spzdj\" (UID: \"262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spzdj" Jul 10 18:58:31.342301 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342238 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-systemd-units\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.342301 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342295 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-run-ovn\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.343004 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342332 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5-registration-dir\") pod \"aws-ebs-csi-driver-node-spzdj\" (UID: \"262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spzdj" Jul 10 18:58:31.343004 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342335 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-host-run-ovn-kubernetes\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.343004 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342370 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/90178079-026e-41e5-94a0-d180a1940cc1-etc-systemd\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.343004 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342375 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-run-ovn\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.343004 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342413 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-host-cni-netd\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.343004 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342432 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/90178079-026e-41e5-94a0-d180a1940cc1-etc-systemd\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.343004 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342414 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-host-run-ovn-kubernetes\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.343004 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342446 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/90178079-026e-41e5-94a0-d180a1940cc1-etc-sysctl-d\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.343004 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342484 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-host-cni-netd\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.343004 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342505 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/90178079-026e-41e5-94a0-d180a1940cc1-etc-sysctl-conf\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.343004 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342529 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-os-release\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.343004 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342550 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-run-openvswitch\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.343004 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342571 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-log-socket\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.343004 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342643 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/90178079-026e-41e5-94a0-d180a1940cc1-etc-sysctl-conf\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.343004 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342649 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-run-openvswitch\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.343004 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342652 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/90178079-026e-41e5-94a0-d180a1940cc1-lib-modules\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.343004 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342691 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-log-socket\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.343004 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342690 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-multus-socket-dir-parent\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.343892 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342693 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-os-release\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.343892 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342737 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-multus-socket-dir-parent\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.343892 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342749 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/90178079-026e-41e5-94a0-d180a1940cc1-lib-modules\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.343892 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342749 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-host-run-netns\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.343892 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342784 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-host-run-netns\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.343892 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342784 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-spzdj\" (UID: \"262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spzdj" Jul 10 18:58:31.343892 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342798 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/90178079-026e-41e5-94a0-d180a1940cc1-etc-sysctl-d\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.343892 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342826 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-run-systemd\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.343892 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342823 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-spzdj\" (UID: \"262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spzdj" Jul 10 18:58:31.343892 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342843 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-cni-binary-copy\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.343892 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342853 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-run-systemd\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.343892 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342890 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-host-var-lib-cni-bin\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.343892 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342916 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-multus-conf-dir\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.343892 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342935 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-host-var-lib-cni-bin\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.343892 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342942 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-host-cni-bin\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.343892 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342960 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-multus-conf-dir\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.343892 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342967 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-host-cni-bin\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.343892 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342972 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90178079-026e-41e5-94a0-d180a1940cc1-etc-kubernetes\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.344824 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.342914 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2a22956a-4e96-41de-945c-a851b587a442-ovnkube-config\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.344824 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343001 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-system-cni-dir\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.344824 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343026 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90178079-026e-41e5-94a0-d180a1940cc1-etc-kubernetes\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.344824 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343028 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-etc-kubernetes\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.344824 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343061 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-etc-kubernetes\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.344824 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343081 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sqh4\" (UniqueName: \"kubernetes.io/projected/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-kube-api-access-4sqh4\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.344824 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343084 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-system-cni-dir\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.344824 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343107 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5-etc-selinux\") pod \"aws-ebs-csi-driver-node-spzdj\" (UID: \"262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spzdj" Jul 10 18:58:31.344824 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343124 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rbr2\" (UniqueName: \"kubernetes.io/projected/262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5-kube-api-access-9rbr2\") pod \"aws-ebs-csi-driver-node-spzdj\" (UID: \"262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spzdj" Jul 10 18:58:31.344824 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343143 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-host-slash\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.344824 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343169 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-host-run-multus-certs\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.344824 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343193 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5-socket-dir\") pod \"aws-ebs-csi-driver-node-spzdj\" (UID: \"262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spzdj" Jul 10 18:58:31.344824 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343212 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-host-slash\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.344824 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343218 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-var-lib-openvswitch\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.344824 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343243 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2a22956a-4e96-41de-945c-a851b587a442-env-overrides\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.344824 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343260 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-host-run-multus-certs\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.344824 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343268 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2a22956a-4e96-41de-945c-a851b587a442-ovnkube-script-lib\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.345818 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343294 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldjlz\" (UniqueName: \"kubernetes.io/projected/2a22956a-4e96-41de-945c-a851b587a442-kube-api-access-ldjlz\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.345818 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343301 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5-etc-selinux\") pod \"aws-ebs-csi-driver-node-spzdj\" (UID: \"262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spzdj" Jul 10 18:58:31.345818 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343336 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5-socket-dir\") pod \"aws-ebs-csi-driver-node-spzdj\" (UID: \"262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spzdj" Jul 10 18:58:31.345818 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343344 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5-device-dir\") pod \"aws-ebs-csi-driver-node-spzdj\" (UID: \"262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spzdj" Jul 10 18:58:31.345818 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343393 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-var-lib-openvswitch\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.345818 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343434 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5-device-dir\") pod \"aws-ebs-csi-driver-node-spzdj\" (UID: \"262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spzdj" Jul 10 18:58:31.345818 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343436 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90178079-026e-41e5-94a0-d180a1940cc1-tmp\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.345818 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343489 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-host-var-lib-kubelet\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.345818 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343519 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-hostroot\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.345818 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343543 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-host-var-lib-kubelet\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.345818 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343545 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-host-run-netns\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.345818 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343573 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-host-run-netns\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.345818 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343585 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.345818 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343611 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-hostroot\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.345818 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343612 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90178079-026e-41e5-94a0-d180a1940cc1-sys\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.345818 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343638 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90178079-026e-41e5-94a0-d180a1940cc1-sys\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.345818 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343639 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/90178079-026e-41e5-94a0-d180a1940cc1-etc-tuned\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.345818 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343665 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-host-kubelet\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.346465 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343690 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-node-log\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.346465 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343699 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.346465 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343708 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2a22956a-4e96-41de-945c-a851b587a442-env-overrides\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.346465 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343718 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/90178079-026e-41e5-94a0-d180a1940cc1-host\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.346465 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343744 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-host-kubelet\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.346465 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343749 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-multus-daemon-config\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.346465 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343779 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2a22956a-4e96-41de-945c-a851b587a442-node-log\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.346465 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343787 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/90178079-026e-41e5-94a0-d180a1940cc1-etc-modprobe-d\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.346465 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343789 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/90178079-026e-41e5-94a0-d180a1940cc1-host\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.346465 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343814 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/90178079-026e-41e5-94a0-d180a1940cc1-etc-sysconfig\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.346465 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343840 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/90178079-026e-41e5-94a0-d180a1940cc1-run\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.346465 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343862 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-multus-cni-dir\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.346465 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343890 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-host-run-k8s-cni-cncf-io\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.346465 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343890 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2a22956a-4e96-41de-945c-a851b587a442-ovnkube-script-lib\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.346465 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343903 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/90178079-026e-41e5-94a0-d180a1940cc1-etc-modprobe-d\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.346465 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343939 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/90178079-026e-41e5-94a0-d180a1940cc1-run\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.346465 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343953 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-host-run-k8s-cni-cncf-io\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.346465 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343974 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-multus-cni-dir\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.347023 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.343973 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-host-var-lib-cni-multus\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.347023 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.344009 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/90178079-026e-41e5-94a0-d180a1940cc1-etc-sysconfig\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.347023 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.344017 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a22956a-4e96-41de-945c-a851b587a442-ovn-node-metrics-cert\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.347023 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.344010 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-host-var-lib-cni-multus\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.347023 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.344229 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-multus-daemon-config\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.347023 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.345890 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/90178079-026e-41e5-94a0-d180a1940cc1-etc-tuned\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.347023 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.346108 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a22956a-4e96-41de-945c-a851b587a442-ovn-node-metrics-cert\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.347023 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.346142 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90178079-026e-41e5-94a0-d180a1940cc1-tmp\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.362255 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.362225 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpvvl\" (UniqueName: \"kubernetes.io/projected/90178079-026e-41e5-94a0-d180a1940cc1-kube-api-access-kpvvl\") pod \"tuned-cv4cv\" (UID: \"90178079-026e-41e5-94a0-d180a1940cc1\") " pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.371114 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.371093 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rbr2\" (UniqueName: \"kubernetes.io/projected/262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5-kube-api-access-9rbr2\") pod \"aws-ebs-csi-driver-node-spzdj\" (UID: \"262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spzdj" Jul 10 18:58:31.375743 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.375722 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sqh4\" (UniqueName: \"kubernetes.io/projected/55b10423-8cf8-4f8f-bdb7-b5c7a89165a8-kube-api-access-4sqh4\") pod \"multus-sx2wz\" (UID: \"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8\") " pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.376125 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.376108 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldjlz\" (UniqueName: \"kubernetes.io/projected/2a22956a-4e96-41de-945c-a851b587a442-kube-api-access-ldjlz\") pod \"ovnkube-node-9l52t\" (UID: \"2a22956a-4e96-41de-945c-a851b587a442\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.408609 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.408562 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g5t7c" Jul 10 18:58:31.417738 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:31.417714 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f6e0d5_e879_45c0_b2ec_7b310c717933.slice/crio-58d37e1b89f4009f236f63767e3914d7abc687ab37fee1fe89a9d1460a27cf40 WatchSource:0}: Error finding container 58d37e1b89f4009f236f63767e3914d7abc687ab37fee1fe89a9d1460a27cf40: Status 404 returned error can't find the container with id 58d37e1b89f4009f236f63767e3914d7abc687ab37fee1fe89a9d1460a27cf40 Jul 10 18:58:31.428092 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.428069 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l9qmm" Jul 10 18:58:31.436982 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:31.436953 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd85934ea_40c2_4589_aca7_b0e8efca569c.slice/crio-124ac59b68bdfcec57b50872131fc27844afa48c70fa7edb829facf8390dfdc4 WatchSource:0}: Error finding container 124ac59b68bdfcec57b50872131fc27844afa48c70fa7edb829facf8390dfdc4: Status 404 returned error can't find the container with id 124ac59b68bdfcec57b50872131fc27844afa48c70fa7edb829facf8390dfdc4 Jul 10 18:58:31.448145 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.448125 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x6sx7" Jul 10 18:58:31.452009 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.451694 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4hn8z" Jul 10 18:58:31.459584 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:31.459557 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd6564ad_086b_4c7b_ada7_99f16bfe03c4.slice/crio-ba04f86a4c61cc494372728239fc0a6aa72c16b6a8608b254a6ec88bdee9ead6 WatchSource:0}: Error finding container ba04f86a4c61cc494372728239fc0a6aa72c16b6a8608b254a6ec88bdee9ead6: Status 404 returned error can't find the container with id ba04f86a4c61cc494372728239fc0a6aa72c16b6a8608b254a6ec88bdee9ead6 Jul 10 18:58:31.460732 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:31.460705 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00d8d68a_3d1f_4b7f_b9a7_ef54b3489e4a.slice/crio-3e2ab77f9e54838cc786a08816ea49f0ab45c20d92f61cf2ca9d6204b81a70ae WatchSource:0}: Error finding container 3e2ab77f9e54838cc786a08816ea49f0ab45c20d92f61cf2ca9d6204b81a70ae: Status 404 returned error can't find the container with id 3e2ab77f9e54838cc786a08816ea49f0ab45c20d92f61cf2ca9d6204b81a70ae Jul 10 18:58:31.470179 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.470158 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z58c2" Jul 10 18:58:31.478554 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:31.478532 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda6e317_2a4b_4591_a5ef_0139d0e57e95.slice/crio-25f638729609fe264c4e0abd225c80ea7a8b4401f2513501190c89009fdf2905 WatchSource:0}: Error finding container 25f638729609fe264c4e0abd225c80ea7a8b4401f2513501190c89009fdf2905: Status 404 returned error can't find the container with id 25f638729609fe264c4e0abd225c80ea7a8b4401f2513501190c89009fdf2905 Jul 10 18:58:31.486401 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.486382 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" Jul 10 18:58:31.493544 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:31.493517 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90178079_026e_41e5_94a0_d180a1940cc1.slice/crio-89432089ab155767f3adc9d0062a6c225e67b27bc7a03f22ade79703a5649085 WatchSource:0}: Error finding container 89432089ab155767f3adc9d0062a6c225e67b27bc7a03f22ade79703a5649085: Status 404 returned error can't find the container with id 89432089ab155767f3adc9d0062a6c225e67b27bc7a03f22ade79703a5649085 Jul 10 18:58:31.493662 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.493640 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spzdj" Jul 10 18:58:31.501572 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:31.501545 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod262f3ffd_d7bb_401e_acc1_e9ae04dc3ea5.slice/crio-906b6205c8f652cf34f40f1a114fbc27a6b8ebd61f51f20f3a0f510c3fde43b5 WatchSource:0}: Error finding container 906b6205c8f652cf34f40f1a114fbc27a6b8ebd61f51f20f3a0f510c3fde43b5: Status 404 returned error can't find the container with id 906b6205c8f652cf34f40f1a114fbc27a6b8ebd61f51f20f3a0f510c3fde43b5 Jul 10 18:58:31.527693 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.527656 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sx2wz" Jul 10 18:58:31.531394 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.531376 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:31.537717 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:31.537689 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55b10423_8cf8_4f8f_bdb7_b5c7a89165a8.slice/crio-a101c2aa4b0fb9fa6f5bf04c897e422c2cda4d8358ec5a79f6f1ab101b300ce3 WatchSource:0}: Error finding container a101c2aa4b0fb9fa6f5bf04c897e422c2cda4d8358ec5a79f6f1ab101b300ce3: Status 404 returned error can't find the container with id a101c2aa4b0fb9fa6f5bf04c897e422c2cda4d8358ec5a79f6f1ab101b300ce3 Jul 10 18:58:31.540899 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:58:31.540867 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a22956a_4e96_41de_945c_a851b587a442.slice/crio-00c37ded761d47cc1982f508322c2de48093dea03f5e2a97a9da633312d9e17a WatchSource:0}: Error finding container 00c37ded761d47cc1982f508322c2de48093dea03f5e2a97a9da633312d9e17a: Status 404 returned error can't find the container with id 00c37ded761d47cc1982f508322c2de48093dea03f5e2a97a9da633312d9e17a Jul 10 18:58:31.747936 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.747835 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1bf574f-a4e2-4418-8f18-95fa0c80ca1e-metrics-certs\") pod \"network-metrics-daemon-v9b95\" (UID: \"b1bf574f-a4e2-4418-8f18-95fa0c80ca1e\") " pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 18:58:31.747936 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.747901 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3e3a914-d7de-4771-9d68-a7b05ab56b1f-original-pull-secret\") pod \"global-pull-secret-syncer-zv9t9\" (UID: \"e3e3a914-d7de-4771-9d68-a7b05ab56b1f\") " pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:58:31.748148 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:31.747991 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:58:31.748148 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:31.748015 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 18:58:31.748148 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:31.748073 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3e3a914-d7de-4771-9d68-a7b05ab56b1f-original-pull-secret podName:e3e3a914-d7de-4771-9d68-a7b05ab56b1f nodeName:}" failed. No retries permitted until 2026-07-10 18:58:32.748054848 +0000 UTC m=+4.078195644 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3e3a914-d7de-4771-9d68-a7b05ab56b1f-original-pull-secret") pod "global-pull-secret-syncer-zv9t9" (UID: "e3e3a914-d7de-4771-9d68-a7b05ab56b1f") : object "kube-system"/"original-pull-secret" not registered Jul 10 18:58:31.748148 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:31.748095 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1bf574f-a4e2-4418-8f18-95fa0c80ca1e-metrics-certs podName:b1bf574f-a4e2-4418-8f18-95fa0c80ca1e nodeName:}" failed. No retries permitted until 2026-07-10 18:58:32.748084346 +0000 UTC m=+4.078225143 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b1bf574f-a4e2-4418-8f18-95fa0c80ca1e-metrics-certs") pod "network-metrics-daemon-v9b95" (UID: "b1bf574f-a4e2-4418-8f18-95fa0c80ca1e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:58:31.849238 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:31.849157 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slg7w\" (UniqueName: \"kubernetes.io/projected/4526c66d-17d8-4ed2-9782-42b9b2d9215b-kube-api-access-slg7w\") pod \"network-check-target-bl7wj\" (UID: \"4526c66d-17d8-4ed2-9782-42b9b2d9215b\") " pod="openshift-network-diagnostics/network-check-target-bl7wj" Jul 10 18:58:31.849417 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:31.849360 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 18:58:31.849417 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:31.849390 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 18:58:31.849417 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:31.849404 2560 projected.go:196] Error preparing data for projected volume kube-api-access-slg7w for pod openshift-network-diagnostics/network-check-target-bl7wj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:58:31.849642 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:31.849488 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4526c66d-17d8-4ed2-9782-42b9b2d9215b-kube-api-access-slg7w podName:4526c66d-17d8-4ed2-9782-42b9b2d9215b nodeName:}" failed. No retries permitted until 2026-07-10 18:58:32.849451352 +0000 UTC m=+4.179592160 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-slg7w" (UniqueName: "kubernetes.io/projected/4526c66d-17d8-4ed2-9782-42b9b2d9215b-kube-api-access-slg7w") pod "network-check-target-bl7wj" (UID: "4526c66d-17d8-4ed2-9782-42b9b2d9215b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:58:32.133926 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:32.133804 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 18:53:31 +0000 UTC" deadline="2027-12-21 04:45:07.180342808 +0000 UTC" Jul 10 18:58:32.133926 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:32.133877 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12681h46m35.046470598s" Jul 10 18:58:32.246226 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:32.246181 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:58:32.246429 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:32.246336 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9t9" podUID="e3e3a914-d7de-4771-9d68-a7b05ab56b1f" Jul 10 18:58:32.246823 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:32.246791 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 18:58:32.246934 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:32.246914 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9b95" podUID="b1bf574f-a4e2-4418-8f18-95fa0c80ca1e" Jul 10 18:58:32.264422 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:32.264326 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g5t7c" event={"ID":"13f6e0d5-e879-45c0-b2ec-7b310c717933","Type":"ContainerStarted","Data":"58d37e1b89f4009f236f63767e3914d7abc687ab37fee1fe89a9d1460a27cf40"} Jul 10 18:58:32.269495 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:32.269459 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-213.ec2.internal" event={"ID":"eb6a2004a5a296385f19413f1682752d","Type":"ContainerStarted","Data":"cd02a4487e18e67dc18ac4fb04809b45d1e82f7099a184509ab78dc56bc9e511"} Jul 10 18:58:32.278372 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:32.278335 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-213.ec2.internal" event={"ID":"14b1acdcd9ea09cc31366e606e569025","Type":"ContainerStarted","Data":"4f6c33f909bf3dda1866cdc80bb778500e3c93fab33daabc55415d6f91b20c4d"} Jul 10 18:58:32.282306 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:32.282270 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sx2wz" event={"ID":"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8","Type":"ContainerStarted","Data":"a101c2aa4b0fb9fa6f5bf04c897e422c2cda4d8358ec5a79f6f1ab101b300ce3"} Jul 10 18:58:32.287961 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:32.287925 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spzdj" event={"ID":"262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5","Type":"ContainerStarted","Data":"906b6205c8f652cf34f40f1a114fbc27a6b8ebd61f51f20f3a0f510c3fde43b5"} Jul 10 18:58:32.295851 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:32.295816 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z58c2" event={"ID":"eda6e317-2a4b-4591-a5ef-0139d0e57e95","Type":"ContainerStarted","Data":"25f638729609fe264c4e0abd225c80ea7a8b4401f2513501190c89009fdf2905"} Jul 10 18:58:32.307507 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:32.307472 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4hn8z" event={"ID":"00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a","Type":"ContainerStarted","Data":"3e2ab77f9e54838cc786a08816ea49f0ab45c20d92f61cf2ca9d6204b81a70ae"} Jul 10 18:58:32.330203 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:32.330162 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l9qmm" event={"ID":"d85934ea-40c2-4589-aca7-b0e8efca569c","Type":"ContainerStarted","Data":"124ac59b68bdfcec57b50872131fc27844afa48c70fa7edb829facf8390dfdc4"} Jul 10 18:58:32.339861 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:32.339819 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" event={"ID":"2a22956a-4e96-41de-945c-a851b587a442","Type":"ContainerStarted","Data":"00c37ded761d47cc1982f508322c2de48093dea03f5e2a97a9da633312d9e17a"} Jul 10 18:58:32.350584 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:32.349142 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" event={"ID":"90178079-026e-41e5-94a0-d180a1940cc1","Type":"ContainerStarted","Data":"89432089ab155767f3adc9d0062a6c225e67b27bc7a03f22ade79703a5649085"} Jul 10 18:58:32.355164 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:32.355081 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x6sx7" event={"ID":"fd6564ad-086b-4c7b-ada7-99f16bfe03c4","Type":"ContainerStarted","Data":"ba04f86a4c61cc494372728239fc0a6aa72c16b6a8608b254a6ec88bdee9ead6"} Jul 10 18:58:32.758193 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:32.758153 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3e3a914-d7de-4771-9d68-a7b05ab56b1f-original-pull-secret\") pod \"global-pull-secret-syncer-zv9t9\" (UID: \"e3e3a914-d7de-4771-9d68-a7b05ab56b1f\") " pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:58:32.758422 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:32.758257 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1bf574f-a4e2-4418-8f18-95fa0c80ca1e-metrics-certs\") pod \"network-metrics-daemon-v9b95\" (UID: \"b1bf574f-a4e2-4418-8f18-95fa0c80ca1e\") " pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 18:58:32.758422 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:32.758393 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:58:32.758422 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:32.758402 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 18:58:32.758640 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:32.758458 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1bf574f-a4e2-4418-8f18-95fa0c80ca1e-metrics-certs podName:b1bf574f-a4e2-4418-8f18-95fa0c80ca1e nodeName:}" failed. No retries permitted until 2026-07-10 18:58:34.758439706 +0000 UTC m=+6.088580498 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b1bf574f-a4e2-4418-8f18-95fa0c80ca1e-metrics-certs") pod "network-metrics-daemon-v9b95" (UID: "b1bf574f-a4e2-4418-8f18-95fa0c80ca1e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:58:32.758640 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:32.758478 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3e3a914-d7de-4771-9d68-a7b05ab56b1f-original-pull-secret podName:e3e3a914-d7de-4771-9d68-a7b05ab56b1f nodeName:}" failed. No retries permitted until 2026-07-10 18:58:34.758468214 +0000 UTC m=+6.088609010 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3e3a914-d7de-4771-9d68-a7b05ab56b1f-original-pull-secret") pod "global-pull-secret-syncer-zv9t9" (UID: "e3e3a914-d7de-4771-9d68-a7b05ab56b1f") : object "kube-system"/"original-pull-secret" not registered Jul 10 18:58:32.859359 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:32.858715 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slg7w\" (UniqueName: \"kubernetes.io/projected/4526c66d-17d8-4ed2-9782-42b9b2d9215b-kube-api-access-slg7w\") pod \"network-check-target-bl7wj\" (UID: \"4526c66d-17d8-4ed2-9782-42b9b2d9215b\") " pod="openshift-network-diagnostics/network-check-target-bl7wj" Jul 10 18:58:32.859359 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:32.858916 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 18:58:32.859359 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:32.858938 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 18:58:32.859359 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:32.858953 2560 projected.go:196] Error preparing data for projected volume kube-api-access-slg7w for pod openshift-network-diagnostics/network-check-target-bl7wj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:58:32.859359 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:32.859011 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4526c66d-17d8-4ed2-9782-42b9b2d9215b-kube-api-access-slg7w podName:4526c66d-17d8-4ed2-9782-42b9b2d9215b nodeName:}" failed. No retries permitted until 2026-07-10 18:58:34.858991673 +0000 UTC m=+6.189132466 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-slg7w" (UniqueName: "kubernetes.io/projected/4526c66d-17d8-4ed2-9782-42b9b2d9215b-kube-api-access-slg7w") pod "network-check-target-bl7wj" (UID: "4526c66d-17d8-4ed2-9782-42b9b2d9215b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:58:33.135172 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:33.135073 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 18:53:31 +0000 UTC" deadline="2028-01-27 18:07:41.758206886 +0000 UTC" Jul 10 18:58:33.135172 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:33.135118 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13583h9m8.623093478s" Jul 10 18:58:33.247610 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:33.246642 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bl7wj" Jul 10 18:58:33.247610 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:33.246793 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bl7wj" podUID="4526c66d-17d8-4ed2-9782-42b9b2d9215b" Jul 10 18:58:34.247302 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:34.246460 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:58:34.247302 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:34.246630 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9t9" podUID="e3e3a914-d7de-4771-9d68-a7b05ab56b1f" Jul 10 18:58:34.247302 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:34.247052 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 18:58:34.247302 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:34.247195 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9b95" podUID="b1bf574f-a4e2-4418-8f18-95fa0c80ca1e" Jul 10 18:58:34.784662 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:34.784518 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1bf574f-a4e2-4418-8f18-95fa0c80ca1e-metrics-certs\") pod \"network-metrics-daemon-v9b95\" (UID: \"b1bf574f-a4e2-4418-8f18-95fa0c80ca1e\") " pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 18:58:34.784662 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:34.784603 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3e3a914-d7de-4771-9d68-a7b05ab56b1f-original-pull-secret\") pod \"global-pull-secret-syncer-zv9t9\" (UID: \"e3e3a914-d7de-4771-9d68-a7b05ab56b1f\") " pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:58:34.784944 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:34.784790 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 18:58:34.784944 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:34.784854 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3e3a914-d7de-4771-9d68-a7b05ab56b1f-original-pull-secret podName:e3e3a914-d7de-4771-9d68-a7b05ab56b1f nodeName:}" failed. No retries permitted until 2026-07-10 18:58:38.784835222 +0000 UTC m=+10.114976013 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3e3a914-d7de-4771-9d68-a7b05ab56b1f-original-pull-secret") pod "global-pull-secret-syncer-zv9t9" (UID: "e3e3a914-d7de-4771-9d68-a7b05ab56b1f") : object "kube-system"/"original-pull-secret" not registered Jul 10 18:58:34.785286 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:34.785265 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:58:34.785355 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:34.785319 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1bf574f-a4e2-4418-8f18-95fa0c80ca1e-metrics-certs podName:b1bf574f-a4e2-4418-8f18-95fa0c80ca1e nodeName:}" failed. No retries permitted until 2026-07-10 18:58:38.785305726 +0000 UTC m=+10.115446530 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b1bf574f-a4e2-4418-8f18-95fa0c80ca1e-metrics-certs") pod "network-metrics-daemon-v9b95" (UID: "b1bf574f-a4e2-4418-8f18-95fa0c80ca1e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:58:34.886154 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:34.886112 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slg7w\" (UniqueName: \"kubernetes.io/projected/4526c66d-17d8-4ed2-9782-42b9b2d9215b-kube-api-access-slg7w\") pod \"network-check-target-bl7wj\" (UID: \"4526c66d-17d8-4ed2-9782-42b9b2d9215b\") " pod="openshift-network-diagnostics/network-check-target-bl7wj" Jul 10 18:58:34.886341 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:34.886319 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 18:58:34.886417 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:34.886341 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 18:58:34.886417 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:34.886356 2560 projected.go:196] Error preparing data for projected volume kube-api-access-slg7w for pod openshift-network-diagnostics/network-check-target-bl7wj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:58:34.886539 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:34.886474 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4526c66d-17d8-4ed2-9782-42b9b2d9215b-kube-api-access-slg7w podName:4526c66d-17d8-4ed2-9782-42b9b2d9215b nodeName:}" failed. No retries permitted until 2026-07-10 18:58:38.886403223 +0000 UTC m=+10.216544024 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-slg7w" (UniqueName: "kubernetes.io/projected/4526c66d-17d8-4ed2-9782-42b9b2d9215b-kube-api-access-slg7w") pod "network-check-target-bl7wj" (UID: "4526c66d-17d8-4ed2-9782-42b9b2d9215b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:58:35.246848 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:35.246768 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bl7wj" Jul 10 18:58:35.247014 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:35.246905 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bl7wj" podUID="4526c66d-17d8-4ed2-9782-42b9b2d9215b" Jul 10 18:58:36.246748 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:36.246713 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 18:58:36.247239 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:36.246874 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9b95" podUID="b1bf574f-a4e2-4418-8f18-95fa0c80ca1e" Jul 10 18:58:36.247239 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:36.246959 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:58:36.247239 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:36.247061 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9t9" podUID="e3e3a914-d7de-4771-9d68-a7b05ab56b1f" Jul 10 18:58:37.246350 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:37.246315 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bl7wj" Jul 10 18:58:37.246566 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:37.246478 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bl7wj" podUID="4526c66d-17d8-4ed2-9782-42b9b2d9215b" Jul 10 18:58:38.246560 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:38.246513 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:58:38.246560 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:38.246531 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 18:58:38.247087 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:38.246658 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9t9" podUID="e3e3a914-d7de-4771-9d68-a7b05ab56b1f" Jul 10 18:58:38.247087 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:38.246802 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9b95" podUID="b1bf574f-a4e2-4418-8f18-95fa0c80ca1e" Jul 10 18:58:38.820029 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:38.819707 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1bf574f-a4e2-4418-8f18-95fa0c80ca1e-metrics-certs\") pod \"network-metrics-daemon-v9b95\" (UID: \"b1bf574f-a4e2-4418-8f18-95fa0c80ca1e\") " pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 18:58:38.820029 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:38.819778 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3e3a914-d7de-4771-9d68-a7b05ab56b1f-original-pull-secret\") pod \"global-pull-secret-syncer-zv9t9\" (UID: \"e3e3a914-d7de-4771-9d68-a7b05ab56b1f\") " pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:58:38.820029 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:38.819860 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:58:38.820029 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:38.819923 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1bf574f-a4e2-4418-8f18-95fa0c80ca1e-metrics-certs podName:b1bf574f-a4e2-4418-8f18-95fa0c80ca1e nodeName:}" failed. No retries permitted until 2026-07-10 18:58:46.819906273 +0000 UTC m=+18.150047061 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b1bf574f-a4e2-4418-8f18-95fa0c80ca1e-metrics-certs") pod "network-metrics-daemon-v9b95" (UID: "b1bf574f-a4e2-4418-8f18-95fa0c80ca1e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:58:38.820029 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:38.819940 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 18:58:38.820029 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:38.819997 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3e3a914-d7de-4771-9d68-a7b05ab56b1f-original-pull-secret podName:e3e3a914-d7de-4771-9d68-a7b05ab56b1f nodeName:}" failed. No retries permitted until 2026-07-10 18:58:46.819979904 +0000 UTC m=+18.150120693 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3e3a914-d7de-4771-9d68-a7b05ab56b1f-original-pull-secret") pod "global-pull-secret-syncer-zv9t9" (UID: "e3e3a914-d7de-4771-9d68-a7b05ab56b1f") : object "kube-system"/"original-pull-secret" not registered Jul 10 18:58:38.921146 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:38.920403 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slg7w\" (UniqueName: \"kubernetes.io/projected/4526c66d-17d8-4ed2-9782-42b9b2d9215b-kube-api-access-slg7w\") pod \"network-check-target-bl7wj\" (UID: \"4526c66d-17d8-4ed2-9782-42b9b2d9215b\") " pod="openshift-network-diagnostics/network-check-target-bl7wj" Jul 10 18:58:38.921146 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:38.920689 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 18:58:38.921146 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:38.920714 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 18:58:38.921146 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:38.920729 2560 projected.go:196] Error preparing data for projected volume kube-api-access-slg7w for pod openshift-network-diagnostics/network-check-target-bl7wj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:58:38.921146 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:38.920790 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4526c66d-17d8-4ed2-9782-42b9b2d9215b-kube-api-access-slg7w podName:4526c66d-17d8-4ed2-9782-42b9b2d9215b nodeName:}" failed. No retries permitted until 2026-07-10 18:58:46.920770614 +0000 UTC m=+18.250911404 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-slg7w" (UniqueName: "kubernetes.io/projected/4526c66d-17d8-4ed2-9782-42b9b2d9215b-kube-api-access-slg7w") pod "network-check-target-bl7wj" (UID: "4526c66d-17d8-4ed2-9782-42b9b2d9215b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:58:39.247518 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:39.247419 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bl7wj" Jul 10 18:58:39.247972 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:39.247594 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bl7wj" podUID="4526c66d-17d8-4ed2-9782-42b9b2d9215b" Jul 10 18:58:40.246524 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:40.246482 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:58:40.246722 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:40.246622 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9t9" podUID="e3e3a914-d7de-4771-9d68-a7b05ab56b1f" Jul 10 18:58:40.246947 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:40.246930 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 18:58:40.247049 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:40.247022 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9b95" podUID="b1bf574f-a4e2-4418-8f18-95fa0c80ca1e" Jul 10 18:58:41.246136 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:41.246103 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bl7wj" Jul 10 18:58:41.246605 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:41.246244 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bl7wj" podUID="4526c66d-17d8-4ed2-9782-42b9b2d9215b" Jul 10 18:58:42.247312 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:42.247278 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:58:42.247754 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:42.247325 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 18:58:42.247754 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:42.247666 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9t9" podUID="e3e3a914-d7de-4771-9d68-a7b05ab56b1f" Jul 10 18:58:42.247854 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:42.247800 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9b95" podUID="b1bf574f-a4e2-4418-8f18-95fa0c80ca1e" Jul 10 18:58:43.246375 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:43.246340 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bl7wj" Jul 10 18:58:43.246566 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:43.246468 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bl7wj" podUID="4526c66d-17d8-4ed2-9782-42b9b2d9215b" Jul 10 18:58:44.246218 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:44.246181 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:58:44.246711 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:44.246182 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 18:58:44.246711 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:44.246314 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9t9" podUID="e3e3a914-d7de-4771-9d68-a7b05ab56b1f" Jul 10 18:58:44.246711 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:44.246389 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9b95" podUID="b1bf574f-a4e2-4418-8f18-95fa0c80ca1e" Jul 10 18:58:45.246585 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:45.246543 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bl7wj" Jul 10 18:58:45.247010 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:45.246691 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bl7wj" podUID="4526c66d-17d8-4ed2-9782-42b9b2d9215b" Jul 10 18:58:46.246793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:46.246757 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:58:46.247239 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:46.246769 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 18:58:46.247239 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:46.246893 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9t9" podUID="e3e3a914-d7de-4771-9d68-a7b05ab56b1f" Jul 10 18:58:46.247239 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:46.246989 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9b95" podUID="b1bf574f-a4e2-4418-8f18-95fa0c80ca1e" Jul 10 18:58:46.878705 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:46.878657 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1bf574f-a4e2-4418-8f18-95fa0c80ca1e-metrics-certs\") pod \"network-metrics-daemon-v9b95\" (UID: \"b1bf574f-a4e2-4418-8f18-95fa0c80ca1e\") " pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 18:58:46.878916 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:46.878727 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3e3a914-d7de-4771-9d68-a7b05ab56b1f-original-pull-secret\") pod \"global-pull-secret-syncer-zv9t9\" (UID: \"e3e3a914-d7de-4771-9d68-a7b05ab56b1f\") " pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:58:46.878916 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:46.878852 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:58:46.878916 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:46.878913 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 18:58:46.879044 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:46.878959 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1bf574f-a4e2-4418-8f18-95fa0c80ca1e-metrics-certs podName:b1bf574f-a4e2-4418-8f18-95fa0c80ca1e nodeName:}" failed. No retries permitted until 2026-07-10 18:59:02.878935406 +0000 UTC m=+34.209076194 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b1bf574f-a4e2-4418-8f18-95fa0c80ca1e-metrics-certs") pod "network-metrics-daemon-v9b95" (UID: "b1bf574f-a4e2-4418-8f18-95fa0c80ca1e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:58:46.879044 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:46.878980 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3e3a914-d7de-4771-9d68-a7b05ab56b1f-original-pull-secret podName:e3e3a914-d7de-4771-9d68-a7b05ab56b1f nodeName:}" failed. No retries permitted until 2026-07-10 18:59:02.878970052 +0000 UTC m=+34.209110847 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3e3a914-d7de-4771-9d68-a7b05ab56b1f-original-pull-secret") pod "global-pull-secret-syncer-zv9t9" (UID: "e3e3a914-d7de-4771-9d68-a7b05ab56b1f") : object "kube-system"/"original-pull-secret" not registered Jul 10 18:58:46.980129 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:46.980073 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slg7w\" (UniqueName: \"kubernetes.io/projected/4526c66d-17d8-4ed2-9782-42b9b2d9215b-kube-api-access-slg7w\") pod \"network-check-target-bl7wj\" (UID: \"4526c66d-17d8-4ed2-9782-42b9b2d9215b\") " pod="openshift-network-diagnostics/network-check-target-bl7wj" Jul 10 18:58:46.980305 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:46.980256 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 18:58:46.980305 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:46.980277 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 18:58:46.980305 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:46.980287 2560 projected.go:196] Error preparing data for projected volume kube-api-access-slg7w for pod openshift-network-diagnostics/network-check-target-bl7wj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:58:46.980433 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:46.980354 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4526c66d-17d8-4ed2-9782-42b9b2d9215b-kube-api-access-slg7w podName:4526c66d-17d8-4ed2-9782-42b9b2d9215b nodeName:}" failed. No retries permitted until 2026-07-10 18:59:02.980335754 +0000 UTC m=+34.310476544 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-slg7w" (UniqueName: "kubernetes.io/projected/4526c66d-17d8-4ed2-9782-42b9b2d9215b-kube-api-access-slg7w") pod "network-check-target-bl7wj" (UID: "4526c66d-17d8-4ed2-9782-42b9b2d9215b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:58:47.246631 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:47.246552 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bl7wj" Jul 10 18:58:47.246772 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:47.246695 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bl7wj" podUID="4526c66d-17d8-4ed2-9782-42b9b2d9215b" Jul 10 18:58:48.246208 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:48.246171 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 18:58:48.246208 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:48.246190 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:58:48.246731 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:48.246320 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9b95" podUID="b1bf574f-a4e2-4418-8f18-95fa0c80ca1e" Jul 10 18:58:48.246731 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:48.246475 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9t9" podUID="e3e3a914-d7de-4771-9d68-a7b05ab56b1f" Jul 10 18:58:49.247393 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:49.247202 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bl7wj" Jul 10 18:58:49.247889 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:49.247492 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bl7wj" podUID="4526c66d-17d8-4ed2-9782-42b9b2d9215b" Jul 10 18:58:50.246103 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:50.246062 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:58:50.246292 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:50.246121 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 18:58:50.246292 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:50.246182 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9t9" podUID="e3e3a914-d7de-4771-9d68-a7b05ab56b1f" Jul 10 18:58:50.246401 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:50.246321 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9b95" podUID="b1bf574f-a4e2-4418-8f18-95fa0c80ca1e" Jul 10 18:58:51.246849 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:51.246774 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bl7wj" Jul 10 18:58:51.247292 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:51.246920 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bl7wj" podUID="4526c66d-17d8-4ed2-9782-42b9b2d9215b" Jul 10 18:58:52.246369 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:52.246328 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:58:52.246541 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:52.246333 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 18:58:52.246541 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:52.246457 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9t9" podUID="e3e3a914-d7de-4771-9d68-a7b05ab56b1f" Jul 10 18:58:52.246615 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:52.246547 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9b95" podUID="b1bf574f-a4e2-4418-8f18-95fa0c80ca1e" Jul 10 18:58:52.421075 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:52.421036 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sx2wz" event={"ID":"55b10423-8cf8-4f8f-bdb7-b5c7a89165a8","Type":"ContainerStarted","Data":"939d3fa5a5fb4ce401bb12b8372075c0a82a40deaa24e64d90235b201f01a824"} Jul 10 18:58:52.422843 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:52.422819 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" event={"ID":"2a22956a-4e96-41de-945c-a851b587a442","Type":"ContainerStarted","Data":"dc606caa6ccf32abffe473680e4034a5d7a9526595a9e475e2d786d768595d18"} Jul 10 18:58:52.424277 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:52.424252 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" event={"ID":"90178079-026e-41e5-94a0-d180a1940cc1","Type":"ContainerStarted","Data":"4973a45499e258f93c796307c814207f96836e59db5b883ab087f258b69c9009"} Jul 10 18:58:52.437199 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:52.437169 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-213.ec2.internal" event={"ID":"14b1acdcd9ea09cc31366e606e569025","Type":"ContainerStarted","Data":"21947b868c83781b91e2c256c0e786c78f25ac05411a34bc3dcef97d27bfd22d"} Jul 10 18:58:52.441417 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:52.441373 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-sx2wz" podStartSLOduration=3.026885932 podStartE2EDuration="23.441357914s" totalImagesPullingTime="20.414471982s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:52.441305398 +0000 UTC m=+23.771446207" watchObservedRunningTime="2026-07-10 18:58:52.441357914 +0000 UTC m=+23.771498724" Jul 10 18:58:52.455514 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:52.455469 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-213.ec2.internal" podStartSLOduration=22.455436597 podStartE2EDuration="22.455436597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:52.455172498 +0000 UTC m=+23.785313310" watchObservedRunningTime="2026-07-10 18:58:52.455436597 +0000 UTC m=+23.785577407" Jul 10 18:58:52.471919 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:52.471741 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cv4cv" podStartSLOduration=3.02165186 podStartE2EDuration="23.471721914s" totalImagesPullingTime="20.450070054s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:52.47114525 +0000 UTC m=+23.801286060" watchObservedRunningTime="2026-07-10 18:58:52.471721914 +0000 UTC m=+23.801862725" Jul 10 18:58:53.246328 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:53.246153 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bl7wj" Jul 10 18:58:53.246494 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:53.246423 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bl7wj" podUID="4526c66d-17d8-4ed2-9782-42b9b2d9215b" Jul 10 18:58:53.440755 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:53.440533 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x6sx7" event={"ID":"fd6564ad-086b-4c7b-ada7-99f16bfe03c4","Type":"ContainerStarted","Data":"3200daee539f652678ada1f217eee834f6187778ac7133b14b6169851375a450"} Jul 10 18:58:53.442091 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:53.442065 2560 generic.go:356] "Generic (PLEG): container finished" podID="eb6a2004a5a296385f19413f1682752d" containerID="7ca9d254d38bcee9785ad1cc8b29317db44dec9db0b49432da1ee588e753b0e7" exitCode=0 Jul 10 18:58:53.442178 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:53.442159 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-213.ec2.internal" event={"ID":"eb6a2004a5a296385f19413f1682752d","Type":"ContainerDied","Data":"7ca9d254d38bcee9785ad1cc8b29317db44dec9db0b49432da1ee588e753b0e7"} Jul 10 18:58:53.443677 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:53.443652 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spzdj" event={"ID":"262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5","Type":"ContainerStarted","Data":"3b7977e476102d32161db07ccf506e76830486b658819092fd318c2e8a476a43"} Jul 10 18:58:53.445047 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:53.445026 2560 generic.go:356] "Generic (PLEG): container finished" podID="eda6e317-2a4b-4591-a5ef-0139d0e57e95" containerID="445f5fb45d9515dab9e1497354e874b212e464737e2b3d3e3c842137eaccaf35" exitCode=0 Jul 10 18:58:53.445155 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:53.445087 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z58c2" event={"ID":"eda6e317-2a4b-4591-a5ef-0139d0e57e95","Type":"ContainerDied","Data":"445f5fb45d9515dab9e1497354e874b212e464737e2b3d3e3c842137eaccaf35"} Jul 10 18:58:53.446573 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:53.446553 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4hn8z" event={"ID":"00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a","Type":"ContainerStarted","Data":"5258468171a0df038944a43ac3604043e3951ab3a7e557caad0aab98632135a5"} Jul 10 18:58:53.448030 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:53.448006 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l9qmm" event={"ID":"d85934ea-40c2-4589-aca7-b0e8efca569c","Type":"ContainerStarted","Data":"02b924294981d4a9e61c5f17bcbf9d7bfca0992fdb415793c8ecf8f074cfef8f"} Jul 10 18:58:53.450525 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:53.450509 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l52t_2a22956a-4e96-41de-945c-a851b587a442/ovn-acl-logging/0.log" Jul 10 18:58:53.450848 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:53.450829 2560 generic.go:356] "Generic (PLEG): container finished" podID="2a22956a-4e96-41de-945c-a851b587a442" containerID="00215c45052b6e62b5aa78d6d1ee5ddb5ee076f69c13b05362eecc4a25caefe7" exitCode=1 Jul 10 18:58:53.450948 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:53.450911 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" event={"ID":"2a22956a-4e96-41de-945c-a851b587a442","Type":"ContainerStarted","Data":"9784d1477937dad81349a2e621183e504cbedbc78b3d23c05e44b7db68044951"} Jul 10 18:58:53.450997 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:53.450986 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" event={"ID":"2a22956a-4e96-41de-945c-a851b587a442","Type":"ContainerStarted","Data":"26aabed4258aee09127910e80f80d64aa5effd0a3b1d5e1fe4b339339c2c92af"} Jul 10 18:58:53.451078 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:53.451036 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" event={"ID":"2a22956a-4e96-41de-945c-a851b587a442","Type":"ContainerStarted","Data":"e57b0760b372f9d0a25262d61dffb26cd4f3e746f25b78d52b050e9eadf98de4"} Jul 10 18:58:53.451115 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:53.451088 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" event={"ID":"2a22956a-4e96-41de-945c-a851b587a442","Type":"ContainerStarted","Data":"51d49a67ef4e7f6d45c1d75e203ae45cbc794e6753d769a7ef57a71eee63412e"} Jul 10 18:58:53.451150 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:53.451135 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" event={"ID":"2a22956a-4e96-41de-945c-a851b587a442","Type":"ContainerDied","Data":"00215c45052b6e62b5aa78d6d1ee5ddb5ee076f69c13b05362eecc4a25caefe7"} Jul 10 18:58:53.492177 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:53.492117 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-x6sx7" podStartSLOduration=4.009784677 podStartE2EDuration="24.492097712s" totalImagesPullingTime="20.482313035s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:53.461496531 +0000 UTC m=+24.791637364" watchObservedRunningTime="2026-07-10 18:58:53.492097712 +0000 UTC m=+24.822238522" Jul 10 18:58:53.529847 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:53.529806 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-l9qmm" podStartSLOduration=4.024434116 podStartE2EDuration="24.529791035s" totalImagesPullingTime="20.505356919s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:53.529494265 +0000 UTC m=+24.859635103" watchObservedRunningTime="2026-07-10 18:58:53.529791035 +0000 UTC m=+24.859931844" Jul 10 18:58:53.550592 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:53.548724 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4hn8z" podStartSLOduration=4.066710536 podStartE2EDuration="24.548701345s" totalImagesPullingTime="20.481990809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:58:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:53.547050537 +0000 UTC m=+24.877191347" watchObservedRunningTime="2026-07-10 18:58:53.548701345 +0000 UTC m=+24.878842157" Jul 10 18:58:54.246457 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:54.246410 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:58:54.246711 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:54.246464 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 18:58:54.246711 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:54.246558 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9t9" podUID="e3e3a914-d7de-4771-9d68-a7b05ab56b1f" Jul 10 18:58:54.246833 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:54.246717 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9b95" podUID="b1bf574f-a4e2-4418-8f18-95fa0c80ca1e" Jul 10 18:58:54.455301 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:54.455256 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g5t7c" event={"ID":"13f6e0d5-e879-45c0-b2ec-7b310c717933","Type":"ContainerStarted","Data":"f1ec28d4ebb0303371d035af50419c9daabc2a53fc20c5031f7b36eef0d06112"} Jul 10 18:58:54.457476 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:54.457414 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-213.ec2.internal" event={"ID":"eb6a2004a5a296385f19413f1682752d","Type":"ContainerStarted","Data":"1c885d77364b84b12c812da16f31a1b328b851b0b671268cf3ecce68cb822160"} Jul 10 18:58:54.475536 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:54.475476 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-g5t7c" podStartSLOduration=4.950444603 podStartE2EDuration="25.475453638s" totalImagesPullingTime="20.525009035s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:54.475219454 +0000 UTC m=+25.805360264" watchObservedRunningTime="2026-07-10 18:58:54.475453638 +0000 UTC m=+25.805594441" Jul 10 18:58:54.560197 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:54.560168 2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 18:58:55.169658 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:55.169542 2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T18:58:54.560190891Z","UUID":"317fc700-328c-4eca-89fb-f7c4da29d125","Handler":null,"Name":"","Endpoint":""} Jul 10 18:58:55.171021 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:55.171001 2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 18:58:55.171144 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:55.171041 2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 18:58:55.246395 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:55.246364 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bl7wj" Jul 10 18:58:55.246609 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:55.246518 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bl7wj" podUID="4526c66d-17d8-4ed2-9782-42b9b2d9215b" Jul 10 18:58:55.462192 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:55.462019 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spzdj" event={"ID":"262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5","Type":"ContainerStarted","Data":"87b38804d4501286faafa0a723bfd41d6c29850ab9fd26c3869bbd47a5db6f26"} Jul 10 18:58:56.247041 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:56.246958 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 18:58:56.247200 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:56.246958 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:58:56.247200 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:56.247106 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9b95" podUID="b1bf574f-a4e2-4418-8f18-95fa0c80ca1e" Jul 10 18:58:56.247200 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:56.247139 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9t9" podUID="e3e3a914-d7de-4771-9d68-a7b05ab56b1f" Jul 10 18:58:56.293930 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:56.293889 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4hn8z" Jul 10 18:58:56.294640 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:56.294618 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4hn8z" Jul 10 18:58:56.317776 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:56.317585 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-213.ec2.internal" podStartSLOduration=26.317567554 podStartE2EDuration="26.317567554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:54.496406343 +0000 UTC m=+25.826547153" watchObservedRunningTime="2026-07-10 18:58:56.317567554 +0000 UTC m=+27.647708363" Jul 10 18:58:56.466672 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:56.466628 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spzdj" event={"ID":"262f3ffd-d7bb-401e-acc1-e9ae04dc3ea5","Type":"ContainerStarted","Data":"0b09afde7a67678c925d748505237b0fe271cc84dcf6616df198b4f1e2db6958"} Jul 10 18:58:56.470079 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:56.470056 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l52t_2a22956a-4e96-41de-945c-a851b587a442/ovn-acl-logging/0.log" Jul 10 18:58:56.470513 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:56.470472 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" event={"ID":"2a22956a-4e96-41de-945c-a851b587a442","Type":"ContainerStarted","Data":"a5846d7ba1f07e0d751d4457e111d5cfa46535d883602446c1415f5b1d45ad88"} Jul 10 18:58:56.470745 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:56.470724 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4hn8z" Jul 10 18:58:56.471373 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:56.471355 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4hn8z" Jul 10 18:58:56.518728 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:56.518621 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spzdj" podStartSLOduration=4.605808215 podStartE2EDuration="27.518602621s" totalImagesPullingTime="22.912794406s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:56.492331983 +0000 UTC m=+27.822472796" watchObservedRunningTime="2026-07-10 18:58:56.518602621 +0000 UTC m=+27.848743432" Jul 10 18:58:57.246403 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:57.246354 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bl7wj" Jul 10 18:58:57.246673 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:57.246536 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bl7wj" podUID="4526c66d-17d8-4ed2-9782-42b9b2d9215b" Jul 10 18:58:58.247039 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:58.246803 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 18:58:58.247807 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:58.246802 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:58:58.247807 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:58.247068 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9b95" podUID="b1bf574f-a4e2-4418-8f18-95fa0c80ca1e" Jul 10 18:58:58.247807 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:58.247119 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9t9" podUID="e3e3a914-d7de-4771-9d68-a7b05ab56b1f" Jul 10 18:58:58.476602 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:58.476567 2560 generic.go:356] "Generic (PLEG): container finished" podID="eda6e317-2a4b-4591-a5ef-0139d0e57e95" containerID="0e632f53dcb68c3d3d1a7d17b82382a9f593d1ff2622a2687083e37c8d91c9ed" exitCode=0 Jul 10 18:58:58.476786 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:58.476663 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z58c2" event={"ID":"eda6e317-2a4b-4591-a5ef-0139d0e57e95","Type":"ContainerDied","Data":"0e632f53dcb68c3d3d1a7d17b82382a9f593d1ff2622a2687083e37c8d91c9ed"} Jul 10 18:58:58.479986 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:58.479970 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l52t_2a22956a-4e96-41de-945c-a851b587a442/ovn-acl-logging/0.log" Jul 10 18:58:58.480336 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:58.480311 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" event={"ID":"2a22956a-4e96-41de-945c-a851b587a442","Type":"ContainerStarted","Data":"238b6a4f4ca7bf7ca1d01b3ad7a9ac9b45f67872530eac2d1ccaee5ba36c3890"} Jul 10 18:58:58.480647 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:58.480608 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:58.480758 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:58.480741 2560 scope.go:117] "RemoveContainer" containerID="00215c45052b6e62b5aa78d6d1ee5ddb5ee076f69c13b05362eecc4a25caefe7" Jul 10 18:58:58.497292 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:58.497242 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:59.247713 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:59.247681 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bl7wj" Jul 10 18:58:59.248149 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:59.247795 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bl7wj" podUID="4526c66d-17d8-4ed2-9782-42b9b2d9215b" Jul 10 18:58:59.485913 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:59.485889 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l52t_2a22956a-4e96-41de-945c-a851b587a442/ovn-acl-logging/0.log" Jul 10 18:58:59.486482 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:59.486437 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" event={"ID":"2a22956a-4e96-41de-945c-a851b587a442","Type":"ContainerStarted","Data":"09211ec45bc382bfdd50cb78e56449981352bb16d8bc6b9b9dfc0a3775c0d687"} Jul 10 18:58:59.486638 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:59.486624 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:59.486704 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:59.486671 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:59.502274 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:59.502203 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:58:59.523489 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:59.523414 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" podStartSLOduration=9.738911792 podStartE2EDuration="30.523394224s" totalImagesPullingTime="20.784482432s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:59.520870325 +0000 UTC m=+30.851011134" watchObservedRunningTime="2026-07-10 18:58:59.523394224 +0000 UTC m=+30.853535033" Jul 10 18:58:59.872946 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:59.872911 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zv9t9"] Jul 10 18:58:59.873170 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:59.873049 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:58:59.873232 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:59.873161 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9t9" podUID="e3e3a914-d7de-4771-9d68-a7b05ab56b1f" Jul 10 18:58:59.875411 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:59.875385 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bl7wj"] Jul 10 18:58:59.875561 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:59.875477 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bl7wj" Jul 10 18:58:59.875628 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:59.875596 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bl7wj" podUID="4526c66d-17d8-4ed2-9782-42b9b2d9215b" Jul 10 18:58:59.878273 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:59.878246 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v9b95"] Jul 10 18:58:59.878397 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:58:59.878344 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 18:58:59.878471 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:58:59.878427 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9b95" podUID="b1bf574f-a4e2-4418-8f18-95fa0c80ca1e" Jul 10 18:59:00.492492 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:00.492329 2560 generic.go:356] "Generic (PLEG): container finished" podID="eda6e317-2a4b-4591-a5ef-0139d0e57e95" containerID="03e982cda798aa35ad02f308e477e752f0dec0f0db645d5958b42576982ad4f2" exitCode=0 Jul 10 18:59:00.492492 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:00.492420 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z58c2" event={"ID":"eda6e317-2a4b-4591-a5ef-0139d0e57e95","Type":"ContainerDied","Data":"03e982cda798aa35ad02f308e477e752f0dec0f0db645d5958b42576982ad4f2"} Jul 10 18:59:01.246903 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:01.246859 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 18:59:01.247103 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:01.246981 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9b95" podUID="b1bf574f-a4e2-4418-8f18-95fa0c80ca1e" Jul 10 18:59:01.247287 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:01.247255 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:59:01.247396 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:01.247376 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9t9" podUID="e3e3a914-d7de-4771-9d68-a7b05ab56b1f" Jul 10 18:59:02.247057 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:02.247001 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bl7wj" Jul 10 18:59:02.247429 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:02.247140 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bl7wj" podUID="4526c66d-17d8-4ed2-9782-42b9b2d9215b" Jul 10 18:59:02.499375 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:02.499294 2560 generic.go:356] "Generic (PLEG): container finished" podID="eda6e317-2a4b-4591-a5ef-0139d0e57e95" containerID="c991448349d9004c163048f76c30708dbe3c9d7a6284daa37c540916fefde6f8" exitCode=0 Jul 10 18:59:02.499375 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:02.499356 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z58c2" event={"ID":"eda6e317-2a4b-4591-a5ef-0139d0e57e95","Type":"ContainerDied","Data":"c991448349d9004c163048f76c30708dbe3c9d7a6284daa37c540916fefde6f8"} Jul 10 18:59:02.894744 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:02.894642 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1bf574f-a4e2-4418-8f18-95fa0c80ca1e-metrics-certs\") pod \"network-metrics-daemon-v9b95\" (UID: \"b1bf574f-a4e2-4418-8f18-95fa0c80ca1e\") " pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 18:59:02.894744 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:02.894691 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3e3a914-d7de-4771-9d68-a7b05ab56b1f-original-pull-secret\") pod \"global-pull-secret-syncer-zv9t9\" (UID: \"e3e3a914-d7de-4771-9d68-a7b05ab56b1f\") " pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:59:02.894946 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:02.894796 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 18:59:02.894946 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:02.894802 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:59:02.894946 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:02.894851 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3e3a914-d7de-4771-9d68-a7b05ab56b1f-original-pull-secret podName:e3e3a914-d7de-4771-9d68-a7b05ab56b1f nodeName:}" failed. No retries permitted until 2026-07-10 18:59:34.894837612 +0000 UTC m=+66.224978400 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3e3a914-d7de-4771-9d68-a7b05ab56b1f-original-pull-secret") pod "global-pull-secret-syncer-zv9t9" (UID: "e3e3a914-d7de-4771-9d68-a7b05ab56b1f") : object "kube-system"/"original-pull-secret" not registered Jul 10 18:59:02.894946 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:02.894864 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1bf574f-a4e2-4418-8f18-95fa0c80ca1e-metrics-certs podName:b1bf574f-a4e2-4418-8f18-95fa0c80ca1e nodeName:}" failed. No retries permitted until 2026-07-10 18:59:34.894858494 +0000 UTC m=+66.224999283 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b1bf574f-a4e2-4418-8f18-95fa0c80ca1e-metrics-certs") pod "network-metrics-daemon-v9b95" (UID: "b1bf574f-a4e2-4418-8f18-95fa0c80ca1e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:59:02.995177 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:02.995134 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slg7w\" (UniqueName: \"kubernetes.io/projected/4526c66d-17d8-4ed2-9782-42b9b2d9215b-kube-api-access-slg7w\") pod \"network-check-target-bl7wj\" (UID: \"4526c66d-17d8-4ed2-9782-42b9b2d9215b\") " pod="openshift-network-diagnostics/network-check-target-bl7wj" Jul 10 18:59:02.995355 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:02.995305 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 18:59:02.995355 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:02.995330 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 18:59:02.995355 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:02.995339 2560 projected.go:196] Error preparing data for projected volume kube-api-access-slg7w for pod openshift-network-diagnostics/network-check-target-bl7wj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:59:02.995484 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:02.995394 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4526c66d-17d8-4ed2-9782-42b9b2d9215b-kube-api-access-slg7w podName:4526c66d-17d8-4ed2-9782-42b9b2d9215b nodeName:}" failed. No retries permitted until 2026-07-10 18:59:34.995377119 +0000 UTC m=+66.325517927 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-slg7w" (UniqueName: "kubernetes.io/projected/4526c66d-17d8-4ed2-9782-42b9b2d9215b-kube-api-access-slg7w") pod "network-check-target-bl7wj" (UID: "4526c66d-17d8-4ed2-9782-42b9b2d9215b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:59:03.250733 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:03.250647 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:59:03.251162 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:03.250781 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 18:59:03.251162 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:03.250778 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9t9" podUID="e3e3a914-d7de-4771-9d68-a7b05ab56b1f" Jul 10 18:59:03.251162 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:03.250830 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bl7wj" Jul 10 18:59:03.251162 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:03.250923 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9b95" podUID="b1bf574f-a4e2-4418-8f18-95fa0c80ca1e" Jul 10 18:59:03.251162 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:03.250999 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bl7wj" podUID="4526c66d-17d8-4ed2-9782-42b9b2d9215b" Jul 10 18:59:05.250793 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:05.250583 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:59:05.251254 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:05.250614 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bl7wj" Jul 10 18:59:05.251254 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:05.250884 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zv9t9" podUID="e3e3a914-d7de-4771-9d68-a7b05ab56b1f" Jul 10 18:59:05.251254 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:05.250664 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 18:59:05.251254 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:05.250980 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bl7wj" podUID="4526c66d-17d8-4ed2-9782-42b9b2d9215b" Jul 10 18:59:05.251254 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:05.251064 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9b95" podUID="b1bf574f-a4e2-4418-8f18-95fa0c80ca1e" Jul 10 18:59:05.933529 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:05.933422 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-213.ec2.internal" event="NodeReady" Jul 10 18:59:05.933691 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:05.933587 2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 18:59:05.971146 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:05.971114 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84bdbb94b5-gnz5b"] Jul 10 18:59:05.974233 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:05.974212 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 18:59:05.977246 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:05.977208 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 10 18:59:05.977388 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:05.977320 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 10 18:59:05.977388 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:05.977375 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 10 18:59:05.978075 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:05.978054 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-d494m\"" Jul 10 18:59:05.984891 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:05.984853 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-d7l7x"] Jul 10 18:59:05.985033 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:05.984947 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 10 18:59:05.986937 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:05.986915 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-z8vwd"] Jul 10 18:59:05.987115 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:05.987042 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d7l7x" Jul 10 18:59:05.989622 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:05.989599 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84bdbb94b5-gnz5b"] Jul 10 18:59:05.989753 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:05.989693 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-z8vwd" Jul 10 18:59:05.989753 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:05.989718 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 18:59:05.989880 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:05.989863 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-s6z54\"" Jul 10 18:59:05.990095 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:05.990075 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 18:59:05.992799 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:05.992779 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5z57c\"" Jul 10 18:59:05.992959 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:05.992939 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 18:59:05.993039 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:05.993004 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 18:59:05.993378 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:05.993342 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 18:59:05.999507 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:05.999482 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d7l7x"] Jul 10 18:59:06.014034 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.014003 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-z8vwd"] Jul 10 18:59:06.015141 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.015115 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs8ml\" (UniqueName: \"kubernetes.io/projected/e9ea737c-ac9d-4694-bd6b-980514db5371-kube-api-access-zs8ml\") pod \"dns-default-d7l7x\" (UID: \"e9ea737c-ac9d-4694-bd6b-980514db5371\") " pod="openshift-dns/dns-default-d7l7x" Jul 10 18:59:06.015273 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.015149 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/24c51490-aab2-4fd0-9ffc-06886c5515b3-image-registry-private-configuration\") pod \"image-registry-84bdbb94b5-gnz5b\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 18:59:06.015273 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.015171 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-bound-sa-token\") pod \"image-registry-84bdbb94b5-gnz5b\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 18:59:06.015273 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.015197 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d95e6de4-e5cc-4467-a6b9-4ac8d842d744-cert\") pod \"ingress-canary-z8vwd\" (UID: \"d95e6de4-e5cc-4467-a6b9-4ac8d842d744\") " pod="openshift-ingress-canary/ingress-canary-z8vwd" Jul 10 18:59:06.015427 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.015280 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/24c51490-aab2-4fd0-9ffc-06886c5515b3-installation-pull-secrets\") pod \"image-registry-84bdbb94b5-gnz5b\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 18:59:06.015427 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.015367 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-registry-tls\") pod \"image-registry-84bdbb94b5-gnz5b\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 18:59:06.015427 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.015395 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7lng\" (UniqueName: \"kubernetes.io/projected/d95e6de4-e5cc-4467-a6b9-4ac8d842d744-kube-api-access-r7lng\") pod \"ingress-canary-z8vwd\" (UID: \"d95e6de4-e5cc-4467-a6b9-4ac8d842d744\") " pod="openshift-ingress-canary/ingress-canary-z8vwd" Jul 10 18:59:06.015427 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.015424 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/24c51490-aab2-4fd0-9ffc-06886c5515b3-registry-certificates\") pod \"image-registry-84bdbb94b5-gnz5b\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 18:59:06.015645 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.015478 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24c51490-aab2-4fd0-9ffc-06886c5515b3-trusted-ca\") pod \"image-registry-84bdbb94b5-gnz5b\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 18:59:06.015645 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.015571 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt98t\" (UniqueName: \"kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-kube-api-access-dt98t\") pod \"image-registry-84bdbb94b5-gnz5b\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 18:59:06.015645 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.015633 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ea737c-ac9d-4694-bd6b-980514db5371-config-volume\") pod \"dns-default-d7l7x\" (UID: \"e9ea737c-ac9d-4694-bd6b-980514db5371\") " pod="openshift-dns/dns-default-d7l7x" Jul 10 18:59:06.015770 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.015657 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9ea737c-ac9d-4694-bd6b-980514db5371-metrics-tls\") pod \"dns-default-d7l7x\" (UID: \"e9ea737c-ac9d-4694-bd6b-980514db5371\") " pod="openshift-dns/dns-default-d7l7x" Jul 10 18:59:06.015770 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.015687 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9ea737c-ac9d-4694-bd6b-980514db5371-tmp-dir\") pod \"dns-default-d7l7x\" (UID: \"e9ea737c-ac9d-4694-bd6b-980514db5371\") " pod="openshift-dns/dns-default-d7l7x" Jul 10 18:59:06.015770 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.015729 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/24c51490-aab2-4fd0-9ffc-06886c5515b3-ca-trust-extracted\") pod \"image-registry-84bdbb94b5-gnz5b\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 18:59:06.116416 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.116377 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/24c51490-aab2-4fd0-9ffc-06886c5515b3-installation-pull-secrets\") pod \"image-registry-84bdbb94b5-gnz5b\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 18:59:06.116622 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.116456 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-registry-tls\") pod \"image-registry-84bdbb94b5-gnz5b\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 18:59:06.116622 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.116483 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7lng\" (UniqueName: \"kubernetes.io/projected/d95e6de4-e5cc-4467-a6b9-4ac8d842d744-kube-api-access-r7lng\") pod \"ingress-canary-z8vwd\" (UID: \"d95e6de4-e5cc-4467-a6b9-4ac8d842d744\") " pod="openshift-ingress-canary/ingress-canary-z8vwd" Jul 10 18:59:06.116622 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.116504 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/24c51490-aab2-4fd0-9ffc-06886c5515b3-registry-certificates\") pod \"image-registry-84bdbb94b5-gnz5b\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 18:59:06.116622 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.116528 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24c51490-aab2-4fd0-9ffc-06886c5515b3-trusted-ca\") pod \"image-registry-84bdbb94b5-gnz5b\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 18:59:06.116622 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.116564 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt98t\" (UniqueName: \"kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-kube-api-access-dt98t\") pod \"image-registry-84bdbb94b5-gnz5b\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 18:59:06.116622 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.116608 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ea737c-ac9d-4694-bd6b-980514db5371-config-volume\") pod \"dns-default-d7l7x\" (UID: \"e9ea737c-ac9d-4694-bd6b-980514db5371\") " pod="openshift-dns/dns-default-d7l7x" Jul 10 18:59:06.116938 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:06.116677 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 18:59:06.116938 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:06.116696 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84bdbb94b5-gnz5b: secret "image-registry-tls" not found Jul 10 18:59:06.116938 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:06.116755 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-registry-tls podName:24c51490-aab2-4fd0-9ffc-06886c5515b3 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:06.616736312 +0000 UTC m=+37.946877099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-registry-tls") pod "image-registry-84bdbb94b5-gnz5b" (UID: "24c51490-aab2-4fd0-9ffc-06886c5515b3") : secret "image-registry-tls" not found Jul 10 18:59:06.117089 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.117007 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9ea737c-ac9d-4694-bd6b-980514db5371-metrics-tls\") pod \"dns-default-d7l7x\" (UID: \"e9ea737c-ac9d-4694-bd6b-980514db5371\") " pod="openshift-dns/dns-default-d7l7x" Jul 10 18:59:06.117089 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.117047 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9ea737c-ac9d-4694-bd6b-980514db5371-tmp-dir\") pod \"dns-default-d7l7x\" (UID: \"e9ea737c-ac9d-4694-bd6b-980514db5371\") " pod="openshift-dns/dns-default-d7l7x" Jul 10 18:59:06.117191 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.117095 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/24c51490-aab2-4fd0-9ffc-06886c5515b3-ca-trust-extracted\") pod \"image-registry-84bdbb94b5-gnz5b\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 18:59:06.117191 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.117122 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs8ml\" (UniqueName: \"kubernetes.io/projected/e9ea737c-ac9d-4694-bd6b-980514db5371-kube-api-access-zs8ml\") pod \"dns-default-d7l7x\" (UID: \"e9ea737c-ac9d-4694-bd6b-980514db5371\") " pod="openshift-dns/dns-default-d7l7x" Jul 10 18:59:06.117191 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.117161 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/24c51490-aab2-4fd0-9ffc-06886c5515b3-image-registry-private-configuration\") pod \"image-registry-84bdbb94b5-gnz5b\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 18:59:06.117330 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.117191 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-bound-sa-token\") pod \"image-registry-84bdbb94b5-gnz5b\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 18:59:06.117330 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.117217 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d95e6de4-e5cc-4467-a6b9-4ac8d842d744-cert\") pod \"ingress-canary-z8vwd\" (UID: \"d95e6de4-e5cc-4467-a6b9-4ac8d842d744\") " pod="openshift-ingress-canary/ingress-canary-z8vwd" Jul 10 18:59:06.117330 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.117219 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ea737c-ac9d-4694-bd6b-980514db5371-config-volume\") pod \"dns-default-d7l7x\" (UID: \"e9ea737c-ac9d-4694-bd6b-980514db5371\") " pod="openshift-dns/dns-default-d7l7x" Jul 10 18:59:06.117511 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:06.117347 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 18:59:06.117511 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.117393 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/24c51490-aab2-4fd0-9ffc-06886c5515b3-registry-certificates\") pod \"image-registry-84bdbb94b5-gnz5b\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 18:59:06.117511 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.117412 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9ea737c-ac9d-4694-bd6b-980514db5371-tmp-dir\") pod \"dns-default-d7l7x\" (UID: \"e9ea737c-ac9d-4694-bd6b-980514db5371\") " pod="openshift-dns/dns-default-d7l7x" Jul 10 18:59:06.117511 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:06.117417 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9ea737c-ac9d-4694-bd6b-980514db5371-metrics-tls podName:e9ea737c-ac9d-4694-bd6b-980514db5371 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:06.617400097 +0000 UTC m=+37.947540885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e9ea737c-ac9d-4694-bd6b-980514db5371-metrics-tls") pod "dns-default-d7l7x" (UID: "e9ea737c-ac9d-4694-bd6b-980514db5371") : secret "dns-default-metrics-tls" not found Jul 10 18:59:06.117720 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.117572 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/24c51490-aab2-4fd0-9ffc-06886c5515b3-ca-trust-extracted\") pod \"image-registry-84bdbb94b5-gnz5b\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 18:59:06.117720 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.117700 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24c51490-aab2-4fd0-9ffc-06886c5515b3-trusted-ca\") pod \"image-registry-84bdbb94b5-gnz5b\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 18:59:06.121916 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.121885 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/24c51490-aab2-4fd0-9ffc-06886c5515b3-installation-pull-secrets\") pod \"image-registry-84bdbb94b5-gnz5b\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 18:59:06.121916 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.121900 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d95e6de4-e5cc-4467-a6b9-4ac8d842d744-cert\") pod \"ingress-canary-z8vwd\" (UID: \"d95e6de4-e5cc-4467-a6b9-4ac8d842d744\") " pod="openshift-ingress-canary/ingress-canary-z8vwd" Jul 10 18:59:06.122056 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.121975 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/24c51490-aab2-4fd0-9ffc-06886c5515b3-image-registry-private-configuration\") pod \"image-registry-84bdbb94b5-gnz5b\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 18:59:06.127157 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.127125 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7lng\" (UniqueName: \"kubernetes.io/projected/d95e6de4-e5cc-4467-a6b9-4ac8d842d744-kube-api-access-r7lng\") pod \"ingress-canary-z8vwd\" (UID: \"d95e6de4-e5cc-4467-a6b9-4ac8d842d744\") " pod="openshift-ingress-canary/ingress-canary-z8vwd" Jul 10 18:59:06.127299 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.127267 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt98t\" (UniqueName: \"kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-kube-api-access-dt98t\") pod \"image-registry-84bdbb94b5-gnz5b\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 18:59:06.127783 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.127760 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs8ml\" (UniqueName: \"kubernetes.io/projected/e9ea737c-ac9d-4694-bd6b-980514db5371-kube-api-access-zs8ml\") pod \"dns-default-d7l7x\" (UID: \"e9ea737c-ac9d-4694-bd6b-980514db5371\") " pod="openshift-dns/dns-default-d7l7x" Jul 10 18:59:06.128712 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.128662 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-bound-sa-token\") pod \"image-registry-84bdbb94b5-gnz5b\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 18:59:06.305310 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.305214 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-z8vwd" Jul 10 18:59:06.622270 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.622177 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-registry-tls\") pod \"image-registry-84bdbb94b5-gnz5b\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 18:59:06.622472 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.622276 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9ea737c-ac9d-4694-bd6b-980514db5371-metrics-tls\") pod \"dns-default-d7l7x\" (UID: \"e9ea737c-ac9d-4694-bd6b-980514db5371\") " pod="openshift-dns/dns-default-d7l7x" Jul 10 18:59:06.622472 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:06.622340 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 18:59:06.622472 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:06.622359 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84bdbb94b5-gnz5b: secret "image-registry-tls" not found Jul 10 18:59:06.622472 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:06.622424 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-registry-tls podName:24c51490-aab2-4fd0-9ffc-06886c5515b3 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:07.622405224 +0000 UTC m=+38.952546012 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-registry-tls") pod "image-registry-84bdbb94b5-gnz5b" (UID: "24c51490-aab2-4fd0-9ffc-06886c5515b3") : secret "image-registry-tls" not found Jul 10 18:59:06.622472 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:06.622437 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 18:59:06.622718 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:06.622507 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9ea737c-ac9d-4694-bd6b-980514db5371-metrics-tls podName:e9ea737c-ac9d-4694-bd6b-980514db5371 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:07.622487216 +0000 UTC m=+38.952628003 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e9ea737c-ac9d-4694-bd6b-980514db5371-metrics-tls") pod "dns-default-d7l7x" (UID: "e9ea737c-ac9d-4694-bd6b-980514db5371") : secret "dns-default-metrics-tls" not found Jul 10 18:59:06.782230 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:06.782065 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-z8vwd"] Jul 10 18:59:06.785333 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:59:06.785305 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95e6de4_e5cc_4467_a6b9_4ac8d842d744.slice/crio-0ca9994e1022a15950bae25a5c3162963b72b3eb17e431272c1941eed6fcaad3 WatchSource:0}: Error finding container 0ca9994e1022a15950bae25a5c3162963b72b3eb17e431272c1941eed6fcaad3: Status 404 returned error can't find the container with id 0ca9994e1022a15950bae25a5c3162963b72b3eb17e431272c1941eed6fcaad3 Jul 10 18:59:07.246933 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:07.246896 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bl7wj" Jul 10 18:59:07.247128 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:07.246896 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:59:07.247128 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:07.246896 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 18:59:07.251156 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:07.250298 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 18:59:07.251981 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:07.251696 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 18:59:07.251981 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:07.251737 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xwgdk\"" Jul 10 18:59:07.251981 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:07.251756 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 18:59:07.251981 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:07.251818 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mzc76\"" Jul 10 18:59:07.251981 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:07.251918 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 18:59:07.515420 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:07.515269 2560 generic.go:356] "Generic (PLEG): container finished" podID="eda6e317-2a4b-4591-a5ef-0139d0e57e95" containerID="1c085822680270763378877611fe48e2c5a56a7befbe87c805dd689ebaae0cba" exitCode=0 Jul 10 18:59:07.515420 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:07.515357 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z58c2" event={"ID":"eda6e317-2a4b-4591-a5ef-0139d0e57e95","Type":"ContainerDied","Data":"1c085822680270763378877611fe48e2c5a56a7befbe87c805dd689ebaae0cba"} Jul 10 18:59:07.516772 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:07.516728 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-z8vwd" event={"ID":"d95e6de4-e5cc-4467-a6b9-4ac8d842d744","Type":"ContainerStarted","Data":"0ca9994e1022a15950bae25a5c3162963b72b3eb17e431272c1941eed6fcaad3"} Jul 10 18:59:07.630095 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:07.630044 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-registry-tls\") pod \"image-registry-84bdbb94b5-gnz5b\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 18:59:07.630258 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:07.630236 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 18:59:07.630258 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:07.630258 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84bdbb94b5-gnz5b: secret "image-registry-tls" not found Jul 10 18:59:07.630408 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:07.630318 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-registry-tls podName:24c51490-aab2-4fd0-9ffc-06886c5515b3 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:09.630298287 +0000 UTC m=+40.960439079 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-registry-tls") pod "image-registry-84bdbb94b5-gnz5b" (UID: "24c51490-aab2-4fd0-9ffc-06886c5515b3") : secret "image-registry-tls" not found Jul 10 18:59:07.630536 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:07.630518 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9ea737c-ac9d-4694-bd6b-980514db5371-metrics-tls\") pod \"dns-default-d7l7x\" (UID: \"e9ea737c-ac9d-4694-bd6b-980514db5371\") " pod="openshift-dns/dns-default-d7l7x" Jul 10 18:59:07.630735 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:07.630719 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 18:59:07.630804 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:07.630771 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9ea737c-ac9d-4694-bd6b-980514db5371-metrics-tls podName:e9ea737c-ac9d-4694-bd6b-980514db5371 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:09.630756553 +0000 UTC m=+40.960897344 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e9ea737c-ac9d-4694-bd6b-980514db5371-metrics-tls") pod "dns-default-d7l7x" (UID: "e9ea737c-ac9d-4694-bd6b-980514db5371") : secret "dns-default-metrics-tls" not found Jul 10 18:59:08.523359 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:08.523322 2560 generic.go:356] "Generic (PLEG): container finished" podID="eda6e317-2a4b-4591-a5ef-0139d0e57e95" containerID="015876771de0b23dbe1f3c06f1a78ebb4f8e7b403ce04c03c1ee311b2e331c8f" exitCode=0 Jul 10 18:59:08.523844 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:08.523375 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z58c2" event={"ID":"eda6e317-2a4b-4591-a5ef-0139d0e57e95","Type":"ContainerDied","Data":"015876771de0b23dbe1f3c06f1a78ebb4f8e7b403ce04c03c1ee311b2e331c8f"} Jul 10 18:59:09.529749 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:09.529476 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z58c2" event={"ID":"eda6e317-2a4b-4591-a5ef-0139d0e57e95","Type":"ContainerStarted","Data":"c137fb373b8a35fa6810d1851bd45301e40461e9eeccfb07b1104c1cc62a85fb"} Jul 10 18:59:09.530948 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:09.530925 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-z8vwd" event={"ID":"d95e6de4-e5cc-4467-a6b9-4ac8d842d744","Type":"ContainerStarted","Data":"a149b052b093dca47f6f21a76dbb4ad44e2564f8c68e9814b08d706b95a59556"} Jul 10 18:59:09.565843 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:09.565789 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-z58c2" podStartSLOduration=9.431814805 podStartE2EDuration="40.565772435s" totalImagesPullingTime="31.13395763s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:29 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:09.563553657 +0000 UTC m=+40.893694488" watchObservedRunningTime="2026-07-10 18:59:09.565772435 +0000 UTC m=+40.895913244" Jul 10 18:59:09.588013 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:09.587954 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-z8vwd" podStartSLOduration=2.291408715 podStartE2EDuration="4.58794027s" totalImagesPullingTime="2.296531555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:09.586815712 +0000 UTC m=+40.916956522" watchObservedRunningTime="2026-07-10 18:59:09.58794027 +0000 UTC m=+40.918081082" Jul 10 18:59:09.648180 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:09.648142 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9ea737c-ac9d-4694-bd6b-980514db5371-metrics-tls\") pod \"dns-default-d7l7x\" (UID: \"e9ea737c-ac9d-4694-bd6b-980514db5371\") " pod="openshift-dns/dns-default-d7l7x" Jul 10 18:59:09.648357 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:09.648275 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-registry-tls\") pod \"image-registry-84bdbb94b5-gnz5b\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 18:59:09.648357 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:09.648286 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 18:59:09.648508 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:09.648357 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9ea737c-ac9d-4694-bd6b-980514db5371-metrics-tls podName:e9ea737c-ac9d-4694-bd6b-980514db5371 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:13.64833652 +0000 UTC m=+44.978477309 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e9ea737c-ac9d-4694-bd6b-980514db5371-metrics-tls") pod "dns-default-d7l7x" (UID: "e9ea737c-ac9d-4694-bd6b-980514db5371") : secret "dns-default-metrics-tls" not found Jul 10 18:59:09.648508 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:09.648424 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 18:59:09.648508 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:09.648458 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84bdbb94b5-gnz5b: secret "image-registry-tls" not found Jul 10 18:59:09.648661 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:09.648516 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-registry-tls podName:24c51490-aab2-4fd0-9ffc-06886c5515b3 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:13.648501517 +0000 UTC m=+44.978642328 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-registry-tls") pod "image-registry-84bdbb94b5-gnz5b" (UID: "24c51490-aab2-4fd0-9ffc-06886c5515b3") : secret "image-registry-tls" not found Jul 10 18:59:09.752862 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:09.752819 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:09.872383 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:09.872292 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:09.991925 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:09.991881 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:10.081397 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:10.081363 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:10.185038 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:10.184990 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:10.284899 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:10.284862 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:10.943357 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:10.943318 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:11.012359 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:11.012306 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:12.545593 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:12.545560 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:12.602950 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:12.602900 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:13.676735 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:13.676691 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9ea737c-ac9d-4694-bd6b-980514db5371-metrics-tls\") pod \"dns-default-d7l7x\" (UID: \"e9ea737c-ac9d-4694-bd6b-980514db5371\") " pod="openshift-dns/dns-default-d7l7x" Jul 10 18:59:13.677125 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:13.676763 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-registry-tls\") pod \"image-registry-84bdbb94b5-gnz5b\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 18:59:13.677125 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:13.676858 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 18:59:13.677125 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:13.676922 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9ea737c-ac9d-4694-bd6b-980514db5371-metrics-tls podName:e9ea737c-ac9d-4694-bd6b-980514db5371 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:21.676905693 +0000 UTC m=+53.007046485 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e9ea737c-ac9d-4694-bd6b-980514db5371-metrics-tls") pod "dns-default-d7l7x" (UID: "e9ea737c-ac9d-4694-bd6b-980514db5371") : secret "dns-default-metrics-tls" not found Jul 10 18:59:13.677125 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:13.676861 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 18:59:13.677125 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:13.676962 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84bdbb94b5-gnz5b: secret "image-registry-tls" not found Jul 10 18:59:13.677125 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:13.677019 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-registry-tls podName:24c51490-aab2-4fd0-9ffc-06886c5515b3 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:21.677006701 +0000 UTC m=+53.007147492 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-registry-tls") pod "image-registry-84bdbb94b5-gnz5b" (UID: "24c51490-aab2-4fd0-9ffc-06886c5515b3") : secret "image-registry-tls" not found Jul 10 18:59:15.043640 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:15.043598 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:17.053466 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:17.053400 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:17.142556 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:17.142520 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:17.243295 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:17.243258 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:21.736394 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:21.736344 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9ea737c-ac9d-4694-bd6b-980514db5371-metrics-tls\") pod \"dns-default-d7l7x\" (UID: \"e9ea737c-ac9d-4694-bd6b-980514db5371\") " pod="openshift-dns/dns-default-d7l7x" Jul 10 18:59:21.736822 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:21.736420 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-registry-tls\") pod \"image-registry-84bdbb94b5-gnz5b\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 18:59:21.736822 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:21.736520 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 18:59:21.736822 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:21.736548 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 18:59:21.736822 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:21.736559 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84bdbb94b5-gnz5b: secret "image-registry-tls" not found Jul 10 18:59:21.736822 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:21.736598 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9ea737c-ac9d-4694-bd6b-980514db5371-metrics-tls podName:e9ea737c-ac9d-4694-bd6b-980514db5371 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:37.736581237 +0000 UTC m=+69.066722025 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e9ea737c-ac9d-4694-bd6b-980514db5371-metrics-tls") pod "dns-default-d7l7x" (UID: "e9ea737c-ac9d-4694-bd6b-980514db5371") : secret "dns-default-metrics-tls" not found Jul 10 18:59:21.736822 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:21.736634 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-registry-tls podName:24c51490-aab2-4fd0-9ffc-06886c5515b3 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:37.73662285 +0000 UTC m=+69.066763637 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-registry-tls") pod "image-registry-84bdbb94b5-gnz5b" (UID: "24c51490-aab2-4fd0-9ffc-06886c5515b3") : secret "image-registry-tls" not found Jul 10 18:59:31.511697 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:31.511669 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9l52t" Jul 10 18:59:31.684000 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:31.683966 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:31.780668 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:31.780585 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:31.880375 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:31.880348 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:31.944031 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:31.944000 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:34.936789 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:34.936743 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1bf574f-a4e2-4418-8f18-95fa0c80ca1e-metrics-certs\") pod \"network-metrics-daemon-v9b95\" (UID: \"b1bf574f-a4e2-4418-8f18-95fa0c80ca1e\") " pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 18:59:34.936789 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:34.936813 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3e3a914-d7de-4771-9d68-a7b05ab56b1f-original-pull-secret\") pod \"global-pull-secret-syncer-zv9t9\" (UID: \"e3e3a914-d7de-4771-9d68-a7b05ab56b1f\") " pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:59:34.941240 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:34.941220 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 18:59:34.941311 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:34.941238 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 18:59:34.947993 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:34.947972 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 18:59:34.948053 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:34.948046 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1bf574f-a4e2-4418-8f18-95fa0c80ca1e-metrics-certs podName:b1bf574f-a4e2-4418-8f18-95fa0c80ca1e nodeName:}" failed. No retries permitted until 2026-07-10 19:00:38.948026029 +0000 UTC m=+130.278166829 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b1bf574f-a4e2-4418-8f18-95fa0c80ca1e-metrics-certs") pod "network-metrics-daemon-v9b95" (UID: "b1bf574f-a4e2-4418-8f18-95fa0c80ca1e") : secret "metrics-daemon-secret" not found Jul 10 18:59:34.951781 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:34.951750 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3e3a914-d7de-4771-9d68-a7b05ab56b1f-original-pull-secret\") pod \"global-pull-secret-syncer-zv9t9\" (UID: \"e3e3a914-d7de-4771-9d68-a7b05ab56b1f\") " pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:59:35.037327 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:35.037273 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slg7w\" (UniqueName: \"kubernetes.io/projected/4526c66d-17d8-4ed2-9782-42b9b2d9215b-kube-api-access-slg7w\") pod \"network-check-target-bl7wj\" (UID: \"4526c66d-17d8-4ed2-9782-42b9b2d9215b\") " pod="openshift-network-diagnostics/network-check-target-bl7wj" Jul 10 18:59:35.041030 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:35.041011 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 18:59:35.051071 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:35.051045 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 18:59:35.061197 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:35.061168 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slg7w\" (UniqueName: \"kubernetes.io/projected/4526c66d-17d8-4ed2-9782-42b9b2d9215b-kube-api-access-slg7w\") pod \"network-check-target-bl7wj\" (UID: \"4526c66d-17d8-4ed2-9782-42b9b2d9215b\") " pod="openshift-network-diagnostics/network-check-target-bl7wj" Jul 10 18:59:35.165762 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:35.165733 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xwgdk\"" Jul 10 18:59:35.170146 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:35.170126 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zv9t9" Jul 10 18:59:35.172861 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:35.172840 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bl7wj" Jul 10 18:59:35.313476 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:35.311012 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zv9t9"] Jul 10 18:59:35.326631 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:35.326606 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bl7wj"] Jul 10 18:59:35.329828 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:59:35.329802 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4526c66d_17d8_4ed2_9782_42b9b2d9215b.slice/crio-04d82c9b5a5686fda4a44e78708f4b5d69665fccaa91d933a3a43acadfa1ab73 WatchSource:0}: Error finding container 04d82c9b5a5686fda4a44e78708f4b5d69665fccaa91d933a3a43acadfa1ab73: Status 404 returned error can't find the container with id 04d82c9b5a5686fda4a44e78708f4b5d69665fccaa91d933a3a43acadfa1ab73 Jul 10 18:59:35.592029 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:35.591931 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zv9t9" event={"ID":"e3e3a914-d7de-4771-9d68-a7b05ab56b1f","Type":"ContainerStarted","Data":"7694aec1cabe3d5fff92d1e8fe5bd03584623b1a56835eca62d4ade9e0c5a420"} Jul 10 18:59:35.592963 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:35.592940 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bl7wj" event={"ID":"4526c66d-17d8-4ed2-9782-42b9b2d9215b","Type":"ContainerStarted","Data":"04d82c9b5a5686fda4a44e78708f4b5d69665fccaa91d933a3a43acadfa1ab73"} Jul 10 18:59:37.757899 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:37.757853 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-registry-tls\") pod \"image-registry-84bdbb94b5-gnz5b\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 18:59:37.758516 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:37.757953 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9ea737c-ac9d-4694-bd6b-980514db5371-metrics-tls\") pod \"dns-default-d7l7x\" (UID: \"e9ea737c-ac9d-4694-bd6b-980514db5371\") " pod="openshift-dns/dns-default-d7l7x" Jul 10 18:59:37.758516 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:37.758036 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 18:59:37.758516 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:37.758053 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 18:59:37.758516 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:37.758060 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84bdbb94b5-gnz5b: secret "image-registry-tls" not found Jul 10 18:59:37.758516 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:37.758127 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9ea737c-ac9d-4694-bd6b-980514db5371-metrics-tls podName:e9ea737c-ac9d-4694-bd6b-980514db5371 nodeName:}" failed. No retries permitted until 2026-07-10 19:00:09.758106173 +0000 UTC m=+101.088246963 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e9ea737c-ac9d-4694-bd6b-980514db5371-metrics-tls") pod "dns-default-d7l7x" (UID: "e9ea737c-ac9d-4694-bd6b-980514db5371") : secret "dns-default-metrics-tls" not found Jul 10 18:59:37.758516 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:37.758146 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-registry-tls podName:24c51490-aab2-4fd0-9ffc-06886c5515b3 nodeName:}" failed. No retries permitted until 2026-07-10 19:00:09.758138137 +0000 UTC m=+101.088278939 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-registry-tls") pod "image-registry-84bdbb94b5-gnz5b" (UID: "24c51490-aab2-4fd0-9ffc-06886c5515b3") : secret "image-registry-tls" not found Jul 10 18:59:37.991912 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:37.991879 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:38.107103 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.107016 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:38.376240 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.376157 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-vshg7"] Jul 10 18:59:38.389029 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.388995 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-vshg7" Jul 10 18:59:38.398401 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.398278 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 10 18:59:38.399320 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.399297 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 18:59:38.399468 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.399299 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 10 18:59:38.399699 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.399679 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-s42rq\"" Jul 10 18:59:38.399699 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.399681 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 18:59:38.409702 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.409674 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-vshg7"] Jul 10 18:59:38.411552 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.411529 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 10 18:59:38.464681 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.464647 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88da616b-477f-491f-bf78-d80b9c187a24-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-vshg7\" (UID: \"88da616b-477f-491f-bf78-d80b9c187a24\") " pod="openshift-insights/insights-operator-68d9dd94cf-vshg7" Jul 10 18:59:38.464875 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.464715 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88da616b-477f-491f-bf78-d80b9c187a24-tmp\") pod \"insights-operator-68d9dd94cf-vshg7\" (UID: \"88da616b-477f-491f-bf78-d80b9c187a24\") " pod="openshift-insights/insights-operator-68d9dd94cf-vshg7" Jul 10 18:59:38.464875 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.464746 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/88da616b-477f-491f-bf78-d80b9c187a24-snapshots\") pod \"insights-operator-68d9dd94cf-vshg7\" (UID: \"88da616b-477f-491f-bf78-d80b9c187a24\") " pod="openshift-insights/insights-operator-68d9dd94cf-vshg7" Jul 10 18:59:38.464875 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.464763 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/88da616b-477f-491f-bf78-d80b9c187a24-serving-cert\") pod \"insights-operator-68d9dd94cf-vshg7\" (UID: \"88da616b-477f-491f-bf78-d80b9c187a24\") " pod="openshift-insights/insights-operator-68d9dd94cf-vshg7" Jul 10 18:59:38.464875 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.464799 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88da616b-477f-491f-bf78-d80b9c187a24-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-vshg7\" (UID: \"88da616b-477f-491f-bf78-d80b9c187a24\") " pod="openshift-insights/insights-operator-68d9dd94cf-vshg7" Jul 10 18:59:38.464875 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.464823 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7xlz\" (UniqueName: \"kubernetes.io/projected/88da616b-477f-491f-bf78-d80b9c187a24-kube-api-access-p7xlz\") pod \"insights-operator-68d9dd94cf-vshg7\" (UID: \"88da616b-477f-491f-bf78-d80b9c187a24\") " pod="openshift-insights/insights-operator-68d9dd94cf-vshg7" Jul 10 18:59:38.467662 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.467633 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dd8w8"] Jul 10 18:59:38.480578 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.480543 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rsc2t"] Jul 10 18:59:38.480758 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.480702 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dd8w8" Jul 10 18:59:38.484042 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.484020 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 10 18:59:38.484218 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.484196 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 10 18:59:38.484505 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.484484 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 10 18:59:38.484623 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.484539 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-qwss6\"" Jul 10 18:59:38.484774 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.484754 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 10 18:59:38.493526 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.493431 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rsc2t"] Jul 10 18:59:38.493526 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.493493 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dd8w8"] Jul 10 18:59:38.493754 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.493558 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rsc2t" Jul 10 18:59:38.500353 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.500318 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 10 18:59:38.500502 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.500368 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 10 18:59:38.500739 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.500716 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 10 18:59:38.500818 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.500748 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 10 18:59:38.501488 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.501430 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-dm99x\"" Jul 10 18:59:38.565199 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.565166 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/651e6091-4d1a-4adc-90bf-b0a7e5f8291c-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-dd8w8\" (UID: \"651e6091-4d1a-4adc-90bf-b0a7e5f8291c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dd8w8" Jul 10 18:59:38.565384 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.565211 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/651e6091-4d1a-4adc-90bf-b0a7e5f8291c-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-dd8w8\" (UID: \"651e6091-4d1a-4adc-90bf-b0a7e5f8291c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dd8w8" Jul 10 18:59:38.565384 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.565300 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88da616b-477f-491f-bf78-d80b9c187a24-tmp\") pod \"insights-operator-68d9dd94cf-vshg7\" (UID: \"88da616b-477f-491f-bf78-d80b9c187a24\") " pod="openshift-insights/insights-operator-68d9dd94cf-vshg7" Jul 10 18:59:38.565384 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.565365 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/88da616b-477f-491f-bf78-d80b9c187a24-snapshots\") pod \"insights-operator-68d9dd94cf-vshg7\" (UID: \"88da616b-477f-491f-bf78-d80b9c187a24\") " pod="openshift-insights/insights-operator-68d9dd94cf-vshg7" Jul 10 18:59:38.565575 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.565393 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/66525ef2-fac3-4d3d-b138-cded3b6c0bef-config\") pod \"service-ca-operator-7695fc6c8c-rsc2t\" (UID: \"66525ef2-fac3-4d3d-b138-cded3b6c0bef\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rsc2t" Jul 10 18:59:38.565575 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.565421 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp6tr\" (UniqueName: \"kubernetes.io/projected/651e6091-4d1a-4adc-90bf-b0a7e5f8291c-kube-api-access-zp6tr\") pod \"kube-storage-version-migrator-operator-799cb7f958-dd8w8\" (UID: \"651e6091-4d1a-4adc-90bf-b0a7e5f8291c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dd8w8" Jul 10 18:59:38.565575 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.565512 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/88da616b-477f-491f-bf78-d80b9c187a24-serving-cert\") pod \"insights-operator-68d9dd94cf-vshg7\" (UID: \"88da616b-477f-491f-bf78-d80b9c187a24\") " pod="openshift-insights/insights-operator-68d9dd94cf-vshg7" Jul 10 18:59:38.565575 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.565540 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/66525ef2-fac3-4d3d-b138-cded3b6c0bef-serving-cert\") pod \"service-ca-operator-7695fc6c8c-rsc2t\" (UID: \"66525ef2-fac3-4d3d-b138-cded3b6c0bef\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rsc2t" Jul 10 18:59:38.565747 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.565590 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88da616b-477f-491f-bf78-d80b9c187a24-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-vshg7\" (UID: \"88da616b-477f-491f-bf78-d80b9c187a24\") " pod="openshift-insights/insights-operator-68d9dd94cf-vshg7" Jul 10 18:59:38.565747 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.565610 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7xlz\" (UniqueName: \"kubernetes.io/projected/88da616b-477f-491f-bf78-d80b9c187a24-kube-api-access-p7xlz\") pod \"insights-operator-68d9dd94cf-vshg7\" (UID: \"88da616b-477f-491f-bf78-d80b9c187a24\") " pod="openshift-insights/insights-operator-68d9dd94cf-vshg7" Jul 10 18:59:38.565747 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.565631 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r92bw\" (UniqueName: \"kubernetes.io/projected/66525ef2-fac3-4d3d-b138-cded3b6c0bef-kube-api-access-r92bw\") pod \"service-ca-operator-7695fc6c8c-rsc2t\" (UID: \"66525ef2-fac3-4d3d-b138-cded3b6c0bef\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rsc2t" Jul 10 18:59:38.565747 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.565680 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88da616b-477f-491f-bf78-d80b9c187a24-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-vshg7\" (UID: \"88da616b-477f-491f-bf78-d80b9c187a24\") " pod="openshift-insights/insights-operator-68d9dd94cf-vshg7" Jul 10 18:59:38.565905 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.565774 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88da616b-477f-491f-bf78-d80b9c187a24-tmp\") pod \"insights-operator-68d9dd94cf-vshg7\" (UID: \"88da616b-477f-491f-bf78-d80b9c187a24\") " pod="openshift-insights/insights-operator-68d9dd94cf-vshg7" Jul 10 18:59:38.566048 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.566029 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/88da616b-477f-491f-bf78-d80b9c187a24-snapshots\") pod \"insights-operator-68d9dd94cf-vshg7\" (UID: \"88da616b-477f-491f-bf78-d80b9c187a24\") " pod="openshift-insights/insights-operator-68d9dd94cf-vshg7" Jul 10 18:59:38.566221 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.566196 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88da616b-477f-491f-bf78-d80b9c187a24-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-vshg7\" (UID: \"88da616b-477f-491f-bf78-d80b9c187a24\") " pod="openshift-insights/insights-operator-68d9dd94cf-vshg7" Jul 10 18:59:38.566615 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.566589 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88da616b-477f-491f-bf78-d80b9c187a24-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-vshg7\" (UID: \"88da616b-477f-491f-bf78-d80b9c187a24\") " pod="openshift-insights/insights-operator-68d9dd94cf-vshg7" Jul 10 18:59:38.568654 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.568629 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/88da616b-477f-491f-bf78-d80b9c187a24-serving-cert\") pod \"insights-operator-68d9dd94cf-vshg7\" (UID: \"88da616b-477f-491f-bf78-d80b9c187a24\") " pod="openshift-insights/insights-operator-68d9dd94cf-vshg7" Jul 10 18:59:38.576157 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.576136 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7xlz\" (UniqueName: \"kubernetes.io/projected/88da616b-477f-491f-bf78-d80b9c187a24-kube-api-access-p7xlz\") pod \"insights-operator-68d9dd94cf-vshg7\" (UID: \"88da616b-477f-491f-bf78-d80b9c187a24\") " pod="openshift-insights/insights-operator-68d9dd94cf-vshg7" Jul 10 18:59:38.636404 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.636311 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:38.667106 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.667062 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/66525ef2-fac3-4d3d-b138-cded3b6c0bef-config\") pod \"service-ca-operator-7695fc6c8c-rsc2t\" (UID: \"66525ef2-fac3-4d3d-b138-cded3b6c0bef\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rsc2t" Jul 10 18:59:38.667299 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.667113 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp6tr\" (UniqueName: \"kubernetes.io/projected/651e6091-4d1a-4adc-90bf-b0a7e5f8291c-kube-api-access-zp6tr\") pod \"kube-storage-version-migrator-operator-799cb7f958-dd8w8\" (UID: \"651e6091-4d1a-4adc-90bf-b0a7e5f8291c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dd8w8" Jul 10 18:59:38.667299 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.667144 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/66525ef2-fac3-4d3d-b138-cded3b6c0bef-serving-cert\") pod \"service-ca-operator-7695fc6c8c-rsc2t\" (UID: \"66525ef2-fac3-4d3d-b138-cded3b6c0bef\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rsc2t" Jul 10 18:59:38.667299 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.667193 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r92bw\" (UniqueName: \"kubernetes.io/projected/66525ef2-fac3-4d3d-b138-cded3b6c0bef-kube-api-access-r92bw\") pod \"service-ca-operator-7695fc6c8c-rsc2t\" (UID: \"66525ef2-fac3-4d3d-b138-cded3b6c0bef\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rsc2t" Jul 10 18:59:38.667477 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.667349 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/651e6091-4d1a-4adc-90bf-b0a7e5f8291c-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-dd8w8\" (UID: \"651e6091-4d1a-4adc-90bf-b0a7e5f8291c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dd8w8" Jul 10 18:59:38.667477 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.667401 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/651e6091-4d1a-4adc-90bf-b0a7e5f8291c-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-dd8w8\" (UID: \"651e6091-4d1a-4adc-90bf-b0a7e5f8291c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dd8w8" Jul 10 18:59:38.667879 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.667818 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/66525ef2-fac3-4d3d-b138-cded3b6c0bef-config\") pod \"service-ca-operator-7695fc6c8c-rsc2t\" (UID: \"66525ef2-fac3-4d3d-b138-cded3b6c0bef\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rsc2t" Jul 10 18:59:38.668036 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.668013 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/651e6091-4d1a-4adc-90bf-b0a7e5f8291c-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-dd8w8\" (UID: \"651e6091-4d1a-4adc-90bf-b0a7e5f8291c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dd8w8" Jul 10 18:59:38.669623 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.669603 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/66525ef2-fac3-4d3d-b138-cded3b6c0bef-serving-cert\") pod \"service-ca-operator-7695fc6c8c-rsc2t\" (UID: \"66525ef2-fac3-4d3d-b138-cded3b6c0bef\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rsc2t" Jul 10 18:59:38.679988 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.679959 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/651e6091-4d1a-4adc-90bf-b0a7e5f8291c-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-dd8w8\" (UID: \"651e6091-4d1a-4adc-90bf-b0a7e5f8291c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dd8w8" Jul 10 18:59:38.681582 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.681553 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp6tr\" (UniqueName: \"kubernetes.io/projected/651e6091-4d1a-4adc-90bf-b0a7e5f8291c-kube-api-access-zp6tr\") pod \"kube-storage-version-migrator-operator-799cb7f958-dd8w8\" (UID: \"651e6091-4d1a-4adc-90bf-b0a7e5f8291c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dd8w8" Jul 10 18:59:38.683103 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.683073 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r92bw\" (UniqueName: \"kubernetes.io/projected/66525ef2-fac3-4d3d-b138-cded3b6c0bef-kube-api-access-r92bw\") pod \"service-ca-operator-7695fc6c8c-rsc2t\" (UID: \"66525ef2-fac3-4d3d-b138-cded3b6c0bef\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rsc2t" Jul 10 18:59:38.700172 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.700141 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-vshg7" Jul 10 18:59:38.732696 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.732662 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:38.792782 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.792739 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dd8w8" Jul 10 18:59:38.804579 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:38.804546 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rsc2t" Jul 10 18:59:40.285380 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:40.285315 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dd8w8"] Jul 10 18:59:40.293459 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:59:40.293417 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod651e6091_4d1a_4adc_90bf_b0a7e5f8291c.slice/crio-3288111cade89b5ecc2c7c4182fb48abaa3d2c0640fccf41f0ef54168445f030 WatchSource:0}: Error finding container 3288111cade89b5ecc2c7c4182fb48abaa3d2c0640fccf41f0ef54168445f030: Status 404 returned error can't find the container with id 3288111cade89b5ecc2c7c4182fb48abaa3d2c0640fccf41f0ef54168445f030 Jul 10 18:59:40.304356 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:40.304325 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-vshg7"] Jul 10 18:59:40.307481 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:59:40.307456 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88da616b_477f_491f_bf78_d80b9c187a24.slice/crio-5a371112ed01d1eadf8c1c8326f7a2f23d1c2b5c75f957b91b4888558e80adff WatchSource:0}: Error finding container 5a371112ed01d1eadf8c1c8326f7a2f23d1c2b5c75f957b91b4888558e80adff: Status 404 returned error can't find the container with id 5a371112ed01d1eadf8c1c8326f7a2f23d1c2b5c75f957b91b4888558e80adff Jul 10 18:59:40.325705 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:40.325647 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rsc2t"] Jul 10 18:59:40.328424 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:59:40.328400 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66525ef2_fac3_4d3d_b138_cded3b6c0bef.slice/crio-ccefe6fa6a52cadb1cdc26152622203c07f9a5b00df659a2c744d5bc3b8a8203 WatchSource:0}: Error finding container ccefe6fa6a52cadb1cdc26152622203c07f9a5b00df659a2c744d5bc3b8a8203: Status 404 returned error can't find the container with id ccefe6fa6a52cadb1cdc26152622203c07f9a5b00df659a2c744d5bc3b8a8203 Jul 10 18:59:40.607293 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:40.607256 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rsc2t" event={"ID":"66525ef2-fac3-4d3d-b138-cded3b6c0bef","Type":"ContainerStarted","Data":"ccefe6fa6a52cadb1cdc26152622203c07f9a5b00df659a2c744d5bc3b8a8203"} Jul 10 18:59:40.608586 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:40.608560 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zv9t9" event={"ID":"e3e3a914-d7de-4771-9d68-a7b05ab56b1f","Type":"ContainerStarted","Data":"9259a6f3dc9e2ebb8c135558f6aeb53a040b43e4c90584cf05450e7d09e0718f"} Jul 10 18:59:40.609919 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:40.609893 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bl7wj" event={"ID":"4526c66d-17d8-4ed2-9782-42b9b2d9215b","Type":"ContainerStarted","Data":"111d3f08a93622886e827dcd7d8f10cf1aa2b58a3a1df13c5725d78fd1f2f5dc"} Jul 10 18:59:40.610035 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:40.609958 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bl7wj" Jul 10 18:59:40.610922 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:40.610903 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dd8w8" event={"ID":"651e6091-4d1a-4adc-90bf-b0a7e5f8291c","Type":"ContainerStarted","Data":"3288111cade89b5ecc2c7c4182fb48abaa3d2c0640fccf41f0ef54168445f030"} Jul 10 18:59:40.611826 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:40.611810 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-vshg7" event={"ID":"88da616b-477f-491f-bf78-d80b9c187a24","Type":"ContainerStarted","Data":"5a371112ed01d1eadf8c1c8326f7a2f23d1c2b5c75f957b91b4888558e80adff"} Jul 10 18:59:40.625762 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:40.625715 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zv9t9" podStartSLOduration=66.793759296 podStartE2EDuration="1m11.625699494s" totalImagesPullingTime="4.831940198s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:40.625052585 +0000 UTC m=+71.955193396" watchObservedRunningTime="2026-07-10 18:59:40.625699494 +0000 UTC m=+71.955840282" Jul 10 18:59:40.642555 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:40.642511 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bl7wj" podStartSLOduration=66.84369213 podStartE2EDuration="1m11.642496643s" totalImagesPullingTime="4.798804513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:58:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:40.64234497 +0000 UTC m=+71.972485806" watchObservedRunningTime="2026-07-10 18:59:40.642496643 +0000 UTC m=+71.972637431" Jul 10 18:59:41.045093 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:41.045060 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:41.102014 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:41.101974 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:43.622762 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:43.622721 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dd8w8" event={"ID":"651e6091-4d1a-4adc-90bf-b0a7e5f8291c","Type":"ContainerStarted","Data":"9280ddd1fe48ff29ca376c36196460230792b1aee57c82b60f4313c7a42de828"} Jul 10 18:59:43.624504 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:43.624475 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-vshg7" event={"ID":"88da616b-477f-491f-bf78-d80b9c187a24","Type":"ContainerStarted","Data":"fa85fe52d26954646c65b2ded834436e5939b39bb4eef1693c52dea90057d6c6"} Jul 10 18:59:43.625944 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:43.625919 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rsc2t" event={"ID":"66525ef2-fac3-4d3d-b138-cded3b6c0bef","Type":"ContainerStarted","Data":"96c66babfccd98de5d30110574c47faa2c0f62d7afe0feaa46c919a4fdba3f87"} Jul 10 18:59:43.641189 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:43.641129 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dd8w8" podStartSLOduration=2.798217966 podStartE2EDuration="5.641107345s" totalImagesPullingTime="2.842889379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:43.640736116 +0000 UTC m=+74.970876928" watchObservedRunningTime="2026-07-10 18:59:43.641107345 +0000 UTC m=+74.971248155" Jul 10 18:59:43.660779 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:43.660715 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-vshg7" podStartSLOduration=2.834730176 podStartE2EDuration="5.660695166s" totalImagesPullingTime="2.82596499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:43.660316744 +0000 UTC m=+74.990457565" watchObservedRunningTime="2026-07-10 18:59:43.660695166 +0000 UTC m=+74.990835977" Jul 10 18:59:43.678084 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:43.678023 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rsc2t" podStartSLOduration=2.87564861 podStartE2EDuration="5.678000064s" totalImagesPullingTime="2.802351454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:43.67698891 +0000 UTC m=+75.007129742" watchObservedRunningTime="2026-07-10 18:59:43.678000064 +0000 UTC m=+75.008140882" Jul 10 18:59:44.100265 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:44.100227 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sdt64"] Jul 10 18:59:44.103498 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:44.103470 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sdt64" Jul 10 18:59:44.112434 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:44.112408 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 18:59:44.112627 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:44.112553 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8f6gd\"" Jul 10 18:59:44.112871 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:44.112854 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 18:59:44.123652 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:44.123610 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sdt64"] Jul 10 18:59:44.154538 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:44.154508 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:44.213703 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:44.213667 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-data-volume\") pod \"insights-runtime-extractor-sdt64\" (UID: \"06a74dc2-5187-47af-9eeb-77f4ce8a8d7a\") " pod="openshift-insights/insights-runtime-extractor-sdt64" Jul 10 18:59:44.213883 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:44.213744 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-846kk\" (UniqueName: \"kubernetes.io/projected/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-kube-api-access-846kk\") pod \"insights-runtime-extractor-sdt64\" (UID: \"06a74dc2-5187-47af-9eeb-77f4ce8a8d7a\") " pod="openshift-insights/insights-runtime-extractor-sdt64" Jul 10 18:59:44.213883 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:44.213797 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-crio-socket\") pod \"insights-runtime-extractor-sdt64\" (UID: \"06a74dc2-5187-47af-9eeb-77f4ce8a8d7a\") " pod="openshift-insights/insights-runtime-extractor-sdt64" Jul 10 18:59:44.213883 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:44.213824 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sdt64\" (UID: \"06a74dc2-5187-47af-9eeb-77f4ce8a8d7a\") " pod="openshift-insights/insights-runtime-extractor-sdt64" Jul 10 18:59:44.213883 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:44.213857 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sdt64\" (UID: \"06a74dc2-5187-47af-9eeb-77f4ce8a8d7a\") " pod="openshift-insights/insights-runtime-extractor-sdt64" Jul 10 18:59:44.314968 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:44.314926 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-846kk\" (UniqueName: \"kubernetes.io/projected/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-kube-api-access-846kk\") pod \"insights-runtime-extractor-sdt64\" (UID: \"06a74dc2-5187-47af-9eeb-77f4ce8a8d7a\") " pod="openshift-insights/insights-runtime-extractor-sdt64" Jul 10 18:59:44.314968 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:44.314975 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-crio-socket\") pod \"insights-runtime-extractor-sdt64\" (UID: \"06a74dc2-5187-47af-9eeb-77f4ce8a8d7a\") " pod="openshift-insights/insights-runtime-extractor-sdt64" Jul 10 18:59:44.315219 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:44.314999 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sdt64\" (UID: \"06a74dc2-5187-47af-9eeb-77f4ce8a8d7a\") " pod="openshift-insights/insights-runtime-extractor-sdt64" Jul 10 18:59:44.315219 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:44.315086 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 18:59:44.315219 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:44.315089 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-crio-socket\") pod \"insights-runtime-extractor-sdt64\" (UID: \"06a74dc2-5187-47af-9eeb-77f4ce8a8d7a\") " pod="openshift-insights/insights-runtime-extractor-sdt64" Jul 10 18:59:44.315219 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:44.315099 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sdt64\" (UID: \"06a74dc2-5187-47af-9eeb-77f4ce8a8d7a\") " pod="openshift-insights/insights-runtime-extractor-sdt64" Jul 10 18:59:44.315219 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:44.315146 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-insights-runtime-extractor-tls podName:06a74dc2-5187-47af-9eeb-77f4ce8a8d7a nodeName:}" failed. No retries permitted until 2026-07-10 18:59:44.815126991 +0000 UTC m=+76.145267795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sdt64" (UID: "06a74dc2-5187-47af-9eeb-77f4ce8a8d7a") : secret "insights-runtime-extractor-tls" not found Jul 10 18:59:44.315219 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:44.315145 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 18:59:44.315219 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:44.315200 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-kube-rbac-proxy-cm podName:06a74dc2-5187-47af-9eeb-77f4ce8a8d7a nodeName:}" failed. No retries permitted until 2026-07-10 18:59:44.815189303 +0000 UTC m=+76.145330094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sdt64" (UID: "06a74dc2-5187-47af-9eeb-77f4ce8a8d7a") : configmap "kube-rbac-proxy" not found Jul 10 18:59:44.315586 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:44.315250 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-data-volume\") pod \"insights-runtime-extractor-sdt64\" (UID: \"06a74dc2-5187-47af-9eeb-77f4ce8a8d7a\") " pod="openshift-insights/insights-runtime-extractor-sdt64" Jul 10 18:59:44.315586 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:44.315571 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-data-volume\") pod \"insights-runtime-extractor-sdt64\" (UID: \"06a74dc2-5187-47af-9eeb-77f4ce8a8d7a\") " pod="openshift-insights/insights-runtime-extractor-sdt64" Jul 10 18:59:44.326603 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:44.326577 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-846kk\" (UniqueName: \"kubernetes.io/projected/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-kube-api-access-846kk\") pod \"insights-runtime-extractor-sdt64\" (UID: \"06a74dc2-5187-47af-9eeb-77f4ce8a8d7a\") " pod="openshift-insights/insights-runtime-extractor-sdt64" Jul 10 18:59:44.626647 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:44.626539 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:44.713981 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:44.713942 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:44.819762 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:44.819720 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sdt64\" (UID: \"06a74dc2-5187-47af-9eeb-77f4ce8a8d7a\") " pod="openshift-insights/insights-runtime-extractor-sdt64" Jul 10 18:59:44.819935 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:44.819850 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sdt64\" (UID: \"06a74dc2-5187-47af-9eeb-77f4ce8a8d7a\") " pod="openshift-insights/insights-runtime-extractor-sdt64" Jul 10 18:59:44.819935 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:44.819870 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 18:59:44.820011 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:44.819946 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-kube-rbac-proxy-cm podName:06a74dc2-5187-47af-9eeb-77f4ce8a8d7a nodeName:}" failed. No retries permitted until 2026-07-10 18:59:45.819925379 +0000 UTC m=+77.150066185 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sdt64" (UID: "06a74dc2-5187-47af-9eeb-77f4ce8a8d7a") : configmap "kube-rbac-proxy" not found Jul 10 18:59:44.820011 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:44.819957 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 18:59:44.820011 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:44.819991 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-insights-runtime-extractor-tls podName:06a74dc2-5187-47af-9eeb-77f4ce8a8d7a nodeName:}" failed. No retries permitted until 2026-07-10 18:59:45.819981165 +0000 UTC m=+77.150121959 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sdt64" (UID: "06a74dc2-5187-47af-9eeb-77f4ce8a8d7a") : secret "insights-runtime-extractor-tls" not found Jul 10 18:59:44.823033 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:44.823007 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:44.921052 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:44.921018 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:45.498151 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:45.498118 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:45.598612 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:45.598577 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:45.826795 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:45.826712 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sdt64\" (UID: \"06a74dc2-5187-47af-9eeb-77f4ce8a8d7a\") " pod="openshift-insights/insights-runtime-extractor-sdt64" Jul 10 18:59:45.826795 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:45.826754 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sdt64\" (UID: \"06a74dc2-5187-47af-9eeb-77f4ce8a8d7a\") " pod="openshift-insights/insights-runtime-extractor-sdt64" Jul 10 18:59:45.827179 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:45.826878 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 18:59:45.827179 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:45.826941 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-insights-runtime-extractor-tls podName:06a74dc2-5187-47af-9eeb-77f4ce8a8d7a nodeName:}" failed. No retries permitted until 2026-07-10 18:59:47.826925984 +0000 UTC m=+79.157066773 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sdt64" (UID: "06a74dc2-5187-47af-9eeb-77f4ce8a8d7a") : secret "insights-runtime-extractor-tls" not found Jul 10 18:59:45.827179 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:45.826879 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 18:59:45.827179 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:45.826982 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-kube-rbac-proxy-cm podName:06a74dc2-5187-47af-9eeb-77f4ce8a8d7a nodeName:}" failed. No retries permitted until 2026-07-10 18:59:47.826971367 +0000 UTC m=+79.157112154 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sdt64" (UID: "06a74dc2-5187-47af-9eeb-77f4ce8a8d7a") : configmap "kube-rbac-proxy" not found Jul 10 18:59:45.912511 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:45.912487 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l9qmm_d85934ea-40c2-4589-aca7-b0e8efca569c/dns-node-resolver/0.log" Jul 10 18:59:46.194619 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:46.194582 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:47.110803 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:47.110769 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-xzp6q"] Jul 10 18:59:47.113097 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:47.113074 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-xzp6q" Jul 10 18:59:47.118615 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:47.118590 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 10 18:59:47.118942 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:47.118924 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 10 18:59:47.118998 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:47.118958 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 10 18:59:47.119049 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:47.119014 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-t8b28\"" Jul 10 18:59:47.121504 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:47.121483 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 10 18:59:47.143869 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:47.143809 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-xzp6q"] Jul 10 18:59:47.156376 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:47.156345 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:47.227691 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:47.227649 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:47.237728 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:47.237697 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0b0aaed3-8bca-46cc-b5a6-d5416ea66f49-signing-cabundle\") pod \"service-ca-79f68b9f4f-xzp6q\" (UID: \"0b0aaed3-8bca-46cc-b5a6-d5416ea66f49\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xzp6q" Jul 10 18:59:47.237839 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:47.237821 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4zp9\" (UniqueName: \"kubernetes.io/projected/0b0aaed3-8bca-46cc-b5a6-d5416ea66f49-kube-api-access-f4zp9\") pod \"service-ca-79f68b9f4f-xzp6q\" (UID: \"0b0aaed3-8bca-46cc-b5a6-d5416ea66f49\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xzp6q" Jul 10 18:59:47.237892 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:47.237849 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0b0aaed3-8bca-46cc-b5a6-d5416ea66f49-signing-key\") pod \"service-ca-79f68b9f4f-xzp6q\" (UID: \"0b0aaed3-8bca-46cc-b5a6-d5416ea66f49\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xzp6q" Jul 10 18:59:47.314575 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:47.314546 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x6sx7_fd6564ad-086b-4c7b-ada7-99f16bfe03c4/node-ca/0.log" Jul 10 18:59:47.338568 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:47.338531 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4zp9\" (UniqueName: \"kubernetes.io/projected/0b0aaed3-8bca-46cc-b5a6-d5416ea66f49-kube-api-access-f4zp9\") pod \"service-ca-79f68b9f4f-xzp6q\" (UID: \"0b0aaed3-8bca-46cc-b5a6-d5416ea66f49\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xzp6q" Jul 10 18:59:47.338568 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:47.338570 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0b0aaed3-8bca-46cc-b5a6-d5416ea66f49-signing-key\") pod \"service-ca-79f68b9f4f-xzp6q\" (UID: \"0b0aaed3-8bca-46cc-b5a6-d5416ea66f49\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xzp6q" Jul 10 18:59:47.338753 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:47.338626 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0b0aaed3-8bca-46cc-b5a6-d5416ea66f49-signing-cabundle\") pod \"service-ca-79f68b9f4f-xzp6q\" (UID: \"0b0aaed3-8bca-46cc-b5a6-d5416ea66f49\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xzp6q" Jul 10 18:59:47.339319 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:47.339293 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0b0aaed3-8bca-46cc-b5a6-d5416ea66f49-signing-cabundle\") pod \"service-ca-79f68b9f4f-xzp6q\" (UID: \"0b0aaed3-8bca-46cc-b5a6-d5416ea66f49\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xzp6q" Jul 10 18:59:47.340978 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:47.340956 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0b0aaed3-8bca-46cc-b5a6-d5416ea66f49-signing-key\") pod \"service-ca-79f68b9f4f-xzp6q\" (UID: \"0b0aaed3-8bca-46cc-b5a6-d5416ea66f49\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xzp6q" Jul 10 18:59:47.357432 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:47.357404 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4zp9\" (UniqueName: \"kubernetes.io/projected/0b0aaed3-8bca-46cc-b5a6-d5416ea66f49-kube-api-access-f4zp9\") pod \"service-ca-79f68b9f4f-xzp6q\" (UID: \"0b0aaed3-8bca-46cc-b5a6-d5416ea66f49\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xzp6q" Jul 10 18:59:47.422436 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:47.422397 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-xzp6q" Jul 10 18:59:47.553261 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:47.553185 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-xzp6q"] Jul 10 18:59:47.555614 ip-10-0-138-213 kubenswrapper[2560]: W0710 18:59:47.555586 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b0aaed3_8bca_46cc_b5a6_d5416ea66f49.slice/crio-7f81ff596832acf4c6bb9737a6d93ed3a25fa80fec93c4e0d1519b77951273f3 WatchSource:0}: Error finding container 7f81ff596832acf4c6bb9737a6d93ed3a25fa80fec93c4e0d1519b77951273f3: Status 404 returned error can't find the container with id 7f81ff596832acf4c6bb9737a6d93ed3a25fa80fec93c4e0d1519b77951273f3 Jul 10 18:59:47.640496 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:47.640452 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-xzp6q" event={"ID":"0b0aaed3-8bca-46cc-b5a6-d5416ea66f49","Type":"ContainerStarted","Data":"7f39db9c5d856ca212cca1df05bb22ea13582683183417eaddaf6bbb179e2524"} Jul 10 18:59:47.640680 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:47.640518 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-xzp6q" event={"ID":"0b0aaed3-8bca-46cc-b5a6-d5416ea66f49","Type":"ContainerStarted","Data":"7f81ff596832acf4c6bb9737a6d93ed3a25fa80fec93c4e0d1519b77951273f3"} Jul 10 18:59:47.660485 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:47.660415 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-xzp6q" podStartSLOduration=0.660398387 podStartE2EDuration="660.398387ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:47.660344594 +0000 UTC m=+78.990485404" watchObservedRunningTime="2026-07-10 18:59:47.660398387 +0000 UTC m=+78.990539239" Jul 10 18:59:47.841697 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:47.841607 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sdt64\" (UID: \"06a74dc2-5187-47af-9eeb-77f4ce8a8d7a\") " pod="openshift-insights/insights-runtime-extractor-sdt64" Jul 10 18:59:47.841697 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:47.841657 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sdt64\" (UID: \"06a74dc2-5187-47af-9eeb-77f4ce8a8d7a\") " pod="openshift-insights/insights-runtime-extractor-sdt64" Jul 10 18:59:47.841878 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:47.841768 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 18:59:47.841878 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:47.841809 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 18:59:47.841878 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:47.841827 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-insights-runtime-extractor-tls podName:06a74dc2-5187-47af-9eeb-77f4ce8a8d7a nodeName:}" failed. No retries permitted until 2026-07-10 18:59:51.841813257 +0000 UTC m=+83.171954048 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sdt64" (UID: "06a74dc2-5187-47af-9eeb-77f4ce8a8d7a") : secret "insights-runtime-extractor-tls" not found Jul 10 18:59:47.841878 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:47.841870 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-kube-rbac-proxy-cm podName:06a74dc2-5187-47af-9eeb-77f4ce8a8d7a nodeName:}" failed. No retries permitted until 2026-07-10 18:59:51.841852665 +0000 UTC m=+83.171993452 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sdt64" (UID: "06a74dc2-5187-47af-9eeb-77f4ce8a8d7a") : configmap "kube-rbac-proxy" not found Jul 10 18:59:48.108284 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:48.108212 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-z8vwd_d95e6de4-e5cc-4467-a6b9-4ac8d842d744/serve-healthcheck-canary/0.log" Jul 10 18:59:48.109358 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:48.109338 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:48.193430 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:48.193399 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:48.336832 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:48.336796 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:48.636287 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:48.636257 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:48.814756 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:48.814710 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:48.912578 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:48.912543 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:48.978797 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:48.978764 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:49.727175 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:49.727140 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:50.315806 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:50.315759 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:50.526241 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:50.526209 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:50.724723 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:50.724687 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:50.918769 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:50.918738 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:51.003298 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:51.003219 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:51.876461 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:51.876404 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sdt64\" (UID: \"06a74dc2-5187-47af-9eeb-77f4ce8a8d7a\") " pod="openshift-insights/insights-runtime-extractor-sdt64" Jul 10 18:59:51.876648 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:51.876475 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sdt64\" (UID: \"06a74dc2-5187-47af-9eeb-77f4ce8a8d7a\") " pod="openshift-insights/insights-runtime-extractor-sdt64" Jul 10 18:59:51.876648 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:51.876572 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 18:59:51.876648 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:51.876600 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 18:59:51.876648 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:51.876633 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-insights-runtime-extractor-tls podName:06a74dc2-5187-47af-9eeb-77f4ce8a8d7a nodeName:}" failed. No retries permitted until 2026-07-10 18:59:59.87661716 +0000 UTC m=+91.206757952 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sdt64" (UID: "06a74dc2-5187-47af-9eeb-77f4ce8a8d7a") : secret "insights-runtime-extractor-tls" not found Jul 10 18:59:51.876792 ip-10-0-138-213 kubenswrapper[2560]: E0710 18:59:51.876654 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-kube-rbac-proxy-cm podName:06a74dc2-5187-47af-9eeb-77f4ce8a8d7a nodeName:}" failed. No retries permitted until 2026-07-10 18:59:59.876639422 +0000 UTC m=+91.206780210 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sdt64" (UID: "06a74dc2-5187-47af-9eeb-77f4ce8a8d7a") : configmap "kube-rbac-proxy" not found Jul 10 18:59:51.952625 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:51.952590 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:52.359404 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:52.359364 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:52.433327 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:52.433297 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:52.536455 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:52.536409 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:52.793228 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:52.793198 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:52.898081 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:52.898030 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:53.157318 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:53.157239 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:53.262179 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:53.262146 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:53.929361 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:53.929330 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:54.140220 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:54.140186 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:54.520571 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:54.520543 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:54.719992 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:54.719959 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:54.816384 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:54.816308 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:54.935730 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:54.935703 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:56.131110 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:56.131079 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:56.334204 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:56.334171 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:56.540659 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:56.540606 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:56.730329 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:56.730290 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:56.921280 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:56.921250 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:57.296148 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:57.296064 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:57.545413 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:57.545380 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:57.605467 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:57.605376 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:57.917473 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:57.917429 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:58.331223 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:58.331147 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:58.510594 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:58.510566 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:58.717513 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:58.717481 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:58.925507 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:58.925475 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:59.323903 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:59.323867 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:59.608395 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:59.608329 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 18:59:59.947464 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:59.947395 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sdt64\" (UID: \"06a74dc2-5187-47af-9eeb-77f4ce8a8d7a\") " pod="openshift-insights/insights-runtime-extractor-sdt64" Jul 10 18:59:59.947464 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:59.947471 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sdt64\" (UID: \"06a74dc2-5187-47af-9eeb-77f4ce8a8d7a\") " pod="openshift-insights/insights-runtime-extractor-sdt64" Jul 10 18:59:59.947996 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:59.947979 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sdt64\" (UID: \"06a74dc2-5187-47af-9eeb-77f4ce8a8d7a\") " pod="openshift-insights/insights-runtime-extractor-sdt64" Jul 10 18:59:59.949663 ip-10-0-138-213 kubenswrapper[2560]: I0710 18:59:59.949641 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/06a74dc2-5187-47af-9eeb-77f4ce8a8d7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sdt64\" (UID: \"06a74dc2-5187-47af-9eeb-77f4ce8a8d7a\") " pod="openshift-insights/insights-runtime-extractor-sdt64" Jul 10 19:00:00.012006 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:00.011968 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sdt64" Jul 10 19:00:00.141250 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:00.141221 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sdt64"] Jul 10 19:00:00.144416 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:00:00.144387 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06a74dc2_5187_47af_9eeb_77f4ce8a8d7a.slice/crio-5cced19cfcd48115db7b06cb3892beac6b4eaea6950d36883d65a96d649c1990 WatchSource:0}: Error finding container 5cced19cfcd48115db7b06cb3892beac6b4eaea6950d36883d65a96d649c1990: Status 404 returned error can't find the container with id 5cced19cfcd48115db7b06cb3892beac6b4eaea6950d36883d65a96d649c1990 Jul 10 19:00:00.685585 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:00.685482 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sdt64" event={"ID":"06a74dc2-5187-47af-9eeb-77f4ce8a8d7a","Type":"ContainerStarted","Data":"5cced19cfcd48115db7b06cb3892beac6b4eaea6950d36883d65a96d649c1990"} Jul 10 19:00:04.701831 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:04.701799 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sdt64" event={"ID":"06a74dc2-5187-47af-9eeb-77f4ce8a8d7a","Type":"ContainerStarted","Data":"0ddb300f0d8bb8a33f2f86b26fafdcc1883d4678b57b6da7c0c4f5f8ccc280d0"} Jul 10 19:00:05.706750 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:05.706714 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sdt64" event={"ID":"06a74dc2-5187-47af-9eeb-77f4ce8a8d7a","Type":"ContainerStarted","Data":"7661e4128bf55186325e3a740eab63183a71adfb9e8a7ace662fad1fa1abf883"} Jul 10 19:00:06.325617 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:06.325578 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79774fc478-r6gn6"] Jul 10 19:00:06.327737 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:06.327715 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79774fc478-r6gn6" Jul 10 19:00:06.331593 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:06.331571 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 10 19:00:06.333266 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:06.333239 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 10 19:00:06.333401 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:06.333273 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 10 19:00:06.333401 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:06.333281 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 10 19:00:06.333401 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:06.333355 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-8rqkd\"" Jul 10 19:00:06.349150 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:06.349122 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79774fc478-r6gn6"] Jul 10 19:00:06.402094 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:06.402047 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9fa0bfb6-c7a2-4bc5-a736-193a55a2c59c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-79774fc478-r6gn6\" (UID: \"9fa0bfb6-c7a2-4bc5-a736-193a55a2c59c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79774fc478-r6gn6" Jul 10 19:00:06.402283 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:06.402200 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzlwt\" (UniqueName: \"kubernetes.io/projected/9fa0bfb6-c7a2-4bc5-a736-193a55a2c59c-kube-api-access-kzlwt\") pod \"managed-serviceaccount-addon-agent-79774fc478-r6gn6\" (UID: \"9fa0bfb6-c7a2-4bc5-a736-193a55a2c59c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79774fc478-r6gn6" Jul 10 19:00:06.420981 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:06.420946 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-bjg48"] Jul 10 19:00:06.425467 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:06.423822 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-bjg48" Jul 10 19:00:06.427798 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:06.427586 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-8r8gn\"" Jul 10 19:00:06.428174 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:06.428072 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 10 19:00:06.428174 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:06.428162 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 10 19:00:06.439733 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:06.439652 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:06.440290 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:06.440269 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-bjg48"] Jul 10 19:00:06.503379 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:06.503339 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzlwt\" (UniqueName: \"kubernetes.io/projected/9fa0bfb6-c7a2-4bc5-a736-193a55a2c59c-kube-api-access-kzlwt\") pod \"managed-serviceaccount-addon-agent-79774fc478-r6gn6\" (UID: \"9fa0bfb6-c7a2-4bc5-a736-193a55a2c59c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79774fc478-r6gn6" Jul 10 19:00:06.503586 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:06.503409 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34f5d206-d3be-47c7-a271-7486c9391651-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-bjg48\" (UID: \"34f5d206-d3be-47c7-a271-7486c9391651\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-bjg48" Jul 10 19:00:06.503586 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:06.503487 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9fa0bfb6-c7a2-4bc5-a736-193a55a2c59c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-79774fc478-r6gn6\" (UID: \"9fa0bfb6-c7a2-4bc5-a736-193a55a2c59c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79774fc478-r6gn6" Jul 10 19:00:06.503586 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:06.503555 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/34f5d206-d3be-47c7-a271-7486c9391651-nginx-conf\") pod \"networking-console-plugin-6cf856758c-bjg48\" (UID: \"34f5d206-d3be-47c7-a271-7486c9391651\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-bjg48" Jul 10 19:00:06.506383 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:06.506356 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9fa0bfb6-c7a2-4bc5-a736-193a55a2c59c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-79774fc478-r6gn6\" (UID: \"9fa0bfb6-c7a2-4bc5-a736-193a55a2c59c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79774fc478-r6gn6" Jul 10 19:00:06.517584 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:06.517556 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzlwt\" (UniqueName: \"kubernetes.io/projected/9fa0bfb6-c7a2-4bc5-a736-193a55a2c59c-kube-api-access-kzlwt\") pod \"managed-serviceaccount-addon-agent-79774fc478-r6gn6\" (UID: \"9fa0bfb6-c7a2-4bc5-a736-193a55a2c59c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79774fc478-r6gn6" Jul 10 19:00:06.532644 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:06.532611 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:06.604780 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:06.604698 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/34f5d206-d3be-47c7-a271-7486c9391651-nginx-conf\") pod \"networking-console-plugin-6cf856758c-bjg48\" (UID: \"34f5d206-d3be-47c7-a271-7486c9391651\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-bjg48" Jul 10 19:00:06.604932 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:06.604834 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34f5d206-d3be-47c7-a271-7486c9391651-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-bjg48\" (UID: \"34f5d206-d3be-47c7-a271-7486c9391651\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-bjg48" Jul 10 19:00:06.605457 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:06.605410 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/34f5d206-d3be-47c7-a271-7486c9391651-nginx-conf\") pod \"networking-console-plugin-6cf856758c-bjg48\" (UID: \"34f5d206-d3be-47c7-a271-7486c9391651\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-bjg48" Jul 10 19:00:06.607503 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:06.607477 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34f5d206-d3be-47c7-a271-7486c9391651-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-bjg48\" (UID: \"34f5d206-d3be-47c7-a271-7486c9391651\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-bjg48" Jul 10 19:00:06.644998 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:06.644962 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79774fc478-r6gn6" Jul 10 19:00:06.647728 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:06.647531 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:06.735905 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:06.735882 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-bjg48" Jul 10 19:00:06.791639 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:06.791424 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79774fc478-r6gn6"] Jul 10 19:00:06.794502 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:00:06.794469 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9fa0bfb6_c7a2_4bc5_a736_193a55a2c59c.slice/crio-50c6810f1fd6d84e2a50551ad02f42c487e96001b4e1aafe7585b32a46909985 WatchSource:0}: Error finding container 50c6810f1fd6d84e2a50551ad02f42c487e96001b4e1aafe7585b32a46909985: Status 404 returned error can't find the container with id 50c6810f1fd6d84e2a50551ad02f42c487e96001b4e1aafe7585b32a46909985 Jul 10 19:00:06.875893 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:06.875818 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-bjg48"] Jul 10 19:00:07.716212 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:07.716171 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79774fc478-r6gn6" event={"ID":"9fa0bfb6-c7a2-4bc5-a736-193a55a2c59c","Type":"ContainerStarted","Data":"50c6810f1fd6d84e2a50551ad02f42c487e96001b4e1aafe7585b32a46909985"} Jul 10 19:00:07.932871 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:00:07.932832 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34f5d206_d3be_47c7_a271_7486c9391651.slice/crio-df291838e6bf8a2b85fd2ff71c8fae687fecacf2d82ff9738f8d7e68e50a8999 WatchSource:0}: Error finding container df291838e6bf8a2b85fd2ff71c8fae687fecacf2d82ff9738f8d7e68e50a8999: Status 404 returned error can't find the container with id df291838e6bf8a2b85fd2ff71c8fae687fecacf2d82ff9738f8d7e68e50a8999 Jul 10 19:00:08.722689 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:08.722660 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sdt64" event={"ID":"06a74dc2-5187-47af-9eeb-77f4ce8a8d7a","Type":"ContainerStarted","Data":"cd25c2930c51160486427d9fb585b0dde20852fcca68239d09aa6ecb5b05bc39"} Jul 10 19:00:08.724236 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:08.724205 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-bjg48" event={"ID":"34f5d206-d3be-47c7-a271-7486c9391651","Type":"ContainerStarted","Data":"df291838e6bf8a2b85fd2ff71c8fae687fecacf2d82ff9738f8d7e68e50a8999"} Jul 10 19:00:08.750346 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:08.750251 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-sdt64" podStartSLOduration=17.040745144 podStartE2EDuration="24.750221s" totalImagesPullingTime="7.709475856s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:59:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:08.748908853 +0000 UTC m=+100.079049690" watchObservedRunningTime="2026-07-10 19:00:08.750221 +0000 UTC m=+100.080361806" Jul 10 19:00:09.831331 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:09.831287 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-registry-tls\") pod \"image-registry-84bdbb94b5-gnz5b\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 19:00:09.831848 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:09.831697 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9ea737c-ac9d-4694-bd6b-980514db5371-metrics-tls\") pod \"dns-default-d7l7x\" (UID: \"e9ea737c-ac9d-4694-bd6b-980514db5371\") " pod="openshift-dns/dns-default-d7l7x" Jul 10 19:00:09.834091 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:09.834062 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9ea737c-ac9d-4694-bd6b-980514db5371-metrics-tls\") pod \"dns-default-d7l7x\" (UID: \"e9ea737c-ac9d-4694-bd6b-980514db5371\") " pod="openshift-dns/dns-default-d7l7x" Jul 10 19:00:09.834324 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:09.834304 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-registry-tls\") pod \"image-registry-84bdbb94b5-gnz5b\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 19:00:09.892574 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:09.892531 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-d494m\"" Jul 10 19:00:09.899281 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:09.899252 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 19:00:09.902960 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:09.902404 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-s6z54\"" Jul 10 19:00:09.909724 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:09.909700 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d7l7x" Jul 10 19:00:10.201284 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:10.201254 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d7l7x"] Jul 10 19:00:10.204815 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:00:10.204790 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ea737c_ac9d_4694_bd6b_980514db5371.slice/crio-efacc9cb4d2251d8d549832b0285dc150c7ebf87487e2272222d909b599ab562 WatchSource:0}: Error finding container efacc9cb4d2251d8d549832b0285dc150c7ebf87487e2272222d909b599ab562: Status 404 returned error can't find the container with id efacc9cb4d2251d8d549832b0285dc150c7ebf87487e2272222d909b599ab562 Jul 10 19:00:10.216254 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:10.216230 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84bdbb94b5-gnz5b"] Jul 10 19:00:10.219528 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:00:10.219503 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c51490_aab2_4fd0_9ffc_06886c5515b3.slice/crio-bfc5dc6b9b590013862e5ad728f6b25530f52b9e70d90ac2cb0e3401ce9c44c3 WatchSource:0}: Error finding container bfc5dc6b9b590013862e5ad728f6b25530f52b9e70d90ac2cb0e3401ce9c44c3: Status 404 returned error can't find the container with id bfc5dc6b9b590013862e5ad728f6b25530f52b9e70d90ac2cb0e3401ce9c44c3 Jul 10 19:00:10.732761 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:10.732676 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d7l7x" event={"ID":"e9ea737c-ac9d-4694-bd6b-980514db5371","Type":"ContainerStarted","Data":"efacc9cb4d2251d8d549832b0285dc150c7ebf87487e2272222d909b599ab562"} Jul 10 19:00:10.734434 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:10.734403 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79774fc478-r6gn6" event={"ID":"9fa0bfb6-c7a2-4bc5-a736-193a55a2c59c","Type":"ContainerStarted","Data":"4c83a05d5d25a7d47fe98fdc8a4b868cc934364a7ed7d506eca27da7722a2606"} Jul 10 19:00:10.736251 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:10.736205 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-bjg48" event={"ID":"34f5d206-d3be-47c7-a271-7486c9391651","Type":"ContainerStarted","Data":"776bd62cefb8f030fa447ffded49394960885aa56b139b94de5d890387e13b52"} Jul 10 19:00:10.737851 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:10.737828 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" event={"ID":"24c51490-aab2-4fd0-9ffc-06886c5515b3","Type":"ContainerStarted","Data":"71707cd36588bf55f374385c864f58e1e7d756c48545133fbfd9611f0261e0fd"} Jul 10 19:00:10.737964 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:10.737872 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" event={"ID":"24c51490-aab2-4fd0-9ffc-06886c5515b3","Type":"ContainerStarted","Data":"bfc5dc6b9b590013862e5ad728f6b25530f52b9e70d90ac2cb0e3401ce9c44c3"} Jul 10 19:00:10.737964 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:10.737898 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 19:00:10.770572 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:10.770501 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79774fc478-r6gn6" podStartSLOduration=1.4736562690000001 podStartE2EDuration="4.770479432s" totalImagesPullingTime="3.296823163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:10.769330371 +0000 UTC m=+102.099471183" watchObservedRunningTime="2026-07-10 19:00:10.770479432 +0000 UTC m=+102.100620242" Jul 10 19:00:10.811654 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:10.811600 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-bjg48" podStartSLOduration=2.717863548 podStartE2EDuration="4.81158064s" totalImagesPullingTime="2.093717092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:10.810758112 +0000 UTC m=+102.140898924" watchObservedRunningTime="2026-07-10 19:00:10.81158064 +0000 UTC m=+102.141721452" Jul 10 19:00:10.837082 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:10.837032 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" podStartSLOduration=101.837015043 podStartE2EDuration="1m41.837015043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:58:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:10.835596296 +0000 UTC m=+102.165737133" watchObservedRunningTime="2026-07-10 19:00:10.837015043 +0000 UTC m=+102.167155853" Jul 10 19:00:10.938541 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:10.938498 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:11.013763 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:11.013679 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:11.618500 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:11.618469 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bl7wj" Jul 10 19:00:12.574650 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:12.574612 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:12.642969 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:12.642932 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:12.745774 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:12.745736 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d7l7x" event={"ID":"e9ea737c-ac9d-4694-bd6b-980514db5371","Type":"ContainerStarted","Data":"bb71a7fafd4e0e2da12adaf5a92318b3233ad28681d05cb48279ca748d16975c"} Jul 10 19:00:12.745955 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:12.745791 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d7l7x" event={"ID":"e9ea737c-ac9d-4694-bd6b-980514db5371","Type":"ContainerStarted","Data":"8dc2f372672e6d1639bd60e07914c4900b4ee9bbe25a3ca27b7b1ccdf5c8ad0f"} Jul 10 19:00:12.745955 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:12.745823 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-d7l7x" Jul 10 19:00:12.782786 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:12.782733 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-d7l7x" podStartSLOduration=66.046000303 podStartE2EDuration="1m7.782715554s" totalImagesPullingTime="1.736715251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:12.781640188 +0000 UTC m=+104.111781021" watchObservedRunningTime="2026-07-10 19:00:12.782715554 +0000 UTC m=+104.112856360" Jul 10 19:00:13.227487 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:13.227431 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:13.327675 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:13.327637 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:13.432566 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:13.432532 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-t2x4s"] Jul 10 19:00:13.454568 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:13.454538 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-t2x4s"] Jul 10 19:00:13.454738 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:13.454650 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-t2x4s" Jul 10 19:00:13.458802 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:13.458776 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-7ztlc\"" Jul 10 19:00:13.459350 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:13.459330 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 10 19:00:13.490240 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:13.490152 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:13.562707 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:13.562665 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0c3d7d0c-941d-43cc-9130-64169639313f-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-t2x4s\" (UID: \"0c3d7d0c-941d-43cc-9130-64169639313f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-t2x4s" Jul 10 19:00:13.664037 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:13.663993 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0c3d7d0c-941d-43cc-9130-64169639313f-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-t2x4s\" (UID: \"0c3d7d0c-941d-43cc-9130-64169639313f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-t2x4s" Jul 10 19:00:13.666343 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:13.666320 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0c3d7d0c-941d-43cc-9130-64169639313f-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-t2x4s\" (UID: \"0c3d7d0c-941d-43cc-9130-64169639313f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-t2x4s" Jul 10 19:00:13.669139 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:13.669107 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:13.763993 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:13.763892 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-t2x4s" Jul 10 19:00:13.924032 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:13.923988 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-t2x4s"] Jul 10 19:00:13.928115 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:00:13.928088 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c3d7d0c_941d_43cc_9130_64169639313f.slice/crio-bc48a2ff30a3663cc28dfa7c6bd3c6c8cc604b4f72efdeb686313da4f8eb8a3c WatchSource:0}: Error finding container bc48a2ff30a3663cc28dfa7c6bd3c6c8cc604b4f72efdeb686313da4f8eb8a3c: Status 404 returned error can't find the container with id bc48a2ff30a3663cc28dfa7c6bd3c6c8cc604b4f72efdeb686313da4f8eb8a3c Jul 10 19:00:14.754831 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:14.754791 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-t2x4s" event={"ID":"0c3d7d0c-941d-43cc-9130-64169639313f","Type":"ContainerStarted","Data":"bc48a2ff30a3663cc28dfa7c6bd3c6c8cc604b4f72efdeb686313da4f8eb8a3c"} Jul 10 19:00:15.759489 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:15.759437 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-t2x4s" event={"ID":"0c3d7d0c-941d-43cc-9130-64169639313f","Type":"ContainerStarted","Data":"b5cf28fdf2fd40b502c205e03b1f3db5fd7377a9feb71ab80c78fdd410627a49"} Jul 10 19:00:15.759888 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:15.759653 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-t2x4s" Jul 10 19:00:15.764314 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:15.764291 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-t2x4s" Jul 10 19:00:15.783884 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:15.783804 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-t2x4s" podStartSLOduration=1.184500888 podStartE2EDuration="2.783791383s" totalImagesPullingTime="1.599290495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:15.782973187 +0000 UTC m=+107.113114000" watchObservedRunningTime="2026-07-10 19:00:15.783791383 +0000 UTC m=+107.113932192" Jul 10 19:00:16.525616 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:16.525584 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:16.629246 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:16.629211 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:16.981864 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:16.981830 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:19.843790 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:19.843753 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:21.695813 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:21.695778 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:21.879982 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:21.879953 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:21.987901 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:21.987825 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-zzjcj"] Jul 10 19:00:21.990020 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:21.990004 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zzjcj" Jul 10 19:00:21.993390 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:21.993362 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 19:00:21.993736 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:21.993710 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 19:00:21.993855 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:21.993734 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 10 19:00:21.995000 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:21.994926 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 19:00:21.995000 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:21.994986 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 10 19:00:21.995160 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:21.995002 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-s4hsk\"" Jul 10 19:00:21.995480 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:21.995460 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:21.999509 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:21.999490 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-t5qwt"] Jul 10 19:00:22.001378 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.001357 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-t5qwt" Jul 10 19:00:22.004512 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.004485 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 10 19:00:22.004935 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.004910 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 10 19:00:22.011885 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.011865 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-9qqzd\"" Jul 10 19:00:22.012660 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.012634 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 10 19:00:22.014214 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.014193 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-zzjcj"] Jul 10 19:00:22.015366 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.015345 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8ldbp"] Jul 10 19:00:22.017884 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.017864 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8ldbp" Jul 10 19:00:22.020815 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.020795 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 19:00:22.020945 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.020924 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gq9d6\"" Jul 10 19:00:22.021048 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.021002 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 19:00:22.021202 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.021187 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 19:00:22.031106 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.031080 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-t5qwt"] Jul 10 19:00:22.077914 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.077880 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:22.135209 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.135167 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3b650cfc-a576-4008-afe7-36d8d34462f6-node-exporter-accelerators-collector-config\") pod \"node-exporter-8ldbp\" (UID: \"3b650cfc-a576-4008-afe7-36d8d34462f6\") " pod="openshift-monitoring/node-exporter-8ldbp" Jul 10 19:00:22.135390 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.135214 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/144f82fa-744c-458e-86a7-7d16ebebce5d-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-t5qwt\" (UID: \"144f82fa-744c-458e-86a7-7d16ebebce5d\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-t5qwt" Jul 10 19:00:22.135390 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.135257 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3b650cfc-a576-4008-afe7-36d8d34462f6-node-exporter-textfile\") pod \"node-exporter-8ldbp\" (UID: \"3b650cfc-a576-4008-afe7-36d8d34462f6\") " pod="openshift-monitoring/node-exporter-8ldbp" Jul 10 19:00:22.135390 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.135282 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n5d2\" (UniqueName: \"kubernetes.io/projected/144f82fa-744c-458e-86a7-7d16ebebce5d-kube-api-access-2n5d2\") pod \"kube-state-metrics-785778ff74-t5qwt\" (UID: \"144f82fa-744c-458e-86a7-7d16ebebce5d\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-t5qwt" Jul 10 19:00:22.135390 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.135309 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b650cfc-a576-4008-afe7-36d8d34462f6-sys\") pod \"node-exporter-8ldbp\" (UID: \"3b650cfc-a576-4008-afe7-36d8d34462f6\") " pod="openshift-monitoring/node-exporter-8ldbp" Jul 10 19:00:22.135390 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.135372 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63ee3b4a-66e3-4fe3-88a8-329a848e3faf-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-zzjcj\" (UID: \"63ee3b4a-66e3-4fe3-88a8-329a848e3faf\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zzjcj" Jul 10 19:00:22.135653 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.135411 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/144f82fa-744c-458e-86a7-7d16ebebce5d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-t5qwt\" (UID: \"144f82fa-744c-458e-86a7-7d16ebebce5d\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-t5qwt" Jul 10 19:00:22.135653 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.135435 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b650cfc-a576-4008-afe7-36d8d34462f6-metrics-client-ca\") pod \"node-exporter-8ldbp\" (UID: \"3b650cfc-a576-4008-afe7-36d8d34462f6\") " pod="openshift-monitoring/node-exporter-8ldbp" Jul 10 19:00:22.135653 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.135483 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq7zb\" (UniqueName: \"kubernetes.io/projected/63ee3b4a-66e3-4fe3-88a8-329a848e3faf-kube-api-access-zq7zb\") pod \"openshift-state-metrics-84bd68b896-zzjcj\" (UID: \"63ee3b4a-66e3-4fe3-88a8-329a848e3faf\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zzjcj" Jul 10 19:00:22.135653 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.135507 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/144f82fa-744c-458e-86a7-7d16ebebce5d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-t5qwt\" (UID: \"144f82fa-744c-458e-86a7-7d16ebebce5d\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-t5qwt" Jul 10 19:00:22.135653 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.135526 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/63ee3b4a-66e3-4fe3-88a8-329a848e3faf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-zzjcj\" (UID: \"63ee3b4a-66e3-4fe3-88a8-329a848e3faf\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zzjcj" Jul 10 19:00:22.135653 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.135585 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3b650cfc-a576-4008-afe7-36d8d34462f6-root\") pod \"node-exporter-8ldbp\" (UID: \"3b650cfc-a576-4008-afe7-36d8d34462f6\") " pod="openshift-monitoring/node-exporter-8ldbp" Jul 10 19:00:22.135653 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.135601 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3b650cfc-a576-4008-afe7-36d8d34462f6-node-exporter-tls\") pod \"node-exporter-8ldbp\" (UID: \"3b650cfc-a576-4008-afe7-36d8d34462f6\") " pod="openshift-monitoring/node-exporter-8ldbp" Jul 10 19:00:22.135653 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.135617 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/144f82fa-744c-458e-86a7-7d16ebebce5d-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-t5qwt\" (UID: \"144f82fa-744c-458e-86a7-7d16ebebce5d\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-t5qwt" Jul 10 19:00:22.135653 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.135648 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b650cfc-a576-4008-afe7-36d8d34462f6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8ldbp\" (UID: \"3b650cfc-a576-4008-afe7-36d8d34462f6\") " pod="openshift-monitoring/node-exporter-8ldbp" Jul 10 19:00:22.135968 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.135686 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj6jp\" (UniqueName: \"kubernetes.io/projected/3b650cfc-a576-4008-afe7-36d8d34462f6-kube-api-access-wj6jp\") pod \"node-exporter-8ldbp\" (UID: \"3b650cfc-a576-4008-afe7-36d8d34462f6\") " pod="openshift-monitoring/node-exporter-8ldbp" Jul 10 19:00:22.135968 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.135721 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/144f82fa-744c-458e-86a7-7d16ebebce5d-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-t5qwt\" (UID: \"144f82fa-744c-458e-86a7-7d16ebebce5d\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-t5qwt" Jul 10 19:00:22.135968 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.135756 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63ee3b4a-66e3-4fe3-88a8-329a848e3faf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-zzjcj\" (UID: \"63ee3b4a-66e3-4fe3-88a8-329a848e3faf\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zzjcj" Jul 10 19:00:22.135968 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.135780 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3b650cfc-a576-4008-afe7-36d8d34462f6-node-exporter-wtmp\") pod \"node-exporter-8ldbp\" (UID: \"3b650cfc-a576-4008-afe7-36d8d34462f6\") " pod="openshift-monitoring/node-exporter-8ldbp" Jul 10 19:00:22.190752 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.190721 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:22.237006 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.236971 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3b650cfc-a576-4008-afe7-36d8d34462f6-node-exporter-accelerators-collector-config\") pod \"node-exporter-8ldbp\" (UID: \"3b650cfc-a576-4008-afe7-36d8d34462f6\") " pod="openshift-monitoring/node-exporter-8ldbp" Jul 10 19:00:22.237006 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.237004 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/144f82fa-744c-458e-86a7-7d16ebebce5d-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-t5qwt\" (UID: \"144f82fa-744c-458e-86a7-7d16ebebce5d\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-t5qwt" Jul 10 19:00:22.237253 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.237024 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3b650cfc-a576-4008-afe7-36d8d34462f6-node-exporter-textfile\") pod \"node-exporter-8ldbp\" (UID: \"3b650cfc-a576-4008-afe7-36d8d34462f6\") " pod="openshift-monitoring/node-exporter-8ldbp" Jul 10 19:00:22.237253 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.237041 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n5d2\" (UniqueName: \"kubernetes.io/projected/144f82fa-744c-458e-86a7-7d16ebebce5d-kube-api-access-2n5d2\") pod \"kube-state-metrics-785778ff74-t5qwt\" (UID: \"144f82fa-744c-458e-86a7-7d16ebebce5d\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-t5qwt" Jul 10 19:00:22.237253 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.237059 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b650cfc-a576-4008-afe7-36d8d34462f6-sys\") pod \"node-exporter-8ldbp\" (UID: \"3b650cfc-a576-4008-afe7-36d8d34462f6\") " pod="openshift-monitoring/node-exporter-8ldbp" Jul 10 19:00:22.237253 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.237077 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63ee3b4a-66e3-4fe3-88a8-329a848e3faf-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-zzjcj\" (UID: \"63ee3b4a-66e3-4fe3-88a8-329a848e3faf\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zzjcj" Jul 10 19:00:22.237253 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.237100 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/144f82fa-744c-458e-86a7-7d16ebebce5d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-t5qwt\" (UID: \"144f82fa-744c-458e-86a7-7d16ebebce5d\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-t5qwt" Jul 10 19:00:22.237253 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.237128 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b650cfc-a576-4008-afe7-36d8d34462f6-metrics-client-ca\") pod \"node-exporter-8ldbp\" (UID: \"3b650cfc-a576-4008-afe7-36d8d34462f6\") " pod="openshift-monitoring/node-exporter-8ldbp" Jul 10 19:00:22.237253 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.237140 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b650cfc-a576-4008-afe7-36d8d34462f6-sys\") pod \"node-exporter-8ldbp\" (UID: \"3b650cfc-a576-4008-afe7-36d8d34462f6\") " pod="openshift-monitoring/node-exporter-8ldbp" Jul 10 19:00:22.237253 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.237151 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq7zb\" (UniqueName: \"kubernetes.io/projected/63ee3b4a-66e3-4fe3-88a8-329a848e3faf-kube-api-access-zq7zb\") pod \"openshift-state-metrics-84bd68b896-zzjcj\" (UID: \"63ee3b4a-66e3-4fe3-88a8-329a848e3faf\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zzjcj" Jul 10 19:00:22.237253 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.237178 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/144f82fa-744c-458e-86a7-7d16ebebce5d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-t5qwt\" (UID: \"144f82fa-744c-458e-86a7-7d16ebebce5d\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-t5qwt" Jul 10 19:00:22.237253 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.237207 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/63ee3b4a-66e3-4fe3-88a8-329a848e3faf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-zzjcj\" (UID: \"63ee3b4a-66e3-4fe3-88a8-329a848e3faf\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zzjcj" Jul 10 19:00:22.237253 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.237255 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3b650cfc-a576-4008-afe7-36d8d34462f6-root\") pod \"node-exporter-8ldbp\" (UID: \"3b650cfc-a576-4008-afe7-36d8d34462f6\") " pod="openshift-monitoring/node-exporter-8ldbp" Jul 10 19:00:22.237800 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.237278 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3b650cfc-a576-4008-afe7-36d8d34462f6-node-exporter-tls\") pod \"node-exporter-8ldbp\" (UID: \"3b650cfc-a576-4008-afe7-36d8d34462f6\") " pod="openshift-monitoring/node-exporter-8ldbp" Jul 10 19:00:22.237800 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.237301 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/144f82fa-744c-458e-86a7-7d16ebebce5d-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-t5qwt\" (UID: \"144f82fa-744c-458e-86a7-7d16ebebce5d\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-t5qwt" Jul 10 19:00:22.237800 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.237329 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b650cfc-a576-4008-afe7-36d8d34462f6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8ldbp\" (UID: \"3b650cfc-a576-4008-afe7-36d8d34462f6\") " pod="openshift-monitoring/node-exporter-8ldbp" Jul 10 19:00:22.237800 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.237369 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj6jp\" (UniqueName: \"kubernetes.io/projected/3b650cfc-a576-4008-afe7-36d8d34462f6-kube-api-access-wj6jp\") pod \"node-exporter-8ldbp\" (UID: \"3b650cfc-a576-4008-afe7-36d8d34462f6\") " pod="openshift-monitoring/node-exporter-8ldbp" Jul 10 19:00:22.237800 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.237401 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/144f82fa-744c-458e-86a7-7d16ebebce5d-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-t5qwt\" (UID: \"144f82fa-744c-458e-86a7-7d16ebebce5d\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-t5qwt" Jul 10 19:00:22.237800 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.237420 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/144f82fa-744c-458e-86a7-7d16ebebce5d-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-t5qwt\" (UID: \"144f82fa-744c-458e-86a7-7d16ebebce5d\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-t5qwt" Jul 10 19:00:22.237800 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.237458 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63ee3b4a-66e3-4fe3-88a8-329a848e3faf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-zzjcj\" (UID: \"63ee3b4a-66e3-4fe3-88a8-329a848e3faf\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zzjcj" Jul 10 19:00:22.237800 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.237487 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3b650cfc-a576-4008-afe7-36d8d34462f6-node-exporter-wtmp\") pod \"node-exporter-8ldbp\" (UID: \"3b650cfc-a576-4008-afe7-36d8d34462f6\") " pod="openshift-monitoring/node-exporter-8ldbp" Jul 10 19:00:22.237800 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.237511 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3b650cfc-a576-4008-afe7-36d8d34462f6-root\") pod \"node-exporter-8ldbp\" (UID: \"3b650cfc-a576-4008-afe7-36d8d34462f6\") " pod="openshift-monitoring/node-exporter-8ldbp" Jul 10 19:00:22.237800 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.237673 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3b650cfc-a576-4008-afe7-36d8d34462f6-node-exporter-accelerators-collector-config\") pod \"node-exporter-8ldbp\" (UID: \"3b650cfc-a576-4008-afe7-36d8d34462f6\") " pod="openshift-monitoring/node-exporter-8ldbp" Jul 10 19:00:22.238323 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.238053 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63ee3b4a-66e3-4fe3-88a8-329a848e3faf-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-zzjcj\" (UID: \"63ee3b4a-66e3-4fe3-88a8-329a848e3faf\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zzjcj" Jul 10 19:00:22.238323 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.238131 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b650cfc-a576-4008-afe7-36d8d34462f6-metrics-client-ca\") pod \"node-exporter-8ldbp\" (UID: \"3b650cfc-a576-4008-afe7-36d8d34462f6\") " pod="openshift-monitoring/node-exporter-8ldbp" Jul 10 19:00:22.238323 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.238148 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/144f82fa-744c-458e-86a7-7d16ebebce5d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-t5qwt\" (UID: \"144f82fa-744c-458e-86a7-7d16ebebce5d\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-t5qwt" Jul 10 19:00:22.238323 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.237486 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3b650cfc-a576-4008-afe7-36d8d34462f6-node-exporter-textfile\") pod \"node-exporter-8ldbp\" (UID: \"3b650cfc-a576-4008-afe7-36d8d34462f6\") " pod="openshift-monitoring/node-exporter-8ldbp" Jul 10 19:00:22.238560 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.238327 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3b650cfc-a576-4008-afe7-36d8d34462f6-node-exporter-wtmp\") pod \"node-exporter-8ldbp\" (UID: \"3b650cfc-a576-4008-afe7-36d8d34462f6\") " pod="openshift-monitoring/node-exporter-8ldbp" Jul 10 19:00:22.238811 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.238768 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/144f82fa-744c-458e-86a7-7d16ebebce5d-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-t5qwt\" (UID: \"144f82fa-744c-458e-86a7-7d16ebebce5d\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-t5qwt" Jul 10 19:00:22.240263 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.240216 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/144f82fa-744c-458e-86a7-7d16ebebce5d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-t5qwt\" (UID: \"144f82fa-744c-458e-86a7-7d16ebebce5d\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-t5qwt" Jul 10 19:00:22.240585 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.240564 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3b650cfc-a576-4008-afe7-36d8d34462f6-node-exporter-tls\") pod \"node-exporter-8ldbp\" (UID: \"3b650cfc-a576-4008-afe7-36d8d34462f6\") " pod="openshift-monitoring/node-exporter-8ldbp" Jul 10 19:00:22.240814 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.240791 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/63ee3b4a-66e3-4fe3-88a8-329a848e3faf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-zzjcj\" (UID: \"63ee3b4a-66e3-4fe3-88a8-329a848e3faf\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zzjcj" Jul 10 19:00:22.240865 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.240837 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/144f82fa-744c-458e-86a7-7d16ebebce5d-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-t5qwt\" (UID: \"144f82fa-744c-458e-86a7-7d16ebebce5d\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-t5qwt" Jul 10 19:00:22.240908 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.240861 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b650cfc-a576-4008-afe7-36d8d34462f6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8ldbp\" (UID: \"3b650cfc-a576-4008-afe7-36d8d34462f6\") " pod="openshift-monitoring/node-exporter-8ldbp" Jul 10 19:00:22.241827 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.241805 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63ee3b4a-66e3-4fe3-88a8-329a848e3faf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-zzjcj\" (UID: \"63ee3b4a-66e3-4fe3-88a8-329a848e3faf\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zzjcj" Jul 10 19:00:22.246353 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.246330 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq7zb\" (UniqueName: \"kubernetes.io/projected/63ee3b4a-66e3-4fe3-88a8-329a848e3faf-kube-api-access-zq7zb\") pod \"openshift-state-metrics-84bd68b896-zzjcj\" (UID: \"63ee3b4a-66e3-4fe3-88a8-329a848e3faf\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zzjcj" Jul 10 19:00:22.246646 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.246625 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n5d2\" (UniqueName: \"kubernetes.io/projected/144f82fa-744c-458e-86a7-7d16ebebce5d-kube-api-access-2n5d2\") pod \"kube-state-metrics-785778ff74-t5qwt\" (UID: \"144f82fa-744c-458e-86a7-7d16ebebce5d\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-t5qwt" Jul 10 19:00:22.247092 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.247073 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj6jp\" (UniqueName: \"kubernetes.io/projected/3b650cfc-a576-4008-afe7-36d8d34462f6-kube-api-access-wj6jp\") pod \"node-exporter-8ldbp\" (UID: \"3b650cfc-a576-4008-afe7-36d8d34462f6\") " pod="openshift-monitoring/node-exporter-8ldbp" Jul 10 19:00:22.290923 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.290886 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:22.301850 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.301822 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zzjcj" Jul 10 19:00:22.315628 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.315599 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-t5qwt" Jul 10 19:00:22.327799 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.327771 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8ldbp" Jul 10 19:00:22.337507 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:00:22.337482 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b650cfc_a576_4008_afe7_36d8d34462f6.slice/crio-a42d1cde530cafbbb41576885846c0f93f7f1701e940e0339959a9a9267a1b73 WatchSource:0}: Error finding container a42d1cde530cafbbb41576885846c0f93f7f1701e940e0339959a9a9267a1b73: Status 404 returned error can't find the container with id a42d1cde530cafbbb41576885846c0f93f7f1701e940e0339959a9a9267a1b73 Jul 10 19:00:22.399024 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.398974 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:22.442980 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.442958 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-zzjcj"] Jul 10 19:00:22.445134 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:00:22.445105 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ee3b4a_66e3_4fe3_88a8_329a848e3faf.slice/crio-2b4d2823e391e22245bdb9bd216da948fad674bf7d285a2be2b5385c91baea80 WatchSource:0}: Error finding container 2b4d2823e391e22245bdb9bd216da948fad674bf7d285a2be2b5385c91baea80: Status 404 returned error can't find the container with id 2b4d2823e391e22245bdb9bd216da948fad674bf7d285a2be2b5385c91baea80 Jul 10 19:00:22.463872 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.463850 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-t5qwt"] Jul 10 19:00:22.469392 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:00:22.469370 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144f82fa_744c_458e_86a7_7d16ebebce5d.slice/crio-7c344170b3a34bb5852a720a3649009fe7821b8f3a76f746ff728624fe47bf77 WatchSource:0}: Error finding container 7c344170b3a34bb5852a720a3649009fe7821b8f3a76f746ff728624fe47bf77: Status 404 returned error can't find the container with id 7c344170b3a34bb5852a720a3649009fe7821b8f3a76f746ff728624fe47bf77 Jul 10 19:00:22.751971 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.751897 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-d7l7x" Jul 10 19:00:22.785214 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.785179 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8ldbp" event={"ID":"3b650cfc-a576-4008-afe7-36d8d34462f6","Type":"ContainerStarted","Data":"a42d1cde530cafbbb41576885846c0f93f7f1701e940e0339959a9a9267a1b73"} Jul 10 19:00:22.786416 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.786386 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-t5qwt" event={"ID":"144f82fa-744c-458e-86a7-7d16ebebce5d","Type":"ContainerStarted","Data":"7c344170b3a34bb5852a720a3649009fe7821b8f3a76f746ff728624fe47bf77"} Jul 10 19:00:22.788215 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.788185 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zzjcj" event={"ID":"63ee3b4a-66e3-4fe3-88a8-329a848e3faf","Type":"ContainerStarted","Data":"22fba6472452fe7554225c52869a3eea654fd6ca0440dca7bfb268315b3da14c"} Jul 10 19:00:22.788293 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.788244 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zzjcj" event={"ID":"63ee3b4a-66e3-4fe3-88a8-329a848e3faf","Type":"ContainerStarted","Data":"dd8ef91665ea8170d62e2591d8ce960c71e1a82a24d419a77de68050df42c4b3"} Jul 10 19:00:22.788293 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.788272 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zzjcj" event={"ID":"63ee3b4a-66e3-4fe3-88a8-329a848e3faf","Type":"ContainerStarted","Data":"2b4d2823e391e22245bdb9bd216da948fad674bf7d285a2be2b5385c91baea80"} Jul 10 19:00:22.981243 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:22.981206 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:23.077056 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.076964 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:23.092689 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.092657 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 19:00:23.095664 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.095637 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.098320 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.098292 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 10 19:00:23.098588 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.098567 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 10 19:00:23.099242 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.098847 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-flsnz\"" Jul 10 19:00:23.099242 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.098917 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 10 19:00:23.099242 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.098849 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 10 19:00:23.099242 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.099207 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 10 19:00:23.099512 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.099281 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 10 19:00:23.099628 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.099609 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 10 19:00:23.099697 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.099613 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 10 19:00:23.099697 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.099679 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 10 19:00:23.113351 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.113321 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 19:00:23.177199 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.177167 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:23.245406 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.245275 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-tls-assets\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.245406 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.245358 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfmt8\" (UniqueName: \"kubernetes.io/projected/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-kube-api-access-sfmt8\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.245406 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.245406 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.245705 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.245460 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-config-out\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.245705 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.245486 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.245705 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.245518 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.245705 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.245542 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.245705 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.245563 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-web-config\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.245705 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.245592 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.245705 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.245612 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.245705 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.245639 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.245705 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.245678 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.245705 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.245706 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-config-volume\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.347354 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.347264 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.347354 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.347320 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.347354 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.347352 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.347687 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.347390 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.347687 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.347417 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-config-volume\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.347687 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.347512 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-tls-assets\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.347687 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.347560 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfmt8\" (UniqueName: \"kubernetes.io/projected/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-kube-api-access-sfmt8\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.347687 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.347597 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.347687 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.347628 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-config-out\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.347687 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.347652 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.347687 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.347685 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.348090 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.347720 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.348090 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.347744 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-web-config\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.349122 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.348904 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.354494 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.354149 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-tls-assets\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.354494 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.354202 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-config-volume\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.356008 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.355116 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.356008 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.355598 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-config-out\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.356008 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.355835 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.356008 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.355934 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.356264 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.356161 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.358263 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.358220 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.359739 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.358890 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.359739 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.359302 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-web-config\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.359739 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.359691 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.360967 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.360944 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfmt8\" (UniqueName: \"kubernetes.io/projected/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-kube-api-access-sfmt8\") pod \"alertmanager-main-0\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.406878 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.406818 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:00:23.600042 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.599950 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 19:00:23.756042 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:00:23.756011 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30bfaf4b_6dde_45ac_8a7a_d70a613a31da.slice/crio-57058119a8e0f39578dd7e20d8da06326b9f8d48f871c89d4315e6907f2aebab WatchSource:0}: Error finding container 57058119a8e0f39578dd7e20d8da06326b9f8d48f871c89d4315e6907f2aebab: Status 404 returned error can't find the container with id 57058119a8e0f39578dd7e20d8da06326b9f8d48f871c89d4315e6907f2aebab Jul 10 19:00:23.794217 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.794178 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8ldbp" event={"ID":"3b650cfc-a576-4008-afe7-36d8d34462f6","Type":"ContainerStarted","Data":"7f43303e2f276f3082a78fd9ea397c3e0844de85999c9159793866f9a7970af4"} Jul 10 19:00:23.795507 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:23.795475 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"30bfaf4b-6dde-45ac-8a7a-d70a613a31da","Type":"ContainerStarted","Data":"57058119a8e0f39578dd7e20d8da06326b9f8d48f871c89d4315e6907f2aebab"} Jul 10 19:00:24.801530 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:24.801494 2560 generic.go:356] "Generic (PLEG): container finished" podID="3b650cfc-a576-4008-afe7-36d8d34462f6" containerID="7f43303e2f276f3082a78fd9ea397c3e0844de85999c9159793866f9a7970af4" exitCode=0 Jul 10 19:00:24.801977 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:24.801553 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8ldbp" event={"ID":"3b650cfc-a576-4008-afe7-36d8d34462f6","Type":"ContainerDied","Data":"7f43303e2f276f3082a78fd9ea397c3e0844de85999c9159793866f9a7970af4"} Jul 10 19:00:24.804185 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:24.804157 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-t5qwt" event={"ID":"144f82fa-744c-458e-86a7-7d16ebebce5d","Type":"ContainerStarted","Data":"7a1656a279d78d131f6d237f9737ea7df1f99c9b9c92e965947e6f417f9245a3"} Jul 10 19:00:24.804293 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:24.804217 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-t5qwt" event={"ID":"144f82fa-744c-458e-86a7-7d16ebebce5d","Type":"ContainerStarted","Data":"6f9cab3f427d6848f2ca33e0e2961679d0088fce9d5d7629863680d78b1b6fd3"} Jul 10 19:00:24.804293 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:24.804249 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-t5qwt" event={"ID":"144f82fa-744c-458e-86a7-7d16ebebce5d","Type":"ContainerStarted","Data":"b9881d551219e9fe4b9c38617bffe632cb1353bda141bf109dedb5c480553d69"} Jul 10 19:00:24.806453 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:24.806415 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zzjcj" event={"ID":"63ee3b4a-66e3-4fe3-88a8-329a848e3faf","Type":"ContainerStarted","Data":"b39d770b055b28b12ffd41177d8d06c154a436cebb369b146d19254edef3fd68"} Jul 10 19:00:24.848328 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:24.848274 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-t5qwt" podStartSLOduration=2.233143016 podStartE2EDuration="3.848254072s" totalImagesPullingTime="1.615111056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:24.845720656 +0000 UTC m=+116.175861487" watchObservedRunningTime="2026-07-10 19:00:24.848254072 +0000 UTC m=+116.178394884" Jul 10 19:00:24.867547 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:24.867493 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zzjcj" podStartSLOduration=2.352354625 podStartE2EDuration="3.867471777s" totalImagesPullingTime="1.515117152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:24.865922235 +0000 UTC m=+116.196063082" watchObservedRunningTime="2026-07-10 19:00:24.867471777 +0000 UTC m=+116.197612588" Jul 10 19:00:24.958558 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:24.958519 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:25.041965 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:25.041912 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:25.100776 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:25.100691 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:25.379417 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:25.379328 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:25.672661 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:25.672630 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:25.766081 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:25.766038 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:25.812573 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:25.812537 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8ldbp" event={"ID":"3b650cfc-a576-4008-afe7-36d8d34462f6","Type":"ContainerStarted","Data":"c0f4822d396424de4ec0760710ec3a9961a636bcd1133ed22404411fe73f4a79"} Jul 10 19:00:25.813028 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:25.812624 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8ldbp" event={"ID":"3b650cfc-a576-4008-afe7-36d8d34462f6","Type":"ContainerStarted","Data":"70281b57b0d915914137efb6e25774b1ed5683fd52226acedd7f4f5674650a50"} Jul 10 19:00:25.814081 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:25.814057 2560 generic.go:356] "Generic (PLEG): container finished" podID="30bfaf4b-6dde-45ac-8a7a-d70a613a31da" containerID="93d4e2c95ff91f10be2463f2c04d76bd5e6852d056244f438941d027277542a9" exitCode=0 Jul 10 19:00:25.814186 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:25.814099 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"30bfaf4b-6dde-45ac-8a7a-d70a613a31da","Type":"ContainerDied","Data":"93d4e2c95ff91f10be2463f2c04d76bd5e6852d056244f438941d027277542a9"} Jul 10 19:00:25.823564 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:25.823542 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:25.837870 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:25.837830 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8ldbp" podStartSLOduration=3.638121789 podStartE2EDuration="4.837818942s" totalImagesPullingTime="1.118445861s" totalInitContainerRuntime="81.251292ms" isStatefulPod=true podCreationTimestamp="2026-07-10 19:00:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:25.835589916 +0000 UTC m=+117.165730747" watchObservedRunningTime="2026-07-10 19:00:25.837818942 +0000 UTC m=+117.167959752" Jul 10 19:00:26.011368 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.011281 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:26.137615 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.137580 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:26.194324 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.194290 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:26.321218 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.321138 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-546f8cd949-89c7k"] Jul 10 19:00:26.323610 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.323594 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-546f8cd949-89c7k" Jul 10 19:00:26.331749 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.331732 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5vf8pdfqvgk10\"" Jul 10 19:00:26.333116 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.333100 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 10 19:00:26.333193 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.333101 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 10 19:00:26.333193 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.333138 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-m74lg\"" Jul 10 19:00:26.333193 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.333152 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 10 19:00:26.334405 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.334392 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 10 19:00:26.341010 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.340955 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-546f8cd949-89c7k"] Jul 10 19:00:26.368697 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.368660 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:26.440308 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.440274 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:26.477121 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.477081 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/321cf2d7-c2b3-4b43-9c1a-ebb28da445d7-secret-metrics-server-client-certs\") pod \"metrics-server-546f8cd949-89c7k\" (UID: \"321cf2d7-c2b3-4b43-9c1a-ebb28da445d7\") " pod="openshift-monitoring/metrics-server-546f8cd949-89c7k" Jul 10 19:00:26.477121 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.477115 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/321cf2d7-c2b3-4b43-9c1a-ebb28da445d7-client-ca-bundle\") pod \"metrics-server-546f8cd949-89c7k\" (UID: \"321cf2d7-c2b3-4b43-9c1a-ebb28da445d7\") " pod="openshift-monitoring/metrics-server-546f8cd949-89c7k" Jul 10 19:00:26.477360 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.477214 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/321cf2d7-c2b3-4b43-9c1a-ebb28da445d7-secret-metrics-server-tls\") pod \"metrics-server-546f8cd949-89c7k\" (UID: \"321cf2d7-c2b3-4b43-9c1a-ebb28da445d7\") " pod="openshift-monitoring/metrics-server-546f8cd949-89c7k" Jul 10 19:00:26.477360 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.477253 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/321cf2d7-c2b3-4b43-9c1a-ebb28da445d7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-546f8cd949-89c7k\" (UID: \"321cf2d7-c2b3-4b43-9c1a-ebb28da445d7\") " pod="openshift-monitoring/metrics-server-546f8cd949-89c7k" Jul 10 19:00:26.477360 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.477288 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/321cf2d7-c2b3-4b43-9c1a-ebb28da445d7-audit-log\") pod \"metrics-server-546f8cd949-89c7k\" (UID: \"321cf2d7-c2b3-4b43-9c1a-ebb28da445d7\") " pod="openshift-monitoring/metrics-server-546f8cd949-89c7k" Jul 10 19:00:26.477360 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.477326 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52p9j\" (UniqueName: \"kubernetes.io/projected/321cf2d7-c2b3-4b43-9c1a-ebb28da445d7-kube-api-access-52p9j\") pod \"metrics-server-546f8cd949-89c7k\" (UID: \"321cf2d7-c2b3-4b43-9c1a-ebb28da445d7\") " pod="openshift-monitoring/metrics-server-546f8cd949-89c7k" Jul 10 19:00:26.477548 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.477364 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/321cf2d7-c2b3-4b43-9c1a-ebb28da445d7-metrics-server-audit-profiles\") pod \"metrics-server-546f8cd949-89c7k\" (UID: \"321cf2d7-c2b3-4b43-9c1a-ebb28da445d7\") " pod="openshift-monitoring/metrics-server-546f8cd949-89c7k" Jul 10 19:00:26.578575 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.578493 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/321cf2d7-c2b3-4b43-9c1a-ebb28da445d7-secret-metrics-server-tls\") pod \"metrics-server-546f8cd949-89c7k\" (UID: \"321cf2d7-c2b3-4b43-9c1a-ebb28da445d7\") " pod="openshift-monitoring/metrics-server-546f8cd949-89c7k" Jul 10 19:00:26.578575 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.578530 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/321cf2d7-c2b3-4b43-9c1a-ebb28da445d7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-546f8cd949-89c7k\" (UID: \"321cf2d7-c2b3-4b43-9c1a-ebb28da445d7\") " pod="openshift-monitoring/metrics-server-546f8cd949-89c7k" Jul 10 19:00:26.578575 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.578554 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/321cf2d7-c2b3-4b43-9c1a-ebb28da445d7-audit-log\") pod \"metrics-server-546f8cd949-89c7k\" (UID: \"321cf2d7-c2b3-4b43-9c1a-ebb28da445d7\") " pod="openshift-monitoring/metrics-server-546f8cd949-89c7k" Jul 10 19:00:26.578779 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.578737 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52p9j\" (UniqueName: \"kubernetes.io/projected/321cf2d7-c2b3-4b43-9c1a-ebb28da445d7-kube-api-access-52p9j\") pod \"metrics-server-546f8cd949-89c7k\" (UID: \"321cf2d7-c2b3-4b43-9c1a-ebb28da445d7\") " pod="openshift-monitoring/metrics-server-546f8cd949-89c7k" Jul 10 19:00:26.578813 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.578779 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/321cf2d7-c2b3-4b43-9c1a-ebb28da445d7-metrics-server-audit-profiles\") pod \"metrics-server-546f8cd949-89c7k\" (UID: \"321cf2d7-c2b3-4b43-9c1a-ebb28da445d7\") " pod="openshift-monitoring/metrics-server-546f8cd949-89c7k" Jul 10 19:00:26.578893 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.578876 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/321cf2d7-c2b3-4b43-9c1a-ebb28da445d7-secret-metrics-server-client-certs\") pod \"metrics-server-546f8cd949-89c7k\" (UID: \"321cf2d7-c2b3-4b43-9c1a-ebb28da445d7\") " pod="openshift-monitoring/metrics-server-546f8cd949-89c7k" Jul 10 19:00:26.578946 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.578909 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/321cf2d7-c2b3-4b43-9c1a-ebb28da445d7-client-ca-bundle\") pod \"metrics-server-546f8cd949-89c7k\" (UID: \"321cf2d7-c2b3-4b43-9c1a-ebb28da445d7\") " pod="openshift-monitoring/metrics-server-546f8cd949-89c7k" Jul 10 19:00:26.578946 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.578934 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/321cf2d7-c2b3-4b43-9c1a-ebb28da445d7-audit-log\") pod \"metrics-server-546f8cd949-89c7k\" (UID: \"321cf2d7-c2b3-4b43-9c1a-ebb28da445d7\") " pod="openshift-monitoring/metrics-server-546f8cd949-89c7k" Jul 10 19:00:26.579317 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.579288 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/321cf2d7-c2b3-4b43-9c1a-ebb28da445d7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-546f8cd949-89c7k\" (UID: \"321cf2d7-c2b3-4b43-9c1a-ebb28da445d7\") " pod="openshift-monitoring/metrics-server-546f8cd949-89c7k" Jul 10 19:00:26.579712 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.579680 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/321cf2d7-c2b3-4b43-9c1a-ebb28da445d7-metrics-server-audit-profiles\") pod \"metrics-server-546f8cd949-89c7k\" (UID: \"321cf2d7-c2b3-4b43-9c1a-ebb28da445d7\") " pod="openshift-monitoring/metrics-server-546f8cd949-89c7k" Jul 10 19:00:26.581084 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.581065 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/321cf2d7-c2b3-4b43-9c1a-ebb28da445d7-secret-metrics-server-tls\") pod \"metrics-server-546f8cd949-89c7k\" (UID: \"321cf2d7-c2b3-4b43-9c1a-ebb28da445d7\") " pod="openshift-monitoring/metrics-server-546f8cd949-89c7k" Jul 10 19:00:26.581346 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.581324 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/321cf2d7-c2b3-4b43-9c1a-ebb28da445d7-client-ca-bundle\") pod \"metrics-server-546f8cd949-89c7k\" (UID: \"321cf2d7-c2b3-4b43-9c1a-ebb28da445d7\") " pod="openshift-monitoring/metrics-server-546f8cd949-89c7k" Jul 10 19:00:26.581404 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.581324 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/321cf2d7-c2b3-4b43-9c1a-ebb28da445d7-secret-metrics-server-client-certs\") pod \"metrics-server-546f8cd949-89c7k\" (UID: \"321cf2d7-c2b3-4b43-9c1a-ebb28da445d7\") " pod="openshift-monitoring/metrics-server-546f8cd949-89c7k" Jul 10 19:00:26.587587 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.587563 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52p9j\" (UniqueName: \"kubernetes.io/projected/321cf2d7-c2b3-4b43-9c1a-ebb28da445d7-kube-api-access-52p9j\") pod \"metrics-server-546f8cd949-89c7k\" (UID: \"321cf2d7-c2b3-4b43-9c1a-ebb28da445d7\") " pod="openshift-monitoring/metrics-server-546f8cd949-89c7k" Jul 10 19:00:26.632541 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.632504 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-546f8cd949-89c7k" Jul 10 19:00:26.739513 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.739416 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:26.793879 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.793847 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-546f8cd949-89c7k"] Jul 10 19:00:26.797809 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:00:26.797778 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod321cf2d7_c2b3_4b43_9c1a_ebb28da445d7.slice/crio-7170e1ff836fe8bbb1aa4013ae8d3cc24e6a662fb582e2cb89f44a30136eebea WatchSource:0}: Error finding container 7170e1ff836fe8bbb1aa4013ae8d3cc24e6a662fb582e2cb89f44a30136eebea: Status 404 returned error can't find the container with id 7170e1ff836fe8bbb1aa4013ae8d3cc24e6a662fb582e2cb89f44a30136eebea Jul 10 19:00:26.800742 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.800620 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:26.819797 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.819766 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-546f8cd949-89c7k" event={"ID":"321cf2d7-c2b3-4b43-9c1a-ebb28da445d7","Type":"ContainerStarted","Data":"7170e1ff836fe8bbb1aa4013ae8d3cc24e6a662fb582e2cb89f44a30136eebea"} Jul 10 19:00:26.865198 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.865088 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:26.948741 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:26.948706 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:27.049911 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:27.049875 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:27.368038 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:27.368017 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:27.493029 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:27.492991 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:27.595496 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:27.595457 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:27.827014 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:27.826928 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"30bfaf4b-6dde-45ac-8a7a-d70a613a31da","Type":"ContainerStarted","Data":"2e5f5b4a6fed3d548182ac61c4cd78041fe3e3f962cab88ffef88acaca23c4e9"} Jul 10 19:00:27.827431 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:27.827031 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"30bfaf4b-6dde-45ac-8a7a-d70a613a31da","Type":"ContainerStarted","Data":"d81fa72fa7b84d428b0b788cba0d8a92926adfa6bc0a6fa6b58c845025707270"} Jul 10 19:00:27.827431 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:27.827060 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"30bfaf4b-6dde-45ac-8a7a-d70a613a31da","Type":"ContainerStarted","Data":"a75fca2eaa968e64c918946f6cb8f3911481b9157fac2739021e38b1a05ef6b8"} Jul 10 19:00:27.827431 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:27.827084 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"30bfaf4b-6dde-45ac-8a7a-d70a613a31da","Type":"ContainerStarted","Data":"d5c128940ead47570db005908b53eeea5c7058dbb80bc291bf0d8d12f71e851e"} Jul 10 19:00:27.827431 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:27.827107 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"30bfaf4b-6dde-45ac-8a7a-d70a613a31da","Type":"ContainerStarted","Data":"f0a30779bb0568bab045cbf93e73a05c68ff88a51a24d1c8e9ef78bbb8fd8e97"} Jul 10 19:00:27.971948 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:27.971913 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:28.072506 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.072426 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:28.311045 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.311012 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 19:00:28.314271 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.314248 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.317882 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.317853 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 10 19:00:28.318120 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.318103 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 10 19:00:28.318339 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.318324 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6bcp5ktqqsj24\"" Jul 10 19:00:28.318583 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.318567 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 10 19:00:28.319718 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.319647 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 10 19:00:28.319718 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.319660 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 10 19:00:28.319889 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.319726 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 10 19:00:28.319889 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.319775 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 10 19:00:28.319988 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.319952 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 10 19:00:28.320035 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.320023 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-pjfjz\"" Jul 10 19:00:28.320121 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.320102 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 10 19:00:28.320668 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.320651 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 10 19:00:28.321376 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.321325 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 10 19:00:28.323116 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.323009 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 10 19:00:28.325463 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.325419 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 10 19:00:28.327941 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.327920 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 10 19:00:28.344535 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.344503 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 19:00:28.396597 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.396567 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b26709a-e61c-4a9a-8d89-5557b8f999b9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.396597 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.396601 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.396778 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.396625 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.396778 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.396671 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.396778 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.396699 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.396778 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.396763 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.396931 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.396804 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.396931 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.396829 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st8nc\" (UniqueName: \"kubernetes.io/projected/9b26709a-e61c-4a9a-8d89-5557b8f999b9-kube-api-access-st8nc\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.396931 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.396852 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.396931 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.396883 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b26709a-e61c-4a9a-8d89-5557b8f999b9-config-out\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.396931 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.396899 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-web-config\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.397103 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.396946 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9b26709a-e61c-4a9a-8d89-5557b8f999b9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.397103 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.396967 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.397103 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.396983 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.397103 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.397007 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-config\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.397103 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.397021 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.397103 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.397039 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.397103 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.397053 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.397103 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.397089 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.397342 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.397109 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.399362 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.399342 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:28.497964 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.497926 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9b26709a-e61c-4a9a-8d89-5557b8f999b9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.498144 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.497981 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.498144 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.498010 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.498144 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.498038 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-config\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.498144 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.498061 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.498144 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.498086 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.498144 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.498108 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.498144 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.498139 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.498509 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.498163 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.498509 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.498238 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b26709a-e61c-4a9a-8d89-5557b8f999b9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.498509 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.498268 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.498509 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.498298 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.498509 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.498334 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.498509 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.498336 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9b26709a-e61c-4a9a-8d89-5557b8f999b9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.498509 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.498361 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.498509 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.498399 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.498509 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.498433 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.498509 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.498483 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st8nc\" (UniqueName: \"kubernetes.io/projected/9b26709a-e61c-4a9a-8d89-5557b8f999b9-kube-api-access-st8nc\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.498983 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.498513 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.498983 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.498553 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b26709a-e61c-4a9a-8d89-5557b8f999b9-config-out\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.498983 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.498577 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-web-config\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.499881 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.499855 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.500434 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.500407 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.501325 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.501040 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.502540 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.502518 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.502676 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.502600 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.502784 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.502762 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.503112 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.503090 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-web-config\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.503360 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.503339 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.503911 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.503892 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.504374 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.504350 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.504496 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.504475 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.504633 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.504583 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.504854 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.504830 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.505044 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.505023 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-config\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.505125 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.505049 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.505690 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.505659 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b26709a-e61c-4a9a-8d89-5557b8f999b9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.506179 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.506157 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b26709a-e61c-4a9a-8d89-5557b8f999b9-config-out\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.506300 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.506279 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.509434 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.509410 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st8nc\" (UniqueName: \"kubernetes.io/projected/9b26709a-e61c-4a9a-8d89-5557b8f999b9-kube-api-access-st8nc\") pod \"prometheus-k8s-0\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.626908 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.626888 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:28.772177 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.772152 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 19:00:28.774566 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:00:28.774543 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b26709a_e61c_4a9a_8d89_5557b8f999b9.slice/crio-24fcfee08fb9f56089a693b64f822ae717582eb3254690cd10126247f6d4dfd0 WatchSource:0}: Error finding container 24fcfee08fb9f56089a693b64f822ae717582eb3254690cd10126247f6d4dfd0: Status 404 returned error can't find the container with id 24fcfee08fb9f56089a693b64f822ae717582eb3254690cd10126247f6d4dfd0 Jul 10 19:00:28.834585 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.834549 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"30bfaf4b-6dde-45ac-8a7a-d70a613a31da","Type":"ContainerStarted","Data":"2f0c4ac99f6fe35ac80f707d3b453afdc794cd91c8ae62242b568ad14fb6ce86"} Jul 10 19:00:28.835898 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.835871 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b26709a-e61c-4a9a-8d89-5557b8f999b9","Type":"ContainerStarted","Data":"24fcfee08fb9f56089a693b64f822ae717582eb3254690cd10126247f6d4dfd0"} Jul 10 19:00:28.837423 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.837372 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-546f8cd949-89c7k" event={"ID":"321cf2d7-c2b3-4b43-9c1a-ebb28da445d7","Type":"ContainerStarted","Data":"56358b0102d087daf102688d1598b6a32926c66005bf6e81c8187834ad61fc3b"} Jul 10 19:00:28.864802 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.864740 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.349067213 podStartE2EDuration="5.864717218s" totalImagesPullingTime="3.494441012s" totalInitContainerRuntime="21.208993ms" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:28.861658127 +0000 UTC m=+120.191798962" watchObservedRunningTime="2026-07-10 19:00:28.864717218 +0000 UTC m=+120.194858028" Jul 10 19:00:28.881877 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:28.881819 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-546f8cd949-89c7k" podStartSLOduration=1.126343176 podStartE2EDuration="2.881795078s" totalImagesPullingTime="1.755451902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:28.880475634 +0000 UTC m=+120.210616462" watchObservedRunningTime="2026-07-10 19:00:28.881795078 +0000 UTC m=+120.211935889" Jul 10 19:00:29.096931 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:29.096857 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84bdbb94b5-gnz5b"] Jul 10 19:00:29.101329 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:29.101298 2560 patch_prober.go:28] interesting pod/image-registry-84bdbb94b5-gnz5b container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 10 19:00:29.101489 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:29.101349 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" podUID="24c51490-aab2-4fd0-9ffc-06886c5515b3" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 19:00:29.224256 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:29.224209 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:29.324395 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:29.324362 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:29.847309 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:29.847277 2560 generic.go:356] "Generic (PLEG): container finished" podID="9b26709a-e61c-4a9a-8d89-5557b8f999b9" containerID="5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23" exitCode=0 Jul 10 19:00:29.847821 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:29.847367 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b26709a-e61c-4a9a-8d89-5557b8f999b9","Type":"ContainerDied","Data":"5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23"} Jul 10 19:00:31.940925 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:31.940893 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:32.000690 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:32.000657 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:33.867537 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:33.867504 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b26709a-e61c-4a9a-8d89-5557b8f999b9","Type":"ContainerStarted","Data":"aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5"} Jul 10 19:00:33.867929 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:33.867573 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b26709a-e61c-4a9a-8d89-5557b8f999b9","Type":"ContainerStarted","Data":"fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c"} Jul 10 19:00:35.582207 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:35.582169 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:35.685829 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:35.685796 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:35.879134 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:35.879103 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b26709a-e61c-4a9a-8d89-5557b8f999b9","Type":"ContainerStarted","Data":"f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6"} Jul 10 19:00:35.879294 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:35.879165 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b26709a-e61c-4a9a-8d89-5557b8f999b9","Type":"ContainerStarted","Data":"6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163"} Jul 10 19:00:35.879294 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:35.879203 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b26709a-e61c-4a9a-8d89-5557b8f999b9","Type":"ContainerStarted","Data":"ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c"} Jul 10 19:00:35.879294 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:35.879229 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b26709a-e61c-4a9a-8d89-5557b8f999b9","Type":"ContainerStarted","Data":"470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92"} Jul 10 19:00:35.937327 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:35.937259 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.4525903 podStartE2EDuration="7.937239569s" totalImagesPullingTime="5.45844382s" totalInitContainerRuntime="26.205449ms" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:35.937148377 +0000 UTC m=+127.267289213" watchObservedRunningTime="2026-07-10 19:00:35.937239569 +0000 UTC m=+127.267380424" Jul 10 19:00:38.569252 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:38.569215 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:38.625035 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:38.625005 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:38.627932 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:38.627907 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:00:39.005888 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:39.005852 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1bf574f-a4e2-4418-8f18-95fa0c80ca1e-metrics-certs\") pod \"network-metrics-daemon-v9b95\" (UID: \"b1bf574f-a4e2-4418-8f18-95fa0c80ca1e\") " pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 19:00:39.008161 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:39.008141 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1bf574f-a4e2-4418-8f18-95fa0c80ca1e-metrics-certs\") pod \"network-metrics-daemon-v9b95\" (UID: \"b1bf574f-a4e2-4418-8f18-95fa0c80ca1e\") " pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 19:00:39.079625 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:39.079592 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mzc76\"" Jul 10 19:00:39.086793 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:39.086771 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9b95" Jul 10 19:00:39.102259 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:39.102234 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 19:00:39.216835 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:39.216739 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v9b95"] Jul 10 19:00:39.218799 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:00:39.218776 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1bf574f_a4e2_4418_8f18_95fa0c80ca1e.slice/crio-d96f2a7ee36fafed25518714a1f213fa998d24c6619654937ed6e8e1f0cca27d WatchSource:0}: Error finding container d96f2a7ee36fafed25518714a1f213fa998d24c6619654937ed6e8e1f0cca27d: Status 404 returned error can't find the container with id d96f2a7ee36fafed25518714a1f213fa998d24c6619654937ed6e8e1f0cca27d Jul 10 19:00:39.664128 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:39.664094 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:39.764430 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:39.764386 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:39.836025 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:39.835998 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:39.896071 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:39.896034 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v9b95" event={"ID":"b1bf574f-a4e2-4418-8f18-95fa0c80ca1e","Type":"ContainerStarted","Data":"d96f2a7ee36fafed25518714a1f213fa998d24c6619654937ed6e8e1f0cca27d"} Jul 10 19:00:40.901950 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:40.901911 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v9b95" event={"ID":"b1bf574f-a4e2-4418-8f18-95fa0c80ca1e","Type":"ContainerStarted","Data":"1c9fe25ae9f0ecfaecabad02f7354b3bb17c23b20b5dc509c97f7b4084c2627d"} Jul 10 19:00:40.902398 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:40.902003 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v9b95" event={"ID":"b1bf574f-a4e2-4418-8f18-95fa0c80ca1e","Type":"ContainerStarted","Data":"63545dc6ea76493ff8ba58eee864618dcc48b3deee00ae5e3cf7176e2867509d"} Jul 10 19:00:40.906729 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:40.906696 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:40.918832 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:40.918753 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-v9b95" podStartSLOduration=131.054637495 podStartE2EDuration="2m11.918734449s" totalImagesPullingTime="864.096954ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:58:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:40.918417413 +0000 UTC m=+132.248558246" watchObservedRunningTime="2026-07-10 19:00:40.918734449 +0000 UTC m=+132.248875259" Jul 10 19:00:41.008704 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:41.008665 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:41.068831 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:41.068797 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:42.196404 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:42.196371 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:42.255779 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:42.255753 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:42.315009 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:42.314977 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:42.387293 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:42.387257 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:42.556521 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:42.556422 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:42.623805 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:42.623776 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:43.341574 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:43.341542 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:43.443553 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:43.443521 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:46.633089 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:46.633048 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-546f8cd949-89c7k" Jul 10 19:00:46.633511 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:46.633135 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-546f8cd949-89c7k" Jul 10 19:00:47.220754 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:47.220719 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:47.320242 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:47.320210 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:48.367516 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:48.367482 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:48.472372 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:48.472339 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:48.750291 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:48.750262 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:48.847859 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:48.847828 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:48.935558 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:48.935524 2560 generic.go:356] "Generic (PLEG): container finished" podID="651e6091-4d1a-4adc-90bf-b0a7e5f8291c" containerID="9280ddd1fe48ff29ca376c36196460230792b1aee57c82b60f4313c7a42de828" exitCode=0 Jul 10 19:00:48.935730 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:48.935603 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dd8w8" event={"ID":"651e6091-4d1a-4adc-90bf-b0a7e5f8291c","Type":"ContainerDied","Data":"9280ddd1fe48ff29ca376c36196460230792b1aee57c82b60f4313c7a42de828"} Jul 10 19:00:48.935968 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:48.935953 2560 scope.go:117] "RemoveContainer" containerID="9280ddd1fe48ff29ca376c36196460230792b1aee57c82b60f4313c7a42de828" Jul 10 19:00:49.941890 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:49.941849 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dd8w8" event={"ID":"651e6091-4d1a-4adc-90bf-b0a7e5f8291c","Type":"ContainerStarted","Data":"099f183d78e878008e54b4487621e46a814b3a2a32118383ae308a4457f5ed11"} Jul 10 19:00:50.513114 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:50.513081 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:50.613216 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:50.613182 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:50.673344 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:50.673301 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:51.223295 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:51.223262 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:51.277754 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:51.277724 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:52.983072 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:52.983038 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:53.043999 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:53.043963 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:54.120328 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:54.120285 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" podUID="24c51490-aab2-4fd0-9ffc-06886c5515b3" containerName="registry" containerID="cri-o://71707cd36588bf55f374385c864f58e1e7d756c48545133fbfd9611f0261e0fd" gracePeriod=30 Jul 10 19:00:54.359253 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:54.359230 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 19:00:54.441781 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:54.441749 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/24c51490-aab2-4fd0-9ffc-06886c5515b3-image-registry-private-configuration\") pod \"24c51490-aab2-4fd0-9ffc-06886c5515b3\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " Jul 10 19:00:54.441781 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:54.441785 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-registry-tls\") pod \"24c51490-aab2-4fd0-9ffc-06886c5515b3\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " Jul 10 19:00:54.441975 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:54.441853 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dt98t\" (UniqueName: \"kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-kube-api-access-dt98t\") pod \"24c51490-aab2-4fd0-9ffc-06886c5515b3\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " Jul 10 19:00:54.441975 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:54.441902 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-bound-sa-token\") pod \"24c51490-aab2-4fd0-9ffc-06886c5515b3\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " Jul 10 19:00:54.441975 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:54.441926 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24c51490-aab2-4fd0-9ffc-06886c5515b3-trusted-ca\") pod \"24c51490-aab2-4fd0-9ffc-06886c5515b3\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " Jul 10 19:00:54.441975 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:54.441949 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/24c51490-aab2-4fd0-9ffc-06886c5515b3-ca-trust-extracted\") pod \"24c51490-aab2-4fd0-9ffc-06886c5515b3\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " Jul 10 19:00:54.442159 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:54.441999 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/24c51490-aab2-4fd0-9ffc-06886c5515b3-installation-pull-secrets\") pod \"24c51490-aab2-4fd0-9ffc-06886c5515b3\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " Jul 10 19:00:54.442159 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:54.442023 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/24c51490-aab2-4fd0-9ffc-06886c5515b3-registry-certificates\") pod \"24c51490-aab2-4fd0-9ffc-06886c5515b3\" (UID: \"24c51490-aab2-4fd0-9ffc-06886c5515b3\") " Jul 10 19:00:54.442464 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:54.442402 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24c51490-aab2-4fd0-9ffc-06886c5515b3-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "24c51490-aab2-4fd0-9ffc-06886c5515b3" (UID: "24c51490-aab2-4fd0-9ffc-06886c5515b3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:00:54.442684 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:54.442660 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24c51490-aab2-4fd0-9ffc-06886c5515b3-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "24c51490-aab2-4fd0-9ffc-06886c5515b3" (UID: "24c51490-aab2-4fd0-9ffc-06886c5515b3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:00:54.444498 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:54.444470 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24c51490-aab2-4fd0-9ffc-06886c5515b3-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "24c51490-aab2-4fd0-9ffc-06886c5515b3" (UID: "24c51490-aab2-4fd0-9ffc-06886c5515b3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:00:54.444691 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:54.444661 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-kube-api-access-dt98t" (OuterVolumeSpecName: "kube-api-access-dt98t") pod "24c51490-aab2-4fd0-9ffc-06886c5515b3" (UID: "24c51490-aab2-4fd0-9ffc-06886c5515b3"). InnerVolumeSpecName "kube-api-access-dt98t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:00:54.444782 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:54.444690 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24c51490-aab2-4fd0-9ffc-06886c5515b3-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "24c51490-aab2-4fd0-9ffc-06886c5515b3" (UID: "24c51490-aab2-4fd0-9ffc-06886c5515b3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:00:54.444782 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:54.444732 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "24c51490-aab2-4fd0-9ffc-06886c5515b3" (UID: "24c51490-aab2-4fd0-9ffc-06886c5515b3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:00:54.444782 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:54.444730 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "24c51490-aab2-4fd0-9ffc-06886c5515b3" (UID: "24c51490-aab2-4fd0-9ffc-06886c5515b3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:00:54.450111 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:54.450084 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c51490-aab2-4fd0-9ffc-06886c5515b3-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "24c51490-aab2-4fd0-9ffc-06886c5515b3" (UID: "24c51490-aab2-4fd0-9ffc-06886c5515b3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:00:54.543372 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:54.543335 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt98t\" (UniqueName: \"kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-kube-api-access-dt98t\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:00:54.543372 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:54.543365 2560 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-bound-sa-token\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:00:54.543372 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:54.543378 2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24c51490-aab2-4fd0-9ffc-06886c5515b3-trusted-ca\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:00:54.543636 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:54.543389 2560 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/24c51490-aab2-4fd0-9ffc-06886c5515b3-ca-trust-extracted\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:00:54.543636 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:54.543401 2560 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/24c51490-aab2-4fd0-9ffc-06886c5515b3-installation-pull-secrets\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:00:54.543636 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:54.543413 2560 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/24c51490-aab2-4fd0-9ffc-06886c5515b3-registry-certificates\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:00:54.543636 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:54.543427 2560 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/24c51490-aab2-4fd0-9ffc-06886c5515b3-image-registry-private-configuration\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:00:54.543636 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:54.543454 2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24c51490-aab2-4fd0-9ffc-06886c5515b3-registry-tls\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:00:54.961308 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:54.961275 2560 generic.go:356] "Generic (PLEG): container finished" podID="24c51490-aab2-4fd0-9ffc-06886c5515b3" containerID="71707cd36588bf55f374385c864f58e1e7d756c48545133fbfd9611f0261e0fd" exitCode=0 Jul 10 19:00:54.961487 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:54.961325 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" Jul 10 19:00:54.961487 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:54.961348 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" event={"ID":"24c51490-aab2-4fd0-9ffc-06886c5515b3","Type":"ContainerDied","Data":"71707cd36588bf55f374385c864f58e1e7d756c48545133fbfd9611f0261e0fd"} Jul 10 19:00:54.961487 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:54.961388 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84bdbb94b5-gnz5b" event={"ID":"24c51490-aab2-4fd0-9ffc-06886c5515b3","Type":"ContainerDied","Data":"bfc5dc6b9b590013862e5ad728f6b25530f52b9e70d90ac2cb0e3401ce9c44c3"} Jul 10 19:00:54.961487 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:54.961409 2560 scope.go:117] "RemoveContainer" containerID="71707cd36588bf55f374385c864f58e1e7d756c48545133fbfd9611f0261e0fd" Jul 10 19:00:54.975244 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:54.975221 2560 scope.go:117] "RemoveContainer" containerID="71707cd36588bf55f374385c864f58e1e7d756c48545133fbfd9611f0261e0fd" Jul 10 19:00:54.975668 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:00:54.975647 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71707cd36588bf55f374385c864f58e1e7d756c48545133fbfd9611f0261e0fd\": container with ID starting with 71707cd36588bf55f374385c864f58e1e7d756c48545133fbfd9611f0261e0fd not found: ID does not exist" containerID="71707cd36588bf55f374385c864f58e1e7d756c48545133fbfd9611f0261e0fd" Jul 10 19:00:54.975763 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:54.975676 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71707cd36588bf55f374385c864f58e1e7d756c48545133fbfd9611f0261e0fd"} err="failed to get container status \"71707cd36588bf55f374385c864f58e1e7d756c48545133fbfd9611f0261e0fd\": rpc error: code = NotFound desc = could not find container \"71707cd36588bf55f374385c864f58e1e7d756c48545133fbfd9611f0261e0fd\": container with ID starting with 71707cd36588bf55f374385c864f58e1e7d756c48545133fbfd9611f0261e0fd not found: ID does not exist" Jul 10 19:00:54.997557 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:54.997531 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84bdbb94b5-gnz5b"] Jul 10 19:00:55.003685 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:55.003664 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-84bdbb94b5-gnz5b"] Jul 10 19:00:55.252212 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:55.252134 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24c51490-aab2-4fd0-9ffc-06886c5515b3" path="/var/lib/kubelet/pods/24c51490-aab2-4fd0-9ffc-06886c5515b3/volumes" Jul 10 19:00:57.547085 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:57.547054 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:57.647535 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:57.647502 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:58.785536 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:58.785506 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:58.980693 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:58.980607 2560 generic.go:356] "Generic (PLEG): container finished" podID="88da616b-477f-491f-bf78-d80b9c187a24" containerID="fa85fe52d26954646c65b2ded834436e5939b39bb4eef1693c52dea90057d6c6" exitCode=0 Jul 10 19:00:58.980693 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:58.980681 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-vshg7" event={"ID":"88da616b-477f-491f-bf78-d80b9c187a24","Type":"ContainerDied","Data":"fa85fe52d26954646c65b2ded834436e5939b39bb4eef1693c52dea90057d6c6"} Jul 10 19:00:58.981040 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:58.981026 2560 scope.go:117] "RemoveContainer" containerID="fa85fe52d26954646c65b2ded834436e5939b39bb4eef1693c52dea90057d6c6" Jul 10 19:00:59.795063 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:59.794997 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:59.894170 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:59.894134 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:59.987191 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:00:59.987156 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-vshg7" event={"ID":"88da616b-477f-491f-bf78-d80b9c187a24","Type":"ContainerStarted","Data":"43fe15dc1529824fbf2c56c439ca5bff41efd59e87f8f6189b4f12c31e576710"} Jul 10 19:01:00.579137 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:00.579107 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:06.638699 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:06.638669 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-546f8cd949-89c7k" Jul 10 19:01:06.642545 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:06.642523 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-546f8cd949-89c7k" Jul 10 19:01:12.549536 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:12.549497 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:12.609326 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:12.609286 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:19.068176 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:19.068139 2560 generic.go:356] "Generic (PLEG): container finished" podID="66525ef2-fac3-4d3d-b138-cded3b6c0bef" containerID="96c66babfccd98de5d30110574c47faa2c0f62d7afe0feaa46c919a4fdba3f87" exitCode=0 Jul 10 19:01:19.068575 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:19.068199 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rsc2t" event={"ID":"66525ef2-fac3-4d3d-b138-cded3b6c0bef","Type":"ContainerDied","Data":"96c66babfccd98de5d30110574c47faa2c0f62d7afe0feaa46c919a4fdba3f87"} Jul 10 19:01:19.068575 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:19.068515 2560 scope.go:117] "RemoveContainer" containerID="96c66babfccd98de5d30110574c47faa2c0f62d7afe0feaa46c919a4fdba3f87" Jul 10 19:01:20.074209 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:20.074174 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-rsc2t" event={"ID":"66525ef2-fac3-4d3d-b138-cded3b6c0bef","Type":"ContainerStarted","Data":"ce16a5942b4744232f32eae35143a0fad54de2d355054fb8b2391f36491e0d4a"} Jul 10 19:01:28.627830 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:28.627745 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:28.646937 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:28.646909 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:29.126775 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:29.126749 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:38.211095 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:38.211058 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:41.230324 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:41.230286 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:41.529801 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:41.529722 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:42.310694 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:42.310657 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 19:01:42.311071 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:42.310928 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="30bfaf4b-6dde-45ac-8a7a-d70a613a31da" containerName="alertmanager" containerID="cri-o://f0a30779bb0568bab045cbf93e73a05c68ff88a51a24d1c8e9ef78bbb8fd8e97" gracePeriod=120 Jul 10 19:01:42.311071 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:42.311007 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="30bfaf4b-6dde-45ac-8a7a-d70a613a31da" containerName="kube-rbac-proxy-metric" containerID="cri-o://2e5f5b4a6fed3d548182ac61c4cd78041fe3e3f962cab88ffef88acaca23c4e9" gracePeriod=120 Jul 10 19:01:42.311071 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:42.311030 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="30bfaf4b-6dde-45ac-8a7a-d70a613a31da" containerName="prom-label-proxy" containerID="cri-o://2f0c4ac99f6fe35ac80f707d3b453afdc794cd91c8ae62242b568ad14fb6ce86" gracePeriod=120 Jul 10 19:01:42.311242 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:42.311051 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="30bfaf4b-6dde-45ac-8a7a-d70a613a31da" containerName="kube-rbac-proxy-web" containerID="cri-o://a75fca2eaa968e64c918946f6cb8f3911481b9157fac2739021e38b1a05ef6b8" gracePeriod=120 Jul 10 19:01:42.311242 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:42.311085 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="30bfaf4b-6dde-45ac-8a7a-d70a613a31da" containerName="config-reloader" containerID="cri-o://d5c128940ead47570db005908b53eeea5c7058dbb80bc291bf0d8d12f71e851e" gracePeriod=120 Jul 10 19:01:42.311242 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:42.311095 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="30bfaf4b-6dde-45ac-8a7a-d70a613a31da" containerName="kube-rbac-proxy" containerID="cri-o://d81fa72fa7b84d428b0b788cba0d8a92926adfa6bc0a6fa6b58c845025707270" gracePeriod=120 Jul 10 19:01:43.171188 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.171155 2560 generic.go:356] "Generic (PLEG): container finished" podID="30bfaf4b-6dde-45ac-8a7a-d70a613a31da" containerID="2f0c4ac99f6fe35ac80f707d3b453afdc794cd91c8ae62242b568ad14fb6ce86" exitCode=0 Jul 10 19:01:43.171188 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.171181 2560 generic.go:356] "Generic (PLEG): container finished" podID="30bfaf4b-6dde-45ac-8a7a-d70a613a31da" containerID="d81fa72fa7b84d428b0b788cba0d8a92926adfa6bc0a6fa6b58c845025707270" exitCode=0 Jul 10 19:01:43.171188 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.171189 2560 generic.go:356] "Generic (PLEG): container finished" podID="30bfaf4b-6dde-45ac-8a7a-d70a613a31da" containerID="d5c128940ead47570db005908b53eeea5c7058dbb80bc291bf0d8d12f71e851e" exitCode=0 Jul 10 19:01:43.171188 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.171195 2560 generic.go:356] "Generic (PLEG): container finished" podID="30bfaf4b-6dde-45ac-8a7a-d70a613a31da" containerID="f0a30779bb0568bab045cbf93e73a05c68ff88a51a24d1c8e9ef78bbb8fd8e97" exitCode=0 Jul 10 19:01:43.171463 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.171223 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"30bfaf4b-6dde-45ac-8a7a-d70a613a31da","Type":"ContainerDied","Data":"2f0c4ac99f6fe35ac80f707d3b453afdc794cd91c8ae62242b568ad14fb6ce86"} Jul 10 19:01:43.171463 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.171288 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"30bfaf4b-6dde-45ac-8a7a-d70a613a31da","Type":"ContainerDied","Data":"d81fa72fa7b84d428b0b788cba0d8a92926adfa6bc0a6fa6b58c845025707270"} Jul 10 19:01:43.171463 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.171318 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"30bfaf4b-6dde-45ac-8a7a-d70a613a31da","Type":"ContainerDied","Data":"d5c128940ead47570db005908b53eeea5c7058dbb80bc291bf0d8d12f71e851e"} Jul 10 19:01:43.171463 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.171349 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"30bfaf4b-6dde-45ac-8a7a-d70a613a31da","Type":"ContainerDied","Data":"f0a30779bb0568bab045cbf93e73a05c68ff88a51a24d1c8e9ef78bbb8fd8e97"} Jul 10 19:01:43.557947 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.557924 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:43.673968 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.673887 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-tls-assets\") pod \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " Jul 10 19:01:43.673968 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.673922 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-cluster-tls-config\") pod \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " Jul 10 19:01:43.673968 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.673969 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-config-out\") pod \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " Jul 10 19:01:43.674243 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.673995 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-web-config\") pod \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " Jul 10 19:01:43.674243 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.674038 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-alertmanager-trusted-ca-bundle\") pod \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " Jul 10 19:01:43.674243 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.674077 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-secret-alertmanager-kube-rbac-proxy-metric\") pod \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " Jul 10 19:01:43.674243 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.674112 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-metrics-client-ca\") pod \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " Jul 10 19:01:43.674243 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.674156 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-secret-alertmanager-kube-rbac-proxy\") pod \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " Jul 10 19:01:43.674243 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.674187 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfmt8\" (UniqueName: \"kubernetes.io/projected/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-kube-api-access-sfmt8\") pod \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " Jul 10 19:01:43.674243 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.674211 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-config-volume\") pod \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " Jul 10 19:01:43.674628 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.674250 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-secret-alertmanager-main-tls\") pod \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " Jul 10 19:01:43.674628 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.674290 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-alertmanager-main-db\") pod \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " Jul 10 19:01:43.674628 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.674320 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-secret-alertmanager-kube-rbac-proxy-web\") pod \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\" (UID: \"30bfaf4b-6dde-45ac-8a7a-d70a613a31da\") " Jul 10 19:01:43.674628 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.674490 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "30bfaf4b-6dde-45ac-8a7a-d70a613a31da" (UID: "30bfaf4b-6dde-45ac-8a7a-d70a613a31da"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:01:43.674833 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.674680 2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:01:43.675276 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.675219 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "30bfaf4b-6dde-45ac-8a7a-d70a613a31da" (UID: "30bfaf4b-6dde-45ac-8a7a-d70a613a31da"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:01:43.675552 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.675522 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "30bfaf4b-6dde-45ac-8a7a-d70a613a31da" (UID: "30bfaf4b-6dde-45ac-8a7a-d70a613a31da"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:01:43.677233 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.676895 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-config-out" (OuterVolumeSpecName: "config-out") pod "30bfaf4b-6dde-45ac-8a7a-d70a613a31da" (UID: "30bfaf4b-6dde-45ac-8a7a-d70a613a31da"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:01:43.677233 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.676901 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "30bfaf4b-6dde-45ac-8a7a-d70a613a31da" (UID: "30bfaf4b-6dde-45ac-8a7a-d70a613a31da"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:01:43.677233 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.676907 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "30bfaf4b-6dde-45ac-8a7a-d70a613a31da" (UID: "30bfaf4b-6dde-45ac-8a7a-d70a613a31da"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:01:43.677880 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.677850 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-kube-api-access-sfmt8" (OuterVolumeSpecName: "kube-api-access-sfmt8") pod "30bfaf4b-6dde-45ac-8a7a-d70a613a31da" (UID: "30bfaf4b-6dde-45ac-8a7a-d70a613a31da"). InnerVolumeSpecName "kube-api-access-sfmt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:01:43.678218 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.678183 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-config-volume" (OuterVolumeSpecName: "config-volume") pod "30bfaf4b-6dde-45ac-8a7a-d70a613a31da" (UID: "30bfaf4b-6dde-45ac-8a7a-d70a613a31da"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:01:43.678308 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.678214 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "30bfaf4b-6dde-45ac-8a7a-d70a613a31da" (UID: "30bfaf4b-6dde-45ac-8a7a-d70a613a31da"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:01:43.678308 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.678269 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "30bfaf4b-6dde-45ac-8a7a-d70a613a31da" (UID: "30bfaf4b-6dde-45ac-8a7a-d70a613a31da"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:01:43.678421 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.678370 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "30bfaf4b-6dde-45ac-8a7a-d70a613a31da" (UID: "30bfaf4b-6dde-45ac-8a7a-d70a613a31da"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:01:43.681729 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.681704 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "30bfaf4b-6dde-45ac-8a7a-d70a613a31da" (UID: "30bfaf4b-6dde-45ac-8a7a-d70a613a31da"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:01:43.688675 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.688651 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-web-config" (OuterVolumeSpecName: "web-config") pod "30bfaf4b-6dde-45ac-8a7a-d70a613a31da" (UID: "30bfaf4b-6dde-45ac-8a7a-d70a613a31da"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:01:43.775808 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.775773 2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-tls-assets\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:01:43.775808 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.775802 2560 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-cluster-tls-config\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:01:43.775808 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.775814 2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-config-out\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:01:43.776044 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.775826 2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-web-config\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:01:43.776044 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.775838 2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:01:43.776044 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.775850 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-metrics-client-ca\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:01:43.776044 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.775862 2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:01:43.776044 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.775875 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfmt8\" (UniqueName: \"kubernetes.io/projected/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-kube-api-access-sfmt8\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:01:43.776044 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.775887 2560 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-config-volume\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:01:43.776044 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.775899 2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-secret-alertmanager-main-tls\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:01:43.776044 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.775911 2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-alertmanager-main-db\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:01:43.776044 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:43.775923 2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/30bfaf4b-6dde-45ac-8a7a-d70a613a31da-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:01:44.178057 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.178021 2560 generic.go:356] "Generic (PLEG): container finished" podID="30bfaf4b-6dde-45ac-8a7a-d70a613a31da" containerID="2e5f5b4a6fed3d548182ac61c4cd78041fe3e3f962cab88ffef88acaca23c4e9" exitCode=0 Jul 10 19:01:44.178057 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.178053 2560 generic.go:356] "Generic (PLEG): container finished" podID="30bfaf4b-6dde-45ac-8a7a-d70a613a31da" containerID="a75fca2eaa968e64c918946f6cb8f3911481b9157fac2739021e38b1a05ef6b8" exitCode=0 Jul 10 19:01:44.178260 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.178097 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"30bfaf4b-6dde-45ac-8a7a-d70a613a31da","Type":"ContainerDied","Data":"2e5f5b4a6fed3d548182ac61c4cd78041fe3e3f962cab88ffef88acaca23c4e9"} Jul 10 19:01:44.178260 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.178118 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.178260 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.178169 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"30bfaf4b-6dde-45ac-8a7a-d70a613a31da","Type":"ContainerDied","Data":"a75fca2eaa968e64c918946f6cb8f3911481b9157fac2739021e38b1a05ef6b8"} Jul 10 19:01:44.178260 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.178198 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"30bfaf4b-6dde-45ac-8a7a-d70a613a31da","Type":"ContainerDied","Data":"57058119a8e0f39578dd7e20d8da06326b9f8d48f871c89d4315e6907f2aebab"} Jul 10 19:01:44.178260 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.178228 2560 scope.go:117] "RemoveContainer" containerID="2f0c4ac99f6fe35ac80f707d3b453afdc794cd91c8ae62242b568ad14fb6ce86" Jul 10 19:01:44.191200 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.191177 2560 scope.go:117] "RemoveContainer" containerID="2e5f5b4a6fed3d548182ac61c4cd78041fe3e3f962cab88ffef88acaca23c4e9" Jul 10 19:01:44.198476 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.198456 2560 scope.go:117] "RemoveContainer" containerID="d81fa72fa7b84d428b0b788cba0d8a92926adfa6bc0a6fa6b58c845025707270" Jul 10 19:01:44.205718 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.205680 2560 scope.go:117] "RemoveContainer" containerID="a75fca2eaa968e64c918946f6cb8f3911481b9157fac2739021e38b1a05ef6b8" Jul 10 19:01:44.207120 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.207097 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 19:01:44.212916 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.212891 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 19:01:44.214306 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.214288 2560 scope.go:117] "RemoveContainer" containerID="d5c128940ead47570db005908b53eeea5c7058dbb80bc291bf0d8d12f71e851e" Jul 10 19:01:44.221196 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.221180 2560 scope.go:117] "RemoveContainer" containerID="f0a30779bb0568bab045cbf93e73a05c68ff88a51a24d1c8e9ef78bbb8fd8e97" Jul 10 19:01:44.228166 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.228144 2560 scope.go:117] "RemoveContainer" containerID="93d4e2c95ff91f10be2463f2c04d76bd5e6852d056244f438941d027277542a9" Jul 10 19:01:44.235361 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.235346 2560 scope.go:117] "RemoveContainer" containerID="2f0c4ac99f6fe35ac80f707d3b453afdc794cd91c8ae62242b568ad14fb6ce86" Jul 10 19:01:44.235665 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:01:44.235645 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f0c4ac99f6fe35ac80f707d3b453afdc794cd91c8ae62242b568ad14fb6ce86\": container with ID starting with 2f0c4ac99f6fe35ac80f707d3b453afdc794cd91c8ae62242b568ad14fb6ce86 not found: ID does not exist" containerID="2f0c4ac99f6fe35ac80f707d3b453afdc794cd91c8ae62242b568ad14fb6ce86" Jul 10 19:01:44.235728 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.235673 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f0c4ac99f6fe35ac80f707d3b453afdc794cd91c8ae62242b568ad14fb6ce86"} err="failed to get container status \"2f0c4ac99f6fe35ac80f707d3b453afdc794cd91c8ae62242b568ad14fb6ce86\": rpc error: code = NotFound desc = could not find container \"2f0c4ac99f6fe35ac80f707d3b453afdc794cd91c8ae62242b568ad14fb6ce86\": container with ID starting with 2f0c4ac99f6fe35ac80f707d3b453afdc794cd91c8ae62242b568ad14fb6ce86 not found: ID does not exist" Jul 10 19:01:44.235728 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.235692 2560 scope.go:117] "RemoveContainer" containerID="2e5f5b4a6fed3d548182ac61c4cd78041fe3e3f962cab88ffef88acaca23c4e9" Jul 10 19:01:44.235925 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:01:44.235906 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e5f5b4a6fed3d548182ac61c4cd78041fe3e3f962cab88ffef88acaca23c4e9\": container with ID starting with 2e5f5b4a6fed3d548182ac61c4cd78041fe3e3f962cab88ffef88acaca23c4e9 not found: ID does not exist" containerID="2e5f5b4a6fed3d548182ac61c4cd78041fe3e3f962cab88ffef88acaca23c4e9" Jul 10 19:01:44.235969 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.235933 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e5f5b4a6fed3d548182ac61c4cd78041fe3e3f962cab88ffef88acaca23c4e9"} err="failed to get container status \"2e5f5b4a6fed3d548182ac61c4cd78041fe3e3f962cab88ffef88acaca23c4e9\": rpc error: code = NotFound desc = could not find container \"2e5f5b4a6fed3d548182ac61c4cd78041fe3e3f962cab88ffef88acaca23c4e9\": container with ID starting with 2e5f5b4a6fed3d548182ac61c4cd78041fe3e3f962cab88ffef88acaca23c4e9 not found: ID does not exist" Jul 10 19:01:44.235969 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.235951 2560 scope.go:117] "RemoveContainer" containerID="d81fa72fa7b84d428b0b788cba0d8a92926adfa6bc0a6fa6b58c845025707270" Jul 10 19:01:44.236201 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:01:44.236186 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d81fa72fa7b84d428b0b788cba0d8a92926adfa6bc0a6fa6b58c845025707270\": container with ID starting with d81fa72fa7b84d428b0b788cba0d8a92926adfa6bc0a6fa6b58c845025707270 not found: ID does not exist" containerID="d81fa72fa7b84d428b0b788cba0d8a92926adfa6bc0a6fa6b58c845025707270" Jul 10 19:01:44.236260 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.236213 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d81fa72fa7b84d428b0b788cba0d8a92926adfa6bc0a6fa6b58c845025707270"} err="failed to get container status \"d81fa72fa7b84d428b0b788cba0d8a92926adfa6bc0a6fa6b58c845025707270\": rpc error: code = NotFound desc = could not find container \"d81fa72fa7b84d428b0b788cba0d8a92926adfa6bc0a6fa6b58c845025707270\": container with ID starting with d81fa72fa7b84d428b0b788cba0d8a92926adfa6bc0a6fa6b58c845025707270 not found: ID does not exist" Jul 10 19:01:44.236260 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.236225 2560 scope.go:117] "RemoveContainer" containerID="a75fca2eaa968e64c918946f6cb8f3911481b9157fac2739021e38b1a05ef6b8" Jul 10 19:01:44.236488 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:01:44.236463 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a75fca2eaa968e64c918946f6cb8f3911481b9157fac2739021e38b1a05ef6b8\": container with ID starting with a75fca2eaa968e64c918946f6cb8f3911481b9157fac2739021e38b1a05ef6b8 not found: ID does not exist" containerID="a75fca2eaa968e64c918946f6cb8f3911481b9157fac2739021e38b1a05ef6b8" Jul 10 19:01:44.236540 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.236493 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a75fca2eaa968e64c918946f6cb8f3911481b9157fac2739021e38b1a05ef6b8"} err="failed to get container status \"a75fca2eaa968e64c918946f6cb8f3911481b9157fac2739021e38b1a05ef6b8\": rpc error: code = NotFound desc = could not find container \"a75fca2eaa968e64c918946f6cb8f3911481b9157fac2739021e38b1a05ef6b8\": container with ID starting with a75fca2eaa968e64c918946f6cb8f3911481b9157fac2739021e38b1a05ef6b8 not found: ID does not exist" Jul 10 19:01:44.236540 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.236507 2560 scope.go:117] "RemoveContainer" containerID="d5c128940ead47570db005908b53eeea5c7058dbb80bc291bf0d8d12f71e851e" Jul 10 19:01:44.236746 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:01:44.236732 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5c128940ead47570db005908b53eeea5c7058dbb80bc291bf0d8d12f71e851e\": container with ID starting with d5c128940ead47570db005908b53eeea5c7058dbb80bc291bf0d8d12f71e851e not found: ID does not exist" containerID="d5c128940ead47570db005908b53eeea5c7058dbb80bc291bf0d8d12f71e851e" Jul 10 19:01:44.236793 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.236747 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c128940ead47570db005908b53eeea5c7058dbb80bc291bf0d8d12f71e851e"} err="failed to get container status \"d5c128940ead47570db005908b53eeea5c7058dbb80bc291bf0d8d12f71e851e\": rpc error: code = NotFound desc = could not find container \"d5c128940ead47570db005908b53eeea5c7058dbb80bc291bf0d8d12f71e851e\": container with ID starting with d5c128940ead47570db005908b53eeea5c7058dbb80bc291bf0d8d12f71e851e not found: ID does not exist" Jul 10 19:01:44.236793 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.236758 2560 scope.go:117] "RemoveContainer" containerID="f0a30779bb0568bab045cbf93e73a05c68ff88a51a24d1c8e9ef78bbb8fd8e97" Jul 10 19:01:44.236960 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:01:44.236945 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0a30779bb0568bab045cbf93e73a05c68ff88a51a24d1c8e9ef78bbb8fd8e97\": container with ID starting with f0a30779bb0568bab045cbf93e73a05c68ff88a51a24d1c8e9ef78bbb8fd8e97 not found: ID does not exist" containerID="f0a30779bb0568bab045cbf93e73a05c68ff88a51a24d1c8e9ef78bbb8fd8e97" Jul 10 19:01:44.237009 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.236963 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0a30779bb0568bab045cbf93e73a05c68ff88a51a24d1c8e9ef78bbb8fd8e97"} err="failed to get container status \"f0a30779bb0568bab045cbf93e73a05c68ff88a51a24d1c8e9ef78bbb8fd8e97\": rpc error: code = NotFound desc = could not find container \"f0a30779bb0568bab045cbf93e73a05c68ff88a51a24d1c8e9ef78bbb8fd8e97\": container with ID starting with f0a30779bb0568bab045cbf93e73a05c68ff88a51a24d1c8e9ef78bbb8fd8e97 not found: ID does not exist" Jul 10 19:01:44.237009 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.236976 2560 scope.go:117] "RemoveContainer" containerID="93d4e2c95ff91f10be2463f2c04d76bd5e6852d056244f438941d027277542a9" Jul 10 19:01:44.237208 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:01:44.237192 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93d4e2c95ff91f10be2463f2c04d76bd5e6852d056244f438941d027277542a9\": container with ID starting with 93d4e2c95ff91f10be2463f2c04d76bd5e6852d056244f438941d027277542a9 not found: ID does not exist" containerID="93d4e2c95ff91f10be2463f2c04d76bd5e6852d056244f438941d027277542a9" Jul 10 19:01:44.237251 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.237210 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93d4e2c95ff91f10be2463f2c04d76bd5e6852d056244f438941d027277542a9"} err="failed to get container status \"93d4e2c95ff91f10be2463f2c04d76bd5e6852d056244f438941d027277542a9\": rpc error: code = NotFound desc = could not find container \"93d4e2c95ff91f10be2463f2c04d76bd5e6852d056244f438941d027277542a9\": container with ID starting with 93d4e2c95ff91f10be2463f2c04d76bd5e6852d056244f438941d027277542a9 not found: ID does not exist" Jul 10 19:01:44.237251 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.237232 2560 scope.go:117] "RemoveContainer" containerID="2f0c4ac99f6fe35ac80f707d3b453afdc794cd91c8ae62242b568ad14fb6ce86" Jul 10 19:01:44.237494 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.237470 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f0c4ac99f6fe35ac80f707d3b453afdc794cd91c8ae62242b568ad14fb6ce86"} err="failed to get container status \"2f0c4ac99f6fe35ac80f707d3b453afdc794cd91c8ae62242b568ad14fb6ce86\": rpc error: code = NotFound desc = could not find container \"2f0c4ac99f6fe35ac80f707d3b453afdc794cd91c8ae62242b568ad14fb6ce86\": container with ID starting with 2f0c4ac99f6fe35ac80f707d3b453afdc794cd91c8ae62242b568ad14fb6ce86 not found: ID does not exist" Jul 10 19:01:44.237494 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.237492 2560 scope.go:117] "RemoveContainer" containerID="2e5f5b4a6fed3d548182ac61c4cd78041fe3e3f962cab88ffef88acaca23c4e9" Jul 10 19:01:44.237729 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.237699 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e5f5b4a6fed3d548182ac61c4cd78041fe3e3f962cab88ffef88acaca23c4e9"} err="failed to get container status \"2e5f5b4a6fed3d548182ac61c4cd78041fe3e3f962cab88ffef88acaca23c4e9\": rpc error: code = NotFound desc = could not find container \"2e5f5b4a6fed3d548182ac61c4cd78041fe3e3f962cab88ffef88acaca23c4e9\": container with ID starting with 2e5f5b4a6fed3d548182ac61c4cd78041fe3e3f962cab88ffef88acaca23c4e9 not found: ID does not exist" Jul 10 19:01:44.237729 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.237721 2560 scope.go:117] "RemoveContainer" containerID="d81fa72fa7b84d428b0b788cba0d8a92926adfa6bc0a6fa6b58c845025707270" Jul 10 19:01:44.237921 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.237897 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d81fa72fa7b84d428b0b788cba0d8a92926adfa6bc0a6fa6b58c845025707270"} err="failed to get container status \"d81fa72fa7b84d428b0b788cba0d8a92926adfa6bc0a6fa6b58c845025707270\": rpc error: code = NotFound desc = could not find container \"d81fa72fa7b84d428b0b788cba0d8a92926adfa6bc0a6fa6b58c845025707270\": container with ID starting with d81fa72fa7b84d428b0b788cba0d8a92926adfa6bc0a6fa6b58c845025707270 not found: ID does not exist" Jul 10 19:01:44.237973 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.237924 2560 scope.go:117] "RemoveContainer" containerID="a75fca2eaa968e64c918946f6cb8f3911481b9157fac2739021e38b1a05ef6b8" Jul 10 19:01:44.238194 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.238174 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a75fca2eaa968e64c918946f6cb8f3911481b9157fac2739021e38b1a05ef6b8"} err="failed to get container status \"a75fca2eaa968e64c918946f6cb8f3911481b9157fac2739021e38b1a05ef6b8\": rpc error: code = NotFound desc = could not find container \"a75fca2eaa968e64c918946f6cb8f3911481b9157fac2739021e38b1a05ef6b8\": container with ID starting with a75fca2eaa968e64c918946f6cb8f3911481b9157fac2739021e38b1a05ef6b8 not found: ID does not exist" Jul 10 19:01:44.238194 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.238195 2560 scope.go:117] "RemoveContainer" containerID="d5c128940ead47570db005908b53eeea5c7058dbb80bc291bf0d8d12f71e851e" Jul 10 19:01:44.238422 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.238406 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c128940ead47570db005908b53eeea5c7058dbb80bc291bf0d8d12f71e851e"} err="failed to get container status \"d5c128940ead47570db005908b53eeea5c7058dbb80bc291bf0d8d12f71e851e\": rpc error: code = NotFound desc = could not find container \"d5c128940ead47570db005908b53eeea5c7058dbb80bc291bf0d8d12f71e851e\": container with ID starting with d5c128940ead47570db005908b53eeea5c7058dbb80bc291bf0d8d12f71e851e not found: ID does not exist" Jul 10 19:01:44.238484 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.238423 2560 scope.go:117] "RemoveContainer" containerID="f0a30779bb0568bab045cbf93e73a05c68ff88a51a24d1c8e9ef78bbb8fd8e97" Jul 10 19:01:44.238659 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.238644 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0a30779bb0568bab045cbf93e73a05c68ff88a51a24d1c8e9ef78bbb8fd8e97"} err="failed to get container status \"f0a30779bb0568bab045cbf93e73a05c68ff88a51a24d1c8e9ef78bbb8fd8e97\": rpc error: code = NotFound desc = could not find container \"f0a30779bb0568bab045cbf93e73a05c68ff88a51a24d1c8e9ef78bbb8fd8e97\": container with ID starting with f0a30779bb0568bab045cbf93e73a05c68ff88a51a24d1c8e9ef78bbb8fd8e97 not found: ID does not exist" Jul 10 19:01:44.238712 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.238660 2560 scope.go:117] "RemoveContainer" containerID="93d4e2c95ff91f10be2463f2c04d76bd5e6852d056244f438941d027277542a9" Jul 10 19:01:44.238919 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.238902 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93d4e2c95ff91f10be2463f2c04d76bd5e6852d056244f438941d027277542a9"} err="failed to get container status \"93d4e2c95ff91f10be2463f2c04d76bd5e6852d056244f438941d027277542a9\": rpc error: code = NotFound desc = could not find container \"93d4e2c95ff91f10be2463f2c04d76bd5e6852d056244f438941d027277542a9\": container with ID starting with 93d4e2c95ff91f10be2463f2c04d76bd5e6852d056244f438941d027277542a9 not found: ID does not exist" Jul 10 19:01:44.242301 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.242281 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 19:01:44.242610 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.242597 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30bfaf4b-6dde-45ac-8a7a-d70a613a31da" containerName="kube-rbac-proxy" Jul 10 19:01:44.242657 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.242611 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="30bfaf4b-6dde-45ac-8a7a-d70a613a31da" containerName="kube-rbac-proxy" Jul 10 19:01:44.242657 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.242622 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30bfaf4b-6dde-45ac-8a7a-d70a613a31da" containerName="kube-rbac-proxy-web" Jul 10 19:01:44.242657 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.242629 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="30bfaf4b-6dde-45ac-8a7a-d70a613a31da" containerName="kube-rbac-proxy-web" Jul 10 19:01:44.242657 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.242639 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30bfaf4b-6dde-45ac-8a7a-d70a613a31da" containerName="prom-label-proxy" Jul 10 19:01:44.242657 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.242645 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="30bfaf4b-6dde-45ac-8a7a-d70a613a31da" containerName="prom-label-proxy" Jul 10 19:01:44.242657 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.242654 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24c51490-aab2-4fd0-9ffc-06886c5515b3" containerName="registry" Jul 10 19:01:44.242828 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.242660 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="24c51490-aab2-4fd0-9ffc-06886c5515b3" containerName="registry" Jul 10 19:01:44.242828 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.242666 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30bfaf4b-6dde-45ac-8a7a-d70a613a31da" containerName="kube-rbac-proxy-metric" Jul 10 19:01:44.242828 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.242671 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="30bfaf4b-6dde-45ac-8a7a-d70a613a31da" containerName="kube-rbac-proxy-metric" Jul 10 19:01:44.242828 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.242680 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30bfaf4b-6dde-45ac-8a7a-d70a613a31da" containerName="alertmanager" Jul 10 19:01:44.242828 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.242686 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="30bfaf4b-6dde-45ac-8a7a-d70a613a31da" containerName="alertmanager" Jul 10 19:01:44.242828 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.242697 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30bfaf4b-6dde-45ac-8a7a-d70a613a31da" containerName="config-reloader" Jul 10 19:01:44.242828 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.242702 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="30bfaf4b-6dde-45ac-8a7a-d70a613a31da" containerName="config-reloader" Jul 10 19:01:44.242828 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.242709 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30bfaf4b-6dde-45ac-8a7a-d70a613a31da" containerName="init-config-reloader" Jul 10 19:01:44.242828 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.242714 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="30bfaf4b-6dde-45ac-8a7a-d70a613a31da" containerName="init-config-reloader" Jul 10 19:01:44.242828 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.242772 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="30bfaf4b-6dde-45ac-8a7a-d70a613a31da" containerName="kube-rbac-proxy-web" Jul 10 19:01:44.242828 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.242779 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="30bfaf4b-6dde-45ac-8a7a-d70a613a31da" containerName="kube-rbac-proxy-metric" Jul 10 19:01:44.242828 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.242786 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="30bfaf4b-6dde-45ac-8a7a-d70a613a31da" containerName="config-reloader" Jul 10 19:01:44.242828 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.242792 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="30bfaf4b-6dde-45ac-8a7a-d70a613a31da" containerName="alertmanager" Jul 10 19:01:44.242828 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.242799 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="30bfaf4b-6dde-45ac-8a7a-d70a613a31da" containerName="prom-label-proxy" Jul 10 19:01:44.242828 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.242805 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="30bfaf4b-6dde-45ac-8a7a-d70a613a31da" containerName="kube-rbac-proxy" Jul 10 19:01:44.242828 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.242810 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="24c51490-aab2-4fd0-9ffc-06886c5515b3" containerName="registry" Jul 10 19:01:44.246754 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.246734 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.249675 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.249654 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 10 19:01:44.249772 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.249657 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 10 19:01:44.249772 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.249733 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 10 19:01:44.249772 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.249758 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 10 19:01:44.249938 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.249734 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-flsnz\"" Jul 10 19:01:44.250045 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.250030 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 10 19:01:44.250101 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.250045 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 10 19:01:44.250101 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.250052 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 10 19:01:44.250195 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.250116 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 10 19:01:44.255113 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.255089 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 10 19:01:44.261400 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.261380 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 19:01:44.278381 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.278349 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a527d405-ebba-4ca0-a6fa-a7a29717965c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.278381 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.278383 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a527d405-ebba-4ca0-a6fa-a7a29717965c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.278566 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.278402 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a527d405-ebba-4ca0-a6fa-a7a29717965c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.278566 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.278421 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a527d405-ebba-4ca0-a6fa-a7a29717965c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.278566 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.278516 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a527d405-ebba-4ca0-a6fa-a7a29717965c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.278566 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.278543 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a527d405-ebba-4ca0-a6fa-a7a29717965c-config-volume\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.278566 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.278564 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a527d405-ebba-4ca0-a6fa-a7a29717965c-web-config\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.278747 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.278581 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a527d405-ebba-4ca0-a6fa-a7a29717965c-config-out\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.278747 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.278635 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpp49\" (UniqueName: \"kubernetes.io/projected/a527d405-ebba-4ca0-a6fa-a7a29717965c-kube-api-access-cpp49\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.278747 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.278686 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a527d405-ebba-4ca0-a6fa-a7a29717965c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.278747 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.278703 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a527d405-ebba-4ca0-a6fa-a7a29717965c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.278970 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.278747 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a527d405-ebba-4ca0-a6fa-a7a29717965c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.278970 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.278767 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a527d405-ebba-4ca0-a6fa-a7a29717965c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.379314 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.379281 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a527d405-ebba-4ca0-a6fa-a7a29717965c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.379314 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.379321 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a527d405-ebba-4ca0-a6fa-a7a29717965c-config-volume\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.379576 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.379343 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a527d405-ebba-4ca0-a6fa-a7a29717965c-web-config\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.379576 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.379366 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a527d405-ebba-4ca0-a6fa-a7a29717965c-config-out\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.379576 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.379404 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpp49\" (UniqueName: \"kubernetes.io/projected/a527d405-ebba-4ca0-a6fa-a7a29717965c-kube-api-access-cpp49\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.379576 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.379468 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a527d405-ebba-4ca0-a6fa-a7a29717965c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.379978 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.379621 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a527d405-ebba-4ca0-a6fa-a7a29717965c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.379978 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.379701 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a527d405-ebba-4ca0-a6fa-a7a29717965c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.379978 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.379735 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a527d405-ebba-4ca0-a6fa-a7a29717965c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.379978 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.379777 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a527d405-ebba-4ca0-a6fa-a7a29717965c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.379978 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.379809 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a527d405-ebba-4ca0-a6fa-a7a29717965c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.379978 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.379834 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a527d405-ebba-4ca0-a6fa-a7a29717965c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.379978 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.379864 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a527d405-ebba-4ca0-a6fa-a7a29717965c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.380461 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.380417 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a527d405-ebba-4ca0-a6fa-a7a29717965c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.380531 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.380505 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a527d405-ebba-4ca0-a6fa-a7a29717965c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.381595 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.381546 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a527d405-ebba-4ca0-a6fa-a7a29717965c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.382576 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.382532 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a527d405-ebba-4ca0-a6fa-a7a29717965c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.382798 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.382776 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a527d405-ebba-4ca0-a6fa-a7a29717965c-web-config\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.382921 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.382901 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a527d405-ebba-4ca0-a6fa-a7a29717965c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.383019 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.382998 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a527d405-ebba-4ca0-a6fa-a7a29717965c-config-volume\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.383199 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.383180 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a527d405-ebba-4ca0-a6fa-a7a29717965c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.383373 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.383354 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a527d405-ebba-4ca0-a6fa-a7a29717965c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.383638 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.383617 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a527d405-ebba-4ca0-a6fa-a7a29717965c-config-out\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.383701 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.383675 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a527d405-ebba-4ca0-a6fa-a7a29717965c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.383826 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.383810 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a527d405-ebba-4ca0-a6fa-a7a29717965c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.388049 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.388034 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpp49\" (UniqueName: \"kubernetes.io/projected/a527d405-ebba-4ca0-a6fa-a7a29717965c-kube-api-access-cpp49\") pod \"alertmanager-main-0\" (UID: \"a527d405-ebba-4ca0-a6fa-a7a29717965c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.557865 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.557776 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 19:01:44.695179 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:44.695144 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 19:01:44.698176 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:01:44.698152 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda527d405_ebba_4ca0_a6fa_a7a29717965c.slice/crio-a71fd27c1411f8c53be9d0e14edcfac70d42f9c35d396414f6700b65db131f86 WatchSource:0}: Error finding container a71fd27c1411f8c53be9d0e14edcfac70d42f9c35d396414f6700b65db131f86: Status 404 returned error can't find the container with id a71fd27c1411f8c53be9d0e14edcfac70d42f9c35d396414f6700b65db131f86 Jul 10 19:01:45.183345 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:45.183309 2560 generic.go:356] "Generic (PLEG): container finished" podID="a527d405-ebba-4ca0-a6fa-a7a29717965c" containerID="90dd15b6343b133440ed35807eb5643b743495d6c0229742add4ddd1ec7ec182" exitCode=0 Jul 10 19:01:45.183557 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:45.183402 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a527d405-ebba-4ca0-a6fa-a7a29717965c","Type":"ContainerDied","Data":"90dd15b6343b133440ed35807eb5643b743495d6c0229742add4ddd1ec7ec182"} Jul 10 19:01:45.183557 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:45.183519 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a527d405-ebba-4ca0-a6fa-a7a29717965c","Type":"ContainerStarted","Data":"a71fd27c1411f8c53be9d0e14edcfac70d42f9c35d396414f6700b65db131f86"} Jul 10 19:01:45.253079 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:45.253047 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30bfaf4b-6dde-45ac-8a7a-d70a613a31da" path="/var/lib/kubelet/pods/30bfaf4b-6dde-45ac-8a7a-d70a613a31da/volumes" Jul 10 19:01:46.191895 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.191861 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a527d405-ebba-4ca0-a6fa-a7a29717965c","Type":"ContainerStarted","Data":"35858538de921f8a626d0c3032d95e1583619ddb4d7efb1ab6c55396973dde7d"} Jul 10 19:01:46.192355 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.191977 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a527d405-ebba-4ca0-a6fa-a7a29717965c","Type":"ContainerStarted","Data":"f5178f078e1bdb2328a4537193a6e032c52a1dc9b14f86e71941a4a9cf0d50fb"} Jul 10 19:01:46.192355 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.192028 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a527d405-ebba-4ca0-a6fa-a7a29717965c","Type":"ContainerStarted","Data":"53fd8a9d73fc318f2815462f992b8a2868407ecc413a5d8baafa2e285ef3c899"} Jul 10 19:01:46.192355 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.192075 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a527d405-ebba-4ca0-a6fa-a7a29717965c","Type":"ContainerStarted","Data":"543a155e3405c6cdf40471b68a5aa9572913b85eef31afda82ccac389c2260a2"} Jul 10 19:01:46.192355 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.192128 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a527d405-ebba-4ca0-a6fa-a7a29717965c","Type":"ContainerStarted","Data":"108933dfa6fd8acba47a91eb87a6b8fa4c45fa3dce8f69c7426df79652301d34"} Jul 10 19:01:46.192355 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.192167 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a527d405-ebba-4ca0-a6fa-a7a29717965c","Type":"ContainerStarted","Data":"2fd554bef7c3fca48196ab6ef593978885f16a08fef546bc8a1cb16755a8be8c"} Jul 10 19:01:46.259120 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.259060 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.242905929 podStartE2EDuration="2.259042237s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.136308ms" isStatefulPod=false podCreationTimestamp="2026-07-10 19:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:46.256607355 +0000 UTC m=+197.586748196" watchObservedRunningTime="2026-07-10 19:01:46.259042237 +0000 UTC m=+197.589183047" Jul 10 19:01:46.397322 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.397291 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-94bd4b7b7-m4h5j"] Jul 10 19:01:46.400062 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.400046 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-94bd4b7b7-m4h5j" Jul 10 19:01:46.403092 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.403048 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 10 19:01:46.403265 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.403242 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 10 19:01:46.403366 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.403273 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 10 19:01:46.403366 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.403284 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 10 19:01:46.403648 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.403632 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-hbl7f\"" Jul 10 19:01:46.403959 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.403944 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 10 19:01:46.408662 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.408643 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 10 19:01:46.415888 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.415869 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-94bd4b7b7-m4h5j"] Jul 10 19:01:46.497168 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.497078 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5efa515b-3b7c-4c71-820e-15c9cf7617b7-serving-certs-ca-bundle\") pod \"telemeter-client-94bd4b7b7-m4h5j\" (UID: \"5efa515b-3b7c-4c71-820e-15c9cf7617b7\") " pod="openshift-monitoring/telemeter-client-94bd4b7b7-m4h5j" Jul 10 19:01:46.497168 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.497144 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5efa515b-3b7c-4c71-820e-15c9cf7617b7-secret-telemeter-client\") pod \"telemeter-client-94bd4b7b7-m4h5j\" (UID: \"5efa515b-3b7c-4c71-820e-15c9cf7617b7\") " pod="openshift-monitoring/telemeter-client-94bd4b7b7-m4h5j" Jul 10 19:01:46.497168 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.497170 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5efa515b-3b7c-4c71-820e-15c9cf7617b7-telemeter-client-tls\") pod \"telemeter-client-94bd4b7b7-m4h5j\" (UID: \"5efa515b-3b7c-4c71-820e-15c9cf7617b7\") " pod="openshift-monitoring/telemeter-client-94bd4b7b7-m4h5j" Jul 10 19:01:46.497382 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.497204 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5efa515b-3b7c-4c71-820e-15c9cf7617b7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-94bd4b7b7-m4h5j\" (UID: \"5efa515b-3b7c-4c71-820e-15c9cf7617b7\") " pod="openshift-monitoring/telemeter-client-94bd4b7b7-m4h5j" Jul 10 19:01:46.497382 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.497229 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6f9l\" (UniqueName: \"kubernetes.io/projected/5efa515b-3b7c-4c71-820e-15c9cf7617b7-kube-api-access-m6f9l\") pod \"telemeter-client-94bd4b7b7-m4h5j\" (UID: \"5efa515b-3b7c-4c71-820e-15c9cf7617b7\") " pod="openshift-monitoring/telemeter-client-94bd4b7b7-m4h5j" Jul 10 19:01:46.497382 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.497304 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5efa515b-3b7c-4c71-820e-15c9cf7617b7-metrics-client-ca\") pod \"telemeter-client-94bd4b7b7-m4h5j\" (UID: \"5efa515b-3b7c-4c71-820e-15c9cf7617b7\") " pod="openshift-monitoring/telemeter-client-94bd4b7b7-m4h5j" Jul 10 19:01:46.497382 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.497346 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5efa515b-3b7c-4c71-820e-15c9cf7617b7-federate-client-tls\") pod \"telemeter-client-94bd4b7b7-m4h5j\" (UID: \"5efa515b-3b7c-4c71-820e-15c9cf7617b7\") " pod="openshift-monitoring/telemeter-client-94bd4b7b7-m4h5j" Jul 10 19:01:46.497382 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.497373 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5efa515b-3b7c-4c71-820e-15c9cf7617b7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-94bd4b7b7-m4h5j\" (UID: \"5efa515b-3b7c-4c71-820e-15c9cf7617b7\") " pod="openshift-monitoring/telemeter-client-94bd4b7b7-m4h5j" Jul 10 19:01:46.598476 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.598411 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5efa515b-3b7c-4c71-820e-15c9cf7617b7-secret-telemeter-client\") pod \"telemeter-client-94bd4b7b7-m4h5j\" (UID: \"5efa515b-3b7c-4c71-820e-15c9cf7617b7\") " pod="openshift-monitoring/telemeter-client-94bd4b7b7-m4h5j" Jul 10 19:01:46.598476 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.598480 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5efa515b-3b7c-4c71-820e-15c9cf7617b7-telemeter-client-tls\") pod \"telemeter-client-94bd4b7b7-m4h5j\" (UID: \"5efa515b-3b7c-4c71-820e-15c9cf7617b7\") " pod="openshift-monitoring/telemeter-client-94bd4b7b7-m4h5j" Jul 10 19:01:46.598682 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.598519 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5efa515b-3b7c-4c71-820e-15c9cf7617b7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-94bd4b7b7-m4h5j\" (UID: \"5efa515b-3b7c-4c71-820e-15c9cf7617b7\") " pod="openshift-monitoring/telemeter-client-94bd4b7b7-m4h5j" Jul 10 19:01:46.598682 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.598538 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6f9l\" (UniqueName: \"kubernetes.io/projected/5efa515b-3b7c-4c71-820e-15c9cf7617b7-kube-api-access-m6f9l\") pod \"telemeter-client-94bd4b7b7-m4h5j\" (UID: \"5efa515b-3b7c-4c71-820e-15c9cf7617b7\") " pod="openshift-monitoring/telemeter-client-94bd4b7b7-m4h5j" Jul 10 19:01:46.598682 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.598569 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5efa515b-3b7c-4c71-820e-15c9cf7617b7-metrics-client-ca\") pod \"telemeter-client-94bd4b7b7-m4h5j\" (UID: \"5efa515b-3b7c-4c71-820e-15c9cf7617b7\") " pod="openshift-monitoring/telemeter-client-94bd4b7b7-m4h5j" Jul 10 19:01:46.598682 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.598605 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5efa515b-3b7c-4c71-820e-15c9cf7617b7-federate-client-tls\") pod \"telemeter-client-94bd4b7b7-m4h5j\" (UID: \"5efa515b-3b7c-4c71-820e-15c9cf7617b7\") " pod="openshift-monitoring/telemeter-client-94bd4b7b7-m4h5j" Jul 10 19:01:46.598682 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.598622 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5efa515b-3b7c-4c71-820e-15c9cf7617b7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-94bd4b7b7-m4h5j\" (UID: \"5efa515b-3b7c-4c71-820e-15c9cf7617b7\") " pod="openshift-monitoring/telemeter-client-94bd4b7b7-m4h5j" Jul 10 19:01:46.598682 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.598645 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5efa515b-3b7c-4c71-820e-15c9cf7617b7-serving-certs-ca-bundle\") pod \"telemeter-client-94bd4b7b7-m4h5j\" (UID: \"5efa515b-3b7c-4c71-820e-15c9cf7617b7\") " pod="openshift-monitoring/telemeter-client-94bd4b7b7-m4h5j" Jul 10 19:01:46.599384 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.599357 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5efa515b-3b7c-4c71-820e-15c9cf7617b7-serving-certs-ca-bundle\") pod \"telemeter-client-94bd4b7b7-m4h5j\" (UID: \"5efa515b-3b7c-4c71-820e-15c9cf7617b7\") " pod="openshift-monitoring/telemeter-client-94bd4b7b7-m4h5j" Jul 10 19:01:46.599628 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.599602 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5efa515b-3b7c-4c71-820e-15c9cf7617b7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-94bd4b7b7-m4h5j\" (UID: \"5efa515b-3b7c-4c71-820e-15c9cf7617b7\") " pod="openshift-monitoring/telemeter-client-94bd4b7b7-m4h5j" Jul 10 19:01:46.599681 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.599662 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5efa515b-3b7c-4c71-820e-15c9cf7617b7-metrics-client-ca\") pod \"telemeter-client-94bd4b7b7-m4h5j\" (UID: \"5efa515b-3b7c-4c71-820e-15c9cf7617b7\") " pod="openshift-monitoring/telemeter-client-94bd4b7b7-m4h5j" Jul 10 19:01:46.601700 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.601674 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5efa515b-3b7c-4c71-820e-15c9cf7617b7-federate-client-tls\") pod \"telemeter-client-94bd4b7b7-m4h5j\" (UID: \"5efa515b-3b7c-4c71-820e-15c9cf7617b7\") " pod="openshift-monitoring/telemeter-client-94bd4b7b7-m4h5j" Jul 10 19:01:46.601798 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.601705 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5efa515b-3b7c-4c71-820e-15c9cf7617b7-secret-telemeter-client\") pod \"telemeter-client-94bd4b7b7-m4h5j\" (UID: \"5efa515b-3b7c-4c71-820e-15c9cf7617b7\") " pod="openshift-monitoring/telemeter-client-94bd4b7b7-m4h5j" Jul 10 19:01:46.601798 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.601681 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5efa515b-3b7c-4c71-820e-15c9cf7617b7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-94bd4b7b7-m4h5j\" (UID: \"5efa515b-3b7c-4c71-820e-15c9cf7617b7\") " pod="openshift-monitoring/telemeter-client-94bd4b7b7-m4h5j" Jul 10 19:01:46.601966 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.601950 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5efa515b-3b7c-4c71-820e-15c9cf7617b7-telemeter-client-tls\") pod \"telemeter-client-94bd4b7b7-m4h5j\" (UID: \"5efa515b-3b7c-4c71-820e-15c9cf7617b7\") " pod="openshift-monitoring/telemeter-client-94bd4b7b7-m4h5j" Jul 10 19:01:46.606962 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.606941 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6f9l\" (UniqueName: \"kubernetes.io/projected/5efa515b-3b7c-4c71-820e-15c9cf7617b7-kube-api-access-m6f9l\") pod \"telemeter-client-94bd4b7b7-m4h5j\" (UID: \"5efa515b-3b7c-4c71-820e-15c9cf7617b7\") " pod="openshift-monitoring/telemeter-client-94bd4b7b7-m4h5j" Jul 10 19:01:46.652558 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.652525 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 19:01:46.652825 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.652787 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9b26709a-e61c-4a9a-8d89-5557b8f999b9" containerName="prometheus" containerID="cri-o://fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c" gracePeriod=600 Jul 10 19:01:46.652825 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.652800 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9b26709a-e61c-4a9a-8d89-5557b8f999b9" containerName="kube-rbac-proxy-thanos" containerID="cri-o://f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6" gracePeriod=600 Jul 10 19:01:46.652983 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.652830 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9b26709a-e61c-4a9a-8d89-5557b8f999b9" containerName="kube-rbac-proxy" containerID="cri-o://6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163" gracePeriod=600 Jul 10 19:01:46.652983 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.652843 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9b26709a-e61c-4a9a-8d89-5557b8f999b9" containerName="thanos-sidecar" containerID="cri-o://470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92" gracePeriod=600 Jul 10 19:01:46.652983 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.652808 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9b26709a-e61c-4a9a-8d89-5557b8f999b9" containerName="kube-rbac-proxy-web" containerID="cri-o://ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c" gracePeriod=600 Jul 10 19:01:46.652983 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.652841 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9b26709a-e61c-4a9a-8d89-5557b8f999b9" containerName="config-reloader" containerID="cri-o://aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5" gracePeriod=600 Jul 10 19:01:46.710030 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.710007 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-94bd4b7b7-m4h5j" Jul 10 19:01:46.855676 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.855646 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-94bd4b7b7-m4h5j"] Jul 10 19:01:46.856829 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:01:46.856808 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5efa515b_3b7c_4c71_820e_15c9cf7617b7.slice/crio-10db2a9508a9e8491eb1d5c2d8c75821a496b3fb2ac5e59a75d5023d25ec0621 WatchSource:0}: Error finding container 10db2a9508a9e8491eb1d5c2d8c75821a496b3fb2ac5e59a75d5023d25ec0621: Status 404 returned error can't find the container with id 10db2a9508a9e8491eb1d5c2d8c75821a496b3fb2ac5e59a75d5023d25ec0621 Jul 10 19:01:46.905102 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:46.905079 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.001468 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.001407 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-prometheus-k8s-rulefiles-2\") pod \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " Jul 10 19:01:47.001653 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.001477 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-configmap-kubelet-serving-ca-bundle\") pod \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " Jul 10 19:01:47.001653 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.001517 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-prometheus-k8s-tls\") pod \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " Jul 10 19:01:47.001653 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.001543 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9b26709a-e61c-4a9a-8d89-5557b8f999b9-prometheus-k8s-db\") pod \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " Jul 10 19:01:47.001653 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.001563 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-config\") pod \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " Jul 10 19:01:47.001653 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.001585 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-thanos-prometheus-http-client-file\") pod \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " Jul 10 19:01:47.001904 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.001640 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-metrics-client-certs\") pod \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " Jul 10 19:01:47.001904 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.001704 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-web-config\") pod \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " Jul 10 19:01:47.001904 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.001752 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " Jul 10 19:01:47.001904 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.001780 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b26709a-e61c-4a9a-8d89-5557b8f999b9-config-out\") pod \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " Jul 10 19:01:47.001904 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.001809 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-configmap-serving-certs-ca-bundle\") pod \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " Jul 10 19:01:47.001904 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.001828 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "9b26709a-e61c-4a9a-8d89-5557b8f999b9" (UID: "9b26709a-e61c-4a9a-8d89-5557b8f999b9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:01:47.001904 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.001843 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-st8nc\" (UniqueName: \"kubernetes.io/projected/9b26709a-e61c-4a9a-8d89-5557b8f999b9-kube-api-access-st8nc\") pod \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " Jul 10 19:01:47.002254 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.001910 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b26709a-e61c-4a9a-8d89-5557b8f999b9-tls-assets\") pod \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " Jul 10 19:01:47.002254 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.001935 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-configmap-metrics-client-ca\") pod \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " Jul 10 19:01:47.002254 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.001969 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-prometheus-k8s-rulefiles-0\") pod \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " Jul 10 19:01:47.002254 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.001969 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "9b26709a-e61c-4a9a-8d89-5557b8f999b9" (UID: "9b26709a-e61c-4a9a-8d89-5557b8f999b9"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:01:47.002254 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.002004 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-grpc-tls\") pod \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " Jul 10 19:01:47.002254 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.002034 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-kube-rbac-proxy\") pod \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " Jul 10 19:01:47.002254 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.002065 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-prometheus-k8s-rulefiles-1\") pod \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " Jul 10 19:01:47.002254 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.002112 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-prometheus-trusted-ca-bundle\") pod \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " Jul 10 19:01:47.002254 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.002149 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\" (UID: \"9b26709a-e61c-4a9a-8d89-5557b8f999b9\") " Jul 10 19:01:47.002724 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.002484 2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:01:47.002724 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.002507 2560 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:01:47.002724 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.002699 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b26709a-e61c-4a9a-8d89-5557b8f999b9-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "9b26709a-e61c-4a9a-8d89-5557b8f999b9" (UID: "9b26709a-e61c-4a9a-8d89-5557b8f999b9"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:01:47.004714 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.004454 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b26709a-e61c-4a9a-8d89-5557b8f999b9-kube-api-access-st8nc" (OuterVolumeSpecName: "kube-api-access-st8nc") pod "9b26709a-e61c-4a9a-8d89-5557b8f999b9" (UID: "9b26709a-e61c-4a9a-8d89-5557b8f999b9"). InnerVolumeSpecName "kube-api-access-st8nc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:01:47.004714 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.004481 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "9b26709a-e61c-4a9a-8d89-5557b8f999b9" (UID: "9b26709a-e61c-4a9a-8d89-5557b8f999b9"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:01:47.004714 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.004514 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-config" (OuterVolumeSpecName: "config") pod "9b26709a-e61c-4a9a-8d89-5557b8f999b9" (UID: "9b26709a-e61c-4a9a-8d89-5557b8f999b9"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:01:47.004714 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.004396 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "9b26709a-e61c-4a9a-8d89-5557b8f999b9" (UID: "9b26709a-e61c-4a9a-8d89-5557b8f999b9"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:01:47.004714 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.004526 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "9b26709a-e61c-4a9a-8d89-5557b8f999b9" (UID: "9b26709a-e61c-4a9a-8d89-5557b8f999b9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:01:47.005082 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.004903 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "9b26709a-e61c-4a9a-8d89-5557b8f999b9" (UID: "9b26709a-e61c-4a9a-8d89-5557b8f999b9"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:01:47.005082 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.004905 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "9b26709a-e61c-4a9a-8d89-5557b8f999b9" (UID: "9b26709a-e61c-4a9a-8d89-5557b8f999b9"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:01:47.005082 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.004993 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "9b26709a-e61c-4a9a-8d89-5557b8f999b9" (UID: "9b26709a-e61c-4a9a-8d89-5557b8f999b9"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:01:47.005244 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.005172 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b26709a-e61c-4a9a-8d89-5557b8f999b9-config-out" (OuterVolumeSpecName: "config-out") pod "9b26709a-e61c-4a9a-8d89-5557b8f999b9" (UID: "9b26709a-e61c-4a9a-8d89-5557b8f999b9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:01:47.005474 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.005402 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "9b26709a-e61c-4a9a-8d89-5557b8f999b9" (UID: "9b26709a-e61c-4a9a-8d89-5557b8f999b9"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:01:47.005615 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.005534 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b26709a-e61c-4a9a-8d89-5557b8f999b9-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "9b26709a-e61c-4a9a-8d89-5557b8f999b9" (UID: "9b26709a-e61c-4a9a-8d89-5557b8f999b9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:01:47.005721 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.005686 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "9b26709a-e61c-4a9a-8d89-5557b8f999b9" (UID: "9b26709a-e61c-4a9a-8d89-5557b8f999b9"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:01:47.005884 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.005863 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "9b26709a-e61c-4a9a-8d89-5557b8f999b9" (UID: "9b26709a-e61c-4a9a-8d89-5557b8f999b9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:01:47.006563 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.006510 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "9b26709a-e61c-4a9a-8d89-5557b8f999b9" (UID: "9b26709a-e61c-4a9a-8d89-5557b8f999b9"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:01:47.007108 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.007092 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "9b26709a-e61c-4a9a-8d89-5557b8f999b9" (UID: "9b26709a-e61c-4a9a-8d89-5557b8f999b9"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:01:47.007171 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.007117 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "9b26709a-e61c-4a9a-8d89-5557b8f999b9" (UID: "9b26709a-e61c-4a9a-8d89-5557b8f999b9"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:01:47.017284 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.017261 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-web-config" (OuterVolumeSpecName: "web-config") pod "9b26709a-e61c-4a9a-8d89-5557b8f999b9" (UID: "9b26709a-e61c-4a9a-8d89-5557b8f999b9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:01:47.103291 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.103251 2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:01:47.103291 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.103282 2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b26709a-e61c-4a9a-8d89-5557b8f999b9-config-out\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:01:47.103291 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.103292 2560 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:01:47.103555 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.103305 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st8nc\" (UniqueName: \"kubernetes.io/projected/9b26709a-e61c-4a9a-8d89-5557b8f999b9-kube-api-access-st8nc\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:01:47.103555 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.103314 2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b26709a-e61c-4a9a-8d89-5557b8f999b9-tls-assets\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:01:47.103555 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.103323 2560 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-configmap-metrics-client-ca\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:01:47.103555 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.103331 2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:01:47.103555 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.103339 2560 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-grpc-tls\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:01:47.103555 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.103347 2560 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-kube-rbac-proxy\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:01:47.103555 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.103355 2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:01:47.103555 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.103365 2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b26709a-e61c-4a9a-8d89-5557b8f999b9-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:01:47.103555 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.103374 2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:01:47.103555 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.103383 2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:01:47.103555 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.103391 2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9b26709a-e61c-4a9a-8d89-5557b8f999b9-prometheus-k8s-db\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:01:47.103555 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.103402 2560 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-config\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:01:47.103555 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.103411 2560 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:01:47.103555 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.103420 2560 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-secret-metrics-client-certs\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:01:47.103555 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.103428 2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b26709a-e61c-4a9a-8d89-5557b8f999b9-web-config\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:01:47.199537 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.199505 2560 generic.go:356] "Generic (PLEG): container finished" podID="9b26709a-e61c-4a9a-8d89-5557b8f999b9" containerID="f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6" exitCode=0 Jul 10 19:01:47.199537 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.199529 2560 generic.go:356] "Generic (PLEG): container finished" podID="9b26709a-e61c-4a9a-8d89-5557b8f999b9" containerID="6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163" exitCode=0 Jul 10 19:01:47.199537 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.199537 2560 generic.go:356] "Generic (PLEG): container finished" podID="9b26709a-e61c-4a9a-8d89-5557b8f999b9" containerID="ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c" exitCode=0 Jul 10 19:01:47.199537 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.199545 2560 generic.go:356] "Generic (PLEG): container finished" podID="9b26709a-e61c-4a9a-8d89-5557b8f999b9" containerID="470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92" exitCode=0 Jul 10 19:01:47.200119 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.199551 2560 generic.go:356] "Generic (PLEG): container finished" podID="9b26709a-e61c-4a9a-8d89-5557b8f999b9" containerID="aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5" exitCode=0 Jul 10 19:01:47.200119 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.199556 2560 generic.go:356] "Generic (PLEG): container finished" podID="9b26709a-e61c-4a9a-8d89-5557b8f999b9" containerID="fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c" exitCode=0 Jul 10 19:01:47.200119 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.199597 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.200119 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.199597 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b26709a-e61c-4a9a-8d89-5557b8f999b9","Type":"ContainerDied","Data":"f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6"} Jul 10 19:01:47.200119 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.199670 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b26709a-e61c-4a9a-8d89-5557b8f999b9","Type":"ContainerDied","Data":"6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163"} Jul 10 19:01:47.200119 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.199714 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b26709a-e61c-4a9a-8d89-5557b8f999b9","Type":"ContainerDied","Data":"ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c"} Jul 10 19:01:47.200119 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.199733 2560 scope.go:117] "RemoveContainer" containerID="f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6" Jul 10 19:01:47.200119 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.199754 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b26709a-e61c-4a9a-8d89-5557b8f999b9","Type":"ContainerDied","Data":"470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92"} Jul 10 19:01:47.200119 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.199795 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b26709a-e61c-4a9a-8d89-5557b8f999b9","Type":"ContainerDied","Data":"aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5"} Jul 10 19:01:47.200119 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.199836 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b26709a-e61c-4a9a-8d89-5557b8f999b9","Type":"ContainerDied","Data":"fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c"} Jul 10 19:01:47.200119 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.199880 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b26709a-e61c-4a9a-8d89-5557b8f999b9","Type":"ContainerDied","Data":"24fcfee08fb9f56089a693b64f822ae717582eb3254690cd10126247f6d4dfd0"} Jul 10 19:01:47.200933 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.200905 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-94bd4b7b7-m4h5j" event={"ID":"5efa515b-3b7c-4c71-820e-15c9cf7617b7","Type":"ContainerStarted","Data":"10db2a9508a9e8491eb1d5c2d8c75821a496b3fb2ac5e59a75d5023d25ec0621"} Jul 10 19:01:47.207983 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.207961 2560 scope.go:117] "RemoveContainer" containerID="6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163" Jul 10 19:01:47.236262 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.236242 2560 scope.go:117] "RemoveContainer" containerID="ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c" Jul 10 19:01:47.243530 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.243506 2560 scope.go:117] "RemoveContainer" containerID="470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92" Jul 10 19:01:47.255734 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.255709 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 19:01:47.255828 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.255746 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 19:01:47.257468 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.257435 2560 scope.go:117] "RemoveContainer" containerID="aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5" Jul 10 19:01:47.264263 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.264243 2560 scope.go:117] "RemoveContainer" containerID="fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c" Jul 10 19:01:47.272278 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.272258 2560 scope.go:117] "RemoveContainer" containerID="5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23" Jul 10 19:01:47.279256 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.279227 2560 scope.go:117] "RemoveContainer" containerID="f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6" Jul 10 19:01:47.279601 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:01:47.279575 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6\": container with ID starting with f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6 not found: ID does not exist" containerID="f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6" Jul 10 19:01:47.279664 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.279613 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6"} err="failed to get container status \"f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6\": rpc error: code = NotFound desc = could not find container \"f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6\": container with ID starting with f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6 not found: ID does not exist" Jul 10 19:01:47.279664 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.279641 2560 scope.go:117] "RemoveContainer" containerID="6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163" Jul 10 19:01:47.280067 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:01:47.280044 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163\": container with ID starting with 6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163 not found: ID does not exist" containerID="6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163" Jul 10 19:01:47.280150 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.280071 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163"} err="failed to get container status \"6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163\": rpc error: code = NotFound desc = could not find container \"6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163\": container with ID starting with 6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163 not found: ID does not exist" Jul 10 19:01:47.280150 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.280107 2560 scope.go:117] "RemoveContainer" containerID="ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c" Jul 10 19:01:47.280391 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:01:47.280373 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c\": container with ID starting with ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c not found: ID does not exist" containerID="ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c" Jul 10 19:01:47.280558 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.280400 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c"} err="failed to get container status \"ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c\": rpc error: code = NotFound desc = could not find container \"ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c\": container with ID starting with ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c not found: ID does not exist" Jul 10 19:01:47.280558 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.280422 2560 scope.go:117] "RemoveContainer" containerID="470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92" Jul 10 19:01:47.280726 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:01:47.280709 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92\": container with ID starting with 470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92 not found: ID does not exist" containerID="470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92" Jul 10 19:01:47.280799 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.280734 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92"} err="failed to get container status \"470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92\": rpc error: code = NotFound desc = could not find container \"470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92\": container with ID starting with 470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92 not found: ID does not exist" Jul 10 19:01:47.280799 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.280753 2560 scope.go:117] "RemoveContainer" containerID="aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5" Jul 10 19:01:47.281006 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:01:47.280981 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5\": container with ID starting with aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5 not found: ID does not exist" containerID="aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5" Jul 10 19:01:47.281083 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.281010 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5"} err="failed to get container status \"aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5\": rpc error: code = NotFound desc = could not find container \"aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5\": container with ID starting with aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5 not found: ID does not exist" Jul 10 19:01:47.281083 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.281043 2560 scope.go:117] "RemoveContainer" containerID="fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c" Jul 10 19:01:47.281225 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.281207 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 19:01:47.281307 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:01:47.281287 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c\": container with ID starting with fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c not found: ID does not exist" containerID="fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c" Jul 10 19:01:47.281345 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.281312 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c"} err="failed to get container status \"fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c\": rpc error: code = NotFound desc = could not find container \"fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c\": container with ID starting with fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c not found: ID does not exist" Jul 10 19:01:47.281345 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.281327 2560 scope.go:117] "RemoveContainer" containerID="5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23" Jul 10 19:01:47.281613 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.281586 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b26709a-e61c-4a9a-8d89-5557b8f999b9" containerName="config-reloader" Jul 10 19:01:47.281613 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.281610 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b26709a-e61c-4a9a-8d89-5557b8f999b9" containerName="config-reloader" Jul 10 19:01:47.281728 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.281620 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b26709a-e61c-4a9a-8d89-5557b8f999b9" containerName="thanos-sidecar" Jul 10 19:01:47.281728 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.281626 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b26709a-e61c-4a9a-8d89-5557b8f999b9" containerName="thanos-sidecar" Jul 10 19:01:47.281728 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.281638 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b26709a-e61c-4a9a-8d89-5557b8f999b9" containerName="prometheus" Jul 10 19:01:47.281728 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.281644 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b26709a-e61c-4a9a-8d89-5557b8f999b9" containerName="prometheus" Jul 10 19:01:47.281728 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.281654 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b26709a-e61c-4a9a-8d89-5557b8f999b9" containerName="kube-rbac-proxy" Jul 10 19:01:47.281728 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.281659 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b26709a-e61c-4a9a-8d89-5557b8f999b9" containerName="kube-rbac-proxy" Jul 10 19:01:47.281728 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.281667 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b26709a-e61c-4a9a-8d89-5557b8f999b9" containerName="init-config-reloader" Jul 10 19:01:47.281728 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.281672 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b26709a-e61c-4a9a-8d89-5557b8f999b9" containerName="init-config-reloader" Jul 10 19:01:47.281728 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:01:47.281591 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23\": container with ID starting with 5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23 not found: ID does not exist" containerID="5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23" Jul 10 19:01:47.281728 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.281696 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23"} err="failed to get container status \"5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23\": rpc error: code = NotFound desc = could not find container \"5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23\": container with ID starting with 5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23 not found: ID does not exist" Jul 10 19:01:47.281728 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.281718 2560 scope.go:117] "RemoveContainer" containerID="f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6" Jul 10 19:01:47.282126 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.281680 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b26709a-e61c-4a9a-8d89-5557b8f999b9" containerName="kube-rbac-proxy-thanos" Jul 10 19:01:47.282126 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.281750 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b26709a-e61c-4a9a-8d89-5557b8f999b9" containerName="kube-rbac-proxy-thanos" Jul 10 19:01:47.282126 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.281763 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b26709a-e61c-4a9a-8d89-5557b8f999b9" containerName="kube-rbac-proxy-web" Jul 10 19:01:47.282126 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.281768 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b26709a-e61c-4a9a-8d89-5557b8f999b9" containerName="kube-rbac-proxy-web" Jul 10 19:01:47.282126 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.281824 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b26709a-e61c-4a9a-8d89-5557b8f999b9" containerName="config-reloader" Jul 10 19:01:47.282126 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.281835 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b26709a-e61c-4a9a-8d89-5557b8f999b9" containerName="kube-rbac-proxy-web" Jul 10 19:01:47.282126 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.281847 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b26709a-e61c-4a9a-8d89-5557b8f999b9" containerName="prometheus" Jul 10 19:01:47.282126 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.281856 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b26709a-e61c-4a9a-8d89-5557b8f999b9" containerName="kube-rbac-proxy" Jul 10 19:01:47.282126 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.281863 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b26709a-e61c-4a9a-8d89-5557b8f999b9" containerName="kube-rbac-proxy-thanos" Jul 10 19:01:47.282126 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.281868 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b26709a-e61c-4a9a-8d89-5557b8f999b9" containerName="thanos-sidecar" Jul 10 19:01:47.282126 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.281961 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6"} err="failed to get container status \"f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6\": rpc error: code = NotFound desc = could not find container \"f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6\": container with ID starting with f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6 not found: ID does not exist" Jul 10 19:01:47.282126 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.281979 2560 scope.go:117] "RemoveContainer" containerID="6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163" Jul 10 19:01:47.282545 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.282205 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163"} err="failed to get container status \"6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163\": rpc error: code = NotFound desc = could not find container \"6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163\": container with ID starting with 6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163 not found: ID does not exist" Jul 10 19:01:47.282545 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.282219 2560 scope.go:117] "RemoveContainer" containerID="ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c" Jul 10 19:01:47.282545 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.282457 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c"} err="failed to get container status \"ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c\": rpc error: code = NotFound desc = could not find container \"ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c\": container with ID starting with ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c not found: ID does not exist" Jul 10 19:01:47.282545 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.282475 2560 scope.go:117] "RemoveContainer" containerID="470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92" Jul 10 19:01:47.282716 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.282699 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92"} err="failed to get container status \"470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92\": rpc error: code = NotFound desc = could not find container \"470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92\": container with ID starting with 470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92 not found: ID does not exist" Jul 10 19:01:47.282759 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.282716 2560 scope.go:117] "RemoveContainer" containerID="aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5" Jul 10 19:01:47.283019 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.282995 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5"} err="failed to get container status \"aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5\": rpc error: code = NotFound desc = could not find container \"aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5\": container with ID starting with aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5 not found: ID does not exist" Jul 10 19:01:47.283109 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.283023 2560 scope.go:117] "RemoveContainer" containerID="fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c" Jul 10 19:01:47.283282 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.283259 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c"} err="failed to get container status \"fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c\": rpc error: code = NotFound desc = could not find container \"fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c\": container with ID starting with fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c not found: ID does not exist" Jul 10 19:01:47.283336 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.283284 2560 scope.go:117] "RemoveContainer" containerID="5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23" Jul 10 19:01:47.283609 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.283575 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23"} err="failed to get container status \"5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23\": rpc error: code = NotFound desc = could not find container \"5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23\": container with ID starting with 5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23 not found: ID does not exist" Jul 10 19:01:47.283667 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.283611 2560 scope.go:117] "RemoveContainer" containerID="f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6" Jul 10 19:01:47.283854 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.283833 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6"} err="failed to get container status \"f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6\": rpc error: code = NotFound desc = could not find container \"f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6\": container with ID starting with f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6 not found: ID does not exist" Jul 10 19:01:47.283854 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.283853 2560 scope.go:117] "RemoveContainer" containerID="6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163" Jul 10 19:01:47.284088 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.284068 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163"} err="failed to get container status \"6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163\": rpc error: code = NotFound desc = could not find container \"6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163\": container with ID starting with 6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163 not found: ID does not exist" Jul 10 19:01:47.284142 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.284089 2560 scope.go:117] "RemoveContainer" containerID="ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c" Jul 10 19:01:47.284301 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.284284 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c"} err="failed to get container status \"ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c\": rpc error: code = NotFound desc = could not find container \"ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c\": container with ID starting with ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c not found: ID does not exist" Jul 10 19:01:47.284352 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.284302 2560 scope.go:117] "RemoveContainer" containerID="470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92" Jul 10 19:01:47.284548 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.284526 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92"} err="failed to get container status \"470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92\": rpc error: code = NotFound desc = could not find container \"470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92\": container with ID starting with 470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92 not found: ID does not exist" Jul 10 19:01:47.284588 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.284549 2560 scope.go:117] "RemoveContainer" containerID="aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5" Jul 10 19:01:47.284768 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.284753 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5"} err="failed to get container status \"aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5\": rpc error: code = NotFound desc = could not find container \"aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5\": container with ID starting with aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5 not found: ID does not exist" Jul 10 19:01:47.284816 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.284769 2560 scope.go:117] "RemoveContainer" containerID="fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c" Jul 10 19:01:47.284979 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.284963 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c"} err="failed to get container status \"fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c\": rpc error: code = NotFound desc = could not find container \"fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c\": container with ID starting with fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c not found: ID does not exist" Jul 10 19:01:47.285035 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.284979 2560 scope.go:117] "RemoveContainer" containerID="5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23" Jul 10 19:01:47.285196 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.285178 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23"} err="failed to get container status \"5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23\": rpc error: code = NotFound desc = could not find container \"5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23\": container with ID starting with 5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23 not found: ID does not exist" Jul 10 19:01:47.285237 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.285198 2560 scope.go:117] "RemoveContainer" containerID="f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6" Jul 10 19:01:47.285501 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.285479 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6"} err="failed to get container status \"f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6\": rpc error: code = NotFound desc = could not find container \"f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6\": container with ID starting with f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6 not found: ID does not exist" Jul 10 19:01:47.285565 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.285501 2560 scope.go:117] "RemoveContainer" containerID="6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163" Jul 10 19:01:47.285720 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.285701 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163"} err="failed to get container status \"6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163\": rpc error: code = NotFound desc = could not find container \"6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163\": container with ID starting with 6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163 not found: ID does not exist" Jul 10 19:01:47.285761 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.285722 2560 scope.go:117] "RemoveContainer" containerID="ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c" Jul 10 19:01:47.285982 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.285968 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.286072 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.285966 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c"} err="failed to get container status \"ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c\": rpc error: code = NotFound desc = could not find container \"ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c\": container with ID starting with ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c not found: ID does not exist" Jul 10 19:01:47.286072 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.286072 2560 scope.go:117] "RemoveContainer" containerID="470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92" Jul 10 19:01:47.286463 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.286346 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92"} err="failed to get container status \"470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92\": rpc error: code = NotFound desc = could not find container \"470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92\": container with ID starting with 470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92 not found: ID does not exist" Jul 10 19:01:47.286463 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.286373 2560 scope.go:117] "RemoveContainer" containerID="aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5" Jul 10 19:01:47.286631 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.286608 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5"} err="failed to get container status \"aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5\": rpc error: code = NotFound desc = could not find container \"aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5\": container with ID starting with aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5 not found: ID does not exist" Jul 10 19:01:47.286631 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.286628 2560 scope.go:117] "RemoveContainer" containerID="fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c" Jul 10 19:01:47.286857 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.286834 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c"} err="failed to get container status \"fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c\": rpc error: code = NotFound desc = could not find container \"fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c\": container with ID starting with fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c not found: ID does not exist" Jul 10 19:01:47.286901 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.286860 2560 scope.go:117] "RemoveContainer" containerID="5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23" Jul 10 19:01:47.287100 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.287075 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23"} err="failed to get container status \"5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23\": rpc error: code = NotFound desc = could not find container \"5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23\": container with ID starting with 5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23 not found: ID does not exist" Jul 10 19:01:47.287100 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.287094 2560 scope.go:117] "RemoveContainer" containerID="f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6" Jul 10 19:01:47.287303 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.287286 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6"} err="failed to get container status \"f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6\": rpc error: code = NotFound desc = could not find container \"f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6\": container with ID starting with f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6 not found: ID does not exist" Jul 10 19:01:47.287366 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.287306 2560 scope.go:117] "RemoveContainer" containerID="6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163" Jul 10 19:01:47.287581 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.287558 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163"} err="failed to get container status \"6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163\": rpc error: code = NotFound desc = could not find container \"6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163\": container with ID starting with 6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163 not found: ID does not exist" Jul 10 19:01:47.287581 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.287581 2560 scope.go:117] "RemoveContainer" containerID="ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c" Jul 10 19:01:47.287849 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.287830 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c"} err="failed to get container status \"ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c\": rpc error: code = NotFound desc = could not find container \"ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c\": container with ID starting with ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c not found: ID does not exist" Jul 10 19:01:47.287922 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.287851 2560 scope.go:117] "RemoveContainer" containerID="470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92" Jul 10 19:01:47.288107 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.288090 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92"} err="failed to get container status \"470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92\": rpc error: code = NotFound desc = could not find container \"470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92\": container with ID starting with 470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92 not found: ID does not exist" Jul 10 19:01:47.288158 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.288108 2560 scope.go:117] "RemoveContainer" containerID="aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5" Jul 10 19:01:47.288343 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.288326 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5"} err="failed to get container status \"aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5\": rpc error: code = NotFound desc = could not find container \"aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5\": container with ID starting with aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5 not found: ID does not exist" Jul 10 19:01:47.288343 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.288341 2560 scope.go:117] "RemoveContainer" containerID="fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c" Jul 10 19:01:47.288579 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.288560 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c"} err="failed to get container status \"fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c\": rpc error: code = NotFound desc = could not find container \"fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c\": container with ID starting with fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c not found: ID does not exist" Jul 10 19:01:47.288579 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.288579 2560 scope.go:117] "RemoveContainer" containerID="5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23" Jul 10 19:01:47.288799 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.288780 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23"} err="failed to get container status \"5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23\": rpc error: code = NotFound desc = could not find container \"5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23\": container with ID starting with 5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23 not found: ID does not exist" Jul 10 19:01:47.288799 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.288798 2560 scope.go:117] "RemoveContainer" containerID="f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6" Jul 10 19:01:47.289044 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.289029 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 10 19:01:47.289120 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.289055 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6"} err="failed to get container status \"f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6\": rpc error: code = NotFound desc = could not find container \"f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6\": container with ID starting with f39f24784bb7d70018234c53f10b9d4a05b598fbbff4fdb715414b8c29e945b6 not found: ID does not exist" Jul 10 19:01:47.289120 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.289078 2560 scope.go:117] "RemoveContainer" containerID="6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163" Jul 10 19:01:47.289223 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.289125 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 10 19:01:47.289223 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.289030 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 10 19:01:47.289319 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.289255 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-pjfjz\"" Jul 10 19:01:47.289369 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.289309 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163"} err="failed to get container status \"6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163\": rpc error: code = NotFound desc = could not find container \"6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163\": container with ID starting with 6124a8417c8e6a9742892506c2bcf200ba55410f37b58df3f756bbd4fd7dd163 not found: ID does not exist" Jul 10 19:01:47.289369 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.289330 2560 scope.go:117] "RemoveContainer" containerID="ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c" Jul 10 19:01:47.289496 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.289454 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 10 19:01:47.289571 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.289554 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 10 19:01:47.289782 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.289636 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c"} err="failed to get container status \"ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c\": rpc error: code = NotFound desc = could not find container \"ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c\": container with ID starting with ed70c09c1c05e9a3945dc7a558759a66d4f754a77fc7a37a4d71fb5ec947053c not found: ID does not exist" Jul 10 19:01:47.289782 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.289664 2560 scope.go:117] "RemoveContainer" containerID="470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92" Jul 10 19:01:47.289782 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.289675 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 10 19:01:47.289782 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.289651 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 10 19:01:47.289782 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.289733 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 10 19:01:47.290045 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.289919 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92"} err="failed to get container status \"470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92\": rpc error: code = NotFound desc = could not find container \"470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92\": container with ID starting with 470946317c54ca1e25081e15583230748225cb9a165b768dcf75787de41ede92 not found: ID does not exist" Jul 10 19:01:47.290045 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.289936 2560 scope.go:117] "RemoveContainer" containerID="aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5" Jul 10 19:01:47.290143 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.290132 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 10 19:01:47.290208 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.290189 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5"} err="failed to get container status \"aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5\": rpc error: code = NotFound desc = could not find container \"aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5\": container with ID starting with aff926455ffe0829b76de6379ea51c6a72cf538369b806e0cd105665a91476b5 not found: ID does not exist" Jul 10 19:01:47.290273 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.290210 2560 scope.go:117] "RemoveContainer" containerID="fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c" Jul 10 19:01:47.290412 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.290395 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 10 19:01:47.290412 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.290409 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 10 19:01:47.290561 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.290505 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c"} err="failed to get container status \"fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c\": rpc error: code = NotFound desc = could not find container \"fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c\": container with ID starting with fbc4625e09a4a56c17b9a6063d963a35cd5c210da6437a0826744f62a62e383c not found: ID does not exist" Jul 10 19:01:47.290561 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.290529 2560 scope.go:117] "RemoveContainer" containerID="5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23" Jul 10 19:01:47.290682 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.290576 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6bcp5ktqqsj24\"" Jul 10 19:01:47.290825 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.290795 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23"} err="failed to get container status \"5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23\": rpc error: code = NotFound desc = could not find container \"5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23\": container with ID starting with 5fdaa58d04e1c8e84ac37c6bca70c0e18bc671215387ad497c6498af2c6fbe23 not found: ID does not exist" Jul 10 19:01:47.290926 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.290862 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 10 19:01:47.292435 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.292416 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 10 19:01:47.295866 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.295850 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 10 19:01:47.308542 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.308506 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 19:01:47.405099 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.405061 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.405244 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.405107 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-config\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.405244 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.405130 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-web-config\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.405244 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.405176 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.405244 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.405212 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.405244 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.405232 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.405419 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.405249 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjgq4\" (UniqueName: \"kubernetes.io/projected/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-kube-api-access-hjgq4\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.405419 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.405271 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.405419 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.405305 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.405419 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.405331 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.405419 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.405347 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.405419 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.405361 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.405419 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.405401 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.405632 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.405430 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-config-out\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.405632 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.405472 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.405632 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.405502 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.405632 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.405537 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.405632 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.405565 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.405632 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.405592 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.405632 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.405607 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.505981 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.505945 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-config\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.505981 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.505981 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-web-config\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.506230 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.505998 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.506230 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.506024 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.506230 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.506049 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.506230 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.506071 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjgq4\" (UniqueName: \"kubernetes.io/projected/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-kube-api-access-hjgq4\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.506230 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.506103 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.506230 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.506147 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.506230 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.506176 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.506230 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.506203 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.506230 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.506226 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.506803 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.506251 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.506803 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.506278 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-config-out\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.506803 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.506755 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.506990 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.506963 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.507047 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.507019 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.507097 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.507042 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.507149 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.507091 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.507149 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.507134 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.507149 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.507142 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.507310 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.507171 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.507310 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.507198 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.507310 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.507248 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.507526 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.507375 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.508061 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.508034 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.508166 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.508035 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.509253 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.509224 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.509384 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.509290 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-web-config\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.509975 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.509679 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-config\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.509975 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.509899 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.509975 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.509935 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.510286 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.510262 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.510689 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.510670 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.511252 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.511230 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.511398 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.511383 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-config-out\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.511687 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.511663 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.511767 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.511748 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.512230 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.512204 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.512811 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.512793 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.514721 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.514704 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjgq4\" (UniqueName: \"kubernetes.io/projected/f21aeaa5-003b-4ba3-86b9-9a21451bc3fb-kube-api-access-hjgq4\") pod \"prometheus-k8s-0\" (UID: \"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.596675 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.596636 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:01:47.760942 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:47.760895 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 19:01:47.763074 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:01:47.763047 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf21aeaa5_003b_4ba3_86b9_9a21451bc3fb.slice/crio-24548e43e5caba7770472395d60342bb9dc9b48dcc52744bf7f3af88f04b1aca WatchSource:0}: Error finding container 24548e43e5caba7770472395d60342bb9dc9b48dcc52744bf7f3af88f04b1aca: Status 404 returned error can't find the container with id 24548e43e5caba7770472395d60342bb9dc9b48dcc52744bf7f3af88f04b1aca Jul 10 19:01:48.209834 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:48.209793 2560 generic.go:356] "Generic (PLEG): container finished" podID="f21aeaa5-003b-4ba3-86b9-9a21451bc3fb" containerID="5d2f717bedd7178988f1075ec6fd07794177b349f7c91aa54da7ef98a384ba4a" exitCode=0 Jul 10 19:01:48.210276 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:48.209838 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb","Type":"ContainerDied","Data":"5d2f717bedd7178988f1075ec6fd07794177b349f7c91aa54da7ef98a384ba4a"} Jul 10 19:01:48.210276 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:48.209928 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb","Type":"ContainerStarted","Data":"24548e43e5caba7770472395d60342bb9dc9b48dcc52744bf7f3af88f04b1aca"} Jul 10 19:01:49.218358 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:49.218021 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb","Type":"ContainerStarted","Data":"6a48a1d826371889aba16e51d225f7cf1460dcb136441a775a98872661824782"} Jul 10 19:01:49.218358 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:49.218147 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb","Type":"ContainerStarted","Data":"381ffc0376ba5d76c5f43410f2db869ec57460a4f834911883abf898ba4b68ed"} Jul 10 19:01:49.218358 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:49.218190 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb","Type":"ContainerStarted","Data":"e25f37b3da2d34c761cfd4e704d9696a572ac17ae2ede95daf73b6af16404521"} Jul 10 19:01:49.218358 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:49.218224 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb","Type":"ContainerStarted","Data":"79fd0b48f0c5c6af2e1db7eaefe269de6e42033aca207936ec4000467977b808"} Jul 10 19:01:49.218358 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:49.218260 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb","Type":"ContainerStarted","Data":"bc753b2c6cc63340a0a3834c3a73a467f1f3c6bb729579373037fb62e909f44d"} Jul 10 19:01:49.218358 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:49.218295 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f21aeaa5-003b-4ba3-86b9-9a21451bc3fb","Type":"ContainerStarted","Data":"b84631ffa9ac32503dbc7c771b83c54dc5e9cd5a514798f5eec9feb4a88cc4bb"} Jul 10 19:01:49.220023 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:49.219987 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-94bd4b7b7-m4h5j" event={"ID":"5efa515b-3b7c-4c71-820e-15c9cf7617b7","Type":"ContainerStarted","Data":"ffeee229134c2271fccb4ed5de90e82d8c83c80b0203c78e128e818111553817"} Jul 10 19:01:49.251293 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:49.251265 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b26709a-e61c-4a9a-8d89-5557b8f999b9" path="/var/lib/kubelet/pods/9b26709a-e61c-4a9a-8d89-5557b8f999b9/volumes" Jul 10 19:01:49.257746 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:49.257699 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.236216864 podStartE2EDuration="2.257683509s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.466645ms" isStatefulPod=false podCreationTimestamp="2026-07-10 19:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:49.255098588 +0000 UTC m=+200.585239422" watchObservedRunningTime="2026-07-10 19:01:49.257683509 +0000 UTC m=+200.587824319" Jul 10 19:01:50.232485 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:50.232426 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-94bd4b7b7-m4h5j" event={"ID":"5efa515b-3b7c-4c71-820e-15c9cf7617b7","Type":"ContainerStarted","Data":"d80bae37e652cc0e5da5cbbd03f5e8d479493dd7333983128beba683065c0b6a"} Jul 10 19:01:50.232941 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:50.232542 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-94bd4b7b7-m4h5j" event={"ID":"5efa515b-3b7c-4c71-820e-15c9cf7617b7","Type":"ContainerStarted","Data":"6b64a9c23ba3f7b3b7d3e8e96c101930994da8e6194d3581dffbcf8913cbf1aa"} Jul 10 19:01:50.259827 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:50.259766 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-94bd4b7b7-m4h5j" podStartSLOduration=1.982294094 podStartE2EDuration="4.259750935s" totalImagesPullingTime="2.277456841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:01:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:50.259387073 +0000 UTC m=+201.589527878" watchObservedRunningTime="2026-07-10 19:01:50.259750935 +0000 UTC m=+201.589891746" Jul 10 19:01:51.218690 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:51.218652 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:51.279421 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:51.279386 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:51.356929 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:51.356891 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:51.418812 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:51.418779 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:51.964881 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:51.964844 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:52.029090 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:52.029058 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:52.597104 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:01:52.597058 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:02:02.518564 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:02:02.518530 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:02:02.573656 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:02:02.573616 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:02:02.630381 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:02:02.630352 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:02:29.526486 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:02:29.526429 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:02:29.581301 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:02:29.581264 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:02:47.597294 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:02:47.597244 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:02:47.613571 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:02:47.613541 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:02:48.492716 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:02:48.492686 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 19:02:56.524003 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:02:56.523928 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:02:56.594795 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:02:56.594763 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:04.243320 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:03:04.243285 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:04.300714 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:03:04.300678 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:04.390000 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:03:04.389963 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:04.490056 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:03:04.490018 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:05.006367 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:03:05.006336 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:05.068002 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:03:05.067968 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:08.500934 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:03:08.500908 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:08.559699 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:03:08.559656 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:12.551915 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:03:12.551884 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:12.652456 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:03:12.652388 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:14.753096 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:03:14.753056 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:14.804726 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:03:14.804695 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:14.860743 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:03:14.860703 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:29.101311 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:03:29.101276 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l52t_2a22956a-4e96-41de-945c-a851b587a442/ovn-acl-logging/0.log" Jul 10 19:03:29.108020 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:03:29.107993 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l52t_2a22956a-4e96-41de-945c-a851b587a442/ovn-acl-logging/0.log" Jul 10 19:03:29.108967 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:03:29.108945 2560 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 19:03:52.017600 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:03:52.017565 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:52.114831 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:03:52.114800 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:13.705011 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:13.704977 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:15.531659 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:15.531622 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:17.704244 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:17.704212 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:17.808250 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:17.808210 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:19.496118 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:19.496080 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:19.592700 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:19.592658 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:19.810172 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:19.810081 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:19.915947 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:19.915914 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:22.237874 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:22.237782 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:22.433358 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:22.433290 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:24.482163 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:24.482123 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:24.577992 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:24.577952 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:24.684717 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:24.684683 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:29.841766 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:29.841708 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:29.984082 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:29.984045 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:30.075137 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:30.075100 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:30.593231 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:30.593197 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-z9zl6"] Jul 10 19:04:30.595592 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:30.595573 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-z9zl6" Jul 10 19:04:30.598508 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:30.598483 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-l5j4n\"" Jul 10 19:04:30.598632 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:30.598515 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 10 19:04:30.599798 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:30.599777 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 10 19:04:30.609394 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:30.609365 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-z9zl6"] Jul 10 19:04:30.684760 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:30.684722 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t86fh\" (UniqueName: \"kubernetes.io/projected/fcb43e91-3e64-49f4-aba6-caed184b47f5-kube-api-access-t86fh\") pod \"cert-manager-55b4d954d5-z9zl6\" (UID: \"fcb43e91-3e64-49f4-aba6-caed184b47f5\") " pod="cert-manager/cert-manager-55b4d954d5-z9zl6" Jul 10 19:04:30.684760 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:30.684766 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcb43e91-3e64-49f4-aba6-caed184b47f5-bound-sa-token\") pod \"cert-manager-55b4d954d5-z9zl6\" (UID: \"fcb43e91-3e64-49f4-aba6-caed184b47f5\") " pod="cert-manager/cert-manager-55b4d954d5-z9zl6" Jul 10 19:04:30.786193 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:30.786144 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t86fh\" (UniqueName: \"kubernetes.io/projected/fcb43e91-3e64-49f4-aba6-caed184b47f5-kube-api-access-t86fh\") pod \"cert-manager-55b4d954d5-z9zl6\" (UID: \"fcb43e91-3e64-49f4-aba6-caed184b47f5\") " pod="cert-manager/cert-manager-55b4d954d5-z9zl6" Jul 10 19:04:30.786193 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:30.786201 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcb43e91-3e64-49f4-aba6-caed184b47f5-bound-sa-token\") pod \"cert-manager-55b4d954d5-z9zl6\" (UID: \"fcb43e91-3e64-49f4-aba6-caed184b47f5\") " pod="cert-manager/cert-manager-55b4d954d5-z9zl6" Jul 10 19:04:30.796372 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:30.796342 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcb43e91-3e64-49f4-aba6-caed184b47f5-bound-sa-token\") pod \"cert-manager-55b4d954d5-z9zl6\" (UID: \"fcb43e91-3e64-49f4-aba6-caed184b47f5\") " pod="cert-manager/cert-manager-55b4d954d5-z9zl6" Jul 10 19:04:30.796940 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:30.796921 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t86fh\" (UniqueName: \"kubernetes.io/projected/fcb43e91-3e64-49f4-aba6-caed184b47f5-kube-api-access-t86fh\") pod \"cert-manager-55b4d954d5-z9zl6\" (UID: \"fcb43e91-3e64-49f4-aba6-caed184b47f5\") " pod="cert-manager/cert-manager-55b4d954d5-z9zl6" Jul 10 19:04:30.905801 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:30.905711 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-z9zl6" Jul 10 19:04:31.042819 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:31.042784 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-z9zl6"] Jul 10 19:04:31.046714 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:04:31.046685 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfcb43e91_3e64_49f4_aba6_caed184b47f5.slice/crio-fc1534d00d93d2da01e9ae69b35ad3fcf0a4b57c748b18073bbc921fc569a6a5 WatchSource:0}: Error finding container fc1534d00d93d2da01e9ae69b35ad3fcf0a4b57c748b18073bbc921fc569a6a5: Status 404 returned error can't find the container with id fc1534d00d93d2da01e9ae69b35ad3fcf0a4b57c748b18073bbc921fc569a6a5 Jul 10 19:04:31.048996 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:31.048976 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 19:04:31.909486 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:31.909432 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-z9zl6" event={"ID":"fcb43e91-3e64-49f4-aba6-caed184b47f5","Type":"ContainerStarted","Data":"fc1534d00d93d2da01e9ae69b35ad3fcf0a4b57c748b18073bbc921fc569a6a5"} Jul 10 19:04:34.926297 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:34.926259 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-z9zl6" event={"ID":"fcb43e91-3e64-49f4-aba6-caed184b47f5","Type":"ContainerStarted","Data":"0f826b46b27e06f7cd3c8befce796104ad76661f463070e155829675a8e0d061"} Jul 10 19:04:34.950321 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:34.950268 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-z9zl6" podStartSLOduration=1.695265376 podStartE2EDuration="4.950253212s" totalImagesPullingTime="3.254987836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:04:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:04:34.94994999 +0000 UTC m=+366.280090802" watchObservedRunningTime="2026-07-10 19:04:34.950253212 +0000 UTC m=+366.280394022" Jul 10 19:04:37.571371 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:37.571333 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:37.679574 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:37.679540 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:37.772972 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:37.772904 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:39.866609 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:39.866578 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:56.087969 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:56.087935 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:56.195295 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:56.195259 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:56.881883 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:56.881852 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:57.060438 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:57.060407 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:57.166782 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:57.166749 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:57.228666 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:57.228629 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-kwh88"] Jul 10 19:04:57.232814 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:57.232781 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-kwh88" Jul 10 19:04:57.236293 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:04:57.236255 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-manager-dockercfg-kshvj\" is forbidden: User \"system:node:ip-10-0-138-213.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-213.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-kshvj\"" type="*v1.Secret" Jul 10 19:04:57.236496 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:04:57.236468 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-138-213.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-213.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 10 19:04:57.236560 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:04:57.236541 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-138-213.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-213.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 10 19:04:57.239088 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:04:57.239046 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-manager-service-cert\" is forbidden: User \"system:node:ip-10-0-138-213.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-213.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret" Jul 10 19:04:57.239360 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:04:57.239299 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"opendatahub-operator-controller-manager-7cfcf59dd-kwh88\" is forbidden: User \"system:node:ip-10-0-138-213.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-213.ec2.internal' and this object" podUID="e004b747-8a02-46be-93aa-0d77e56ab69a" pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-kwh88" Jul 10 19:04:57.242755 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:57.242735 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 19:04:57.255022 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:57.254996 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-kwh88"] Jul 10 19:04:57.318378 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:57.318343 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e004b747-8a02-46be-93aa-0d77e56ab69a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cfcf59dd-kwh88\" (UID: \"e004b747-8a02-46be-93aa-0d77e56ab69a\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-kwh88" Jul 10 19:04:57.318603 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:57.318421 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e004b747-8a02-46be-93aa-0d77e56ab69a-tmp\") pod \"opendatahub-operator-controller-manager-7cfcf59dd-kwh88\" (UID: \"e004b747-8a02-46be-93aa-0d77e56ab69a\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-kwh88" Jul 10 19:04:57.318658 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:57.318596 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e004b747-8a02-46be-93aa-0d77e56ab69a-manifests\") pod \"opendatahub-operator-controller-manager-7cfcf59dd-kwh88\" (UID: \"e004b747-8a02-46be-93aa-0d77e56ab69a\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-kwh88" Jul 10 19:04:57.318711 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:57.318653 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e004b747-8a02-46be-93aa-0d77e56ab69a-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cfcf59dd-kwh88\" (UID: \"e004b747-8a02-46be-93aa-0d77e56ab69a\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-kwh88" Jul 10 19:04:57.318711 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:57.318692 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st7qz\" (UniqueName: \"kubernetes.io/projected/e004b747-8a02-46be-93aa-0d77e56ab69a-kube-api-access-st7qz\") pod \"opendatahub-operator-controller-manager-7cfcf59dd-kwh88\" (UID: \"e004b747-8a02-46be-93aa-0d77e56ab69a\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-kwh88" Jul 10 19:04:57.408060 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:57.408015 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:57.419739 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:57.419623 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e004b747-8a02-46be-93aa-0d77e56ab69a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cfcf59dd-kwh88\" (UID: \"e004b747-8a02-46be-93aa-0d77e56ab69a\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-kwh88" Jul 10 19:04:57.419739 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:57.419710 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e004b747-8a02-46be-93aa-0d77e56ab69a-tmp\") pod \"opendatahub-operator-controller-manager-7cfcf59dd-kwh88\" (UID: \"e004b747-8a02-46be-93aa-0d77e56ab69a\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-kwh88" Jul 10 19:04:57.420038 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:57.419742 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e004b747-8a02-46be-93aa-0d77e56ab69a-manifests\") pod \"opendatahub-operator-controller-manager-7cfcf59dd-kwh88\" (UID: \"e004b747-8a02-46be-93aa-0d77e56ab69a\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-kwh88" Jul 10 19:04:57.420038 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:57.419780 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e004b747-8a02-46be-93aa-0d77e56ab69a-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cfcf59dd-kwh88\" (UID: \"e004b747-8a02-46be-93aa-0d77e56ab69a\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-kwh88" Jul 10 19:04:57.420038 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:57.419812 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st7qz\" (UniqueName: \"kubernetes.io/projected/e004b747-8a02-46be-93aa-0d77e56ab69a-kube-api-access-st7qz\") pod \"opendatahub-operator-controller-manager-7cfcf59dd-kwh88\" (UID: \"e004b747-8a02-46be-93aa-0d77e56ab69a\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-kwh88" Jul 10 19:04:57.420220 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:57.420198 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e004b747-8a02-46be-93aa-0d77e56ab69a-manifests\") pod \"opendatahub-operator-controller-manager-7cfcf59dd-kwh88\" (UID: \"e004b747-8a02-46be-93aa-0d77e56ab69a\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-kwh88" Jul 10 19:04:57.422124 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:57.422100 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e004b747-8a02-46be-93aa-0d77e56ab69a-tmp\") pod \"opendatahub-operator-controller-manager-7cfcf59dd-kwh88\" (UID: \"e004b747-8a02-46be-93aa-0d77e56ab69a\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-kwh88" Jul 10 19:04:57.504879 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:57.504844 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:57.602734 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:57.602705 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:57.673653 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:57.673568 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:57.747905 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:57.747833 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:57.847764 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:57.847730 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:58.270122 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:58.270087 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-kshvj\"" Jul 10 19:04:58.420049 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:04:58.420009 2560 secret.go:189] Couldn't get secret opendatahub/opendatahub-operator-controller-manager-service-cert: failed to sync secret cache: timed out waiting for the condition Jul 10 19:04:58.420236 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:04:58.420015 2560 secret.go:189] Couldn't get secret opendatahub/opendatahub-operator-controller-manager-service-cert: failed to sync secret cache: timed out waiting for the condition Jul 10 19:04:58.420236 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:04:58.420124 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e004b747-8a02-46be-93aa-0d77e56ab69a-apiservice-cert podName:e004b747-8a02-46be-93aa-0d77e56ab69a nodeName:}" failed. No retries permitted until 2026-07-10 19:04:58.920099988 +0000 UTC m=+390.250240782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "apiservice-cert" (UniqueName: "kubernetes.io/secret/e004b747-8a02-46be-93aa-0d77e56ab69a-apiservice-cert") pod "opendatahub-operator-controller-manager-7cfcf59dd-kwh88" (UID: "e004b747-8a02-46be-93aa-0d77e56ab69a") : failed to sync secret cache: timed out waiting for the condition Jul 10 19:04:58.420236 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:04:58.420193 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e004b747-8a02-46be-93aa-0d77e56ab69a-webhook-cert podName:e004b747-8a02-46be-93aa-0d77e56ab69a nodeName:}" failed. No retries permitted until 2026-07-10 19:04:58.92017443 +0000 UTC m=+390.250315219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/e004b747-8a02-46be-93aa-0d77e56ab69a-webhook-cert") pod "opendatahub-operator-controller-manager-7cfcf59dd-kwh88" (UID: "e004b747-8a02-46be-93aa-0d77e56ab69a") : failed to sync secret cache: timed out waiting for the condition Jul 10 19:04:58.428017 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:04:58.427992 2560 projected.go:291] Couldn't get configMap opendatahub/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition Jul 10 19:04:58.474583 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:58.474550 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 19:04:58.494036 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:58.494005 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 19:04:58.498920 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:04:58.498899 2560 projected.go:196] Error preparing data for projected volume kube-api-access-st7qz for pod opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-kwh88: failed to sync configmap cache: timed out waiting for the condition Jul 10 19:04:58.499045 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:04:58.498961 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e004b747-8a02-46be-93aa-0d77e56ab69a-kube-api-access-st7qz podName:e004b747-8a02-46be-93aa-0d77e56ab69a nodeName:}" failed. No retries permitted until 2026-07-10 19:04:58.998946025 +0000 UTC m=+390.329086813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-st7qz" (UniqueName: "kubernetes.io/projected/e004b747-8a02-46be-93aa-0d77e56ab69a-kube-api-access-st7qz") pod "opendatahub-operator-controller-manager-7cfcf59dd-kwh88" (UID: "e004b747-8a02-46be-93aa-0d77e56ab69a") : failed to sync configmap cache: timed out waiting for the condition Jul 10 19:04:58.734753 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:58.734720 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 19:04:58.933022 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:58.932985 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e004b747-8a02-46be-93aa-0d77e56ab69a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cfcf59dd-kwh88\" (UID: \"e004b747-8a02-46be-93aa-0d77e56ab69a\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-kwh88" Jul 10 19:04:58.933201 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:58.933157 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e004b747-8a02-46be-93aa-0d77e56ab69a-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cfcf59dd-kwh88\" (UID: \"e004b747-8a02-46be-93aa-0d77e56ab69a\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-kwh88" Jul 10 19:04:58.935508 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:58.935482 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e004b747-8a02-46be-93aa-0d77e56ab69a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cfcf59dd-kwh88\" (UID: \"e004b747-8a02-46be-93aa-0d77e56ab69a\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-kwh88" Jul 10 19:04:58.935640 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:58.935619 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e004b747-8a02-46be-93aa-0d77e56ab69a-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cfcf59dd-kwh88\" (UID: \"e004b747-8a02-46be-93aa-0d77e56ab69a\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-kwh88" Jul 10 19:04:59.034403 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:59.034317 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st7qz\" (UniqueName: \"kubernetes.io/projected/e004b747-8a02-46be-93aa-0d77e56ab69a-kube-api-access-st7qz\") pod \"opendatahub-operator-controller-manager-7cfcf59dd-kwh88\" (UID: \"e004b747-8a02-46be-93aa-0d77e56ab69a\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-kwh88" Jul 10 19:04:59.036822 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:59.036793 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st7qz\" (UniqueName: \"kubernetes.io/projected/e004b747-8a02-46be-93aa-0d77e56ab69a-kube-api-access-st7qz\") pod \"opendatahub-operator-controller-manager-7cfcf59dd-kwh88\" (UID: \"e004b747-8a02-46be-93aa-0d77e56ab69a\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-kwh88" Jul 10 19:04:59.045179 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:59.045154 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-kwh88" Jul 10 19:04:59.195128 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:04:59.195102 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-kwh88"] Jul 10 19:04:59.196404 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:04:59.196375 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode004b747_8a02_46be_93aa_0d77e56ab69a.slice/crio-ab181e8e743f9f4887c11ab44f393fd102130453306e15e942c17ea9afc19387 WatchSource:0}: Error finding container ab181e8e743f9f4887c11ab44f393fd102130453306e15e942c17ea9afc19387: Status 404 returned error can't find the container with id ab181e8e743f9f4887c11ab44f393fd102130453306e15e942c17ea9afc19387 Jul 10 19:05:00.036261 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:00.036214 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-kwh88" event={"ID":"e004b747-8a02-46be-93aa-0d77e56ab69a","Type":"ContainerStarted","Data":"ab181e8e743f9f4887c11ab44f393fd102130453306e15e942c17ea9afc19387"} Jul 10 19:05:02.126694 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:02.126656 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:02.226958 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:02.226877 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:03.050961 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:03.050924 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-kwh88" event={"ID":"e004b747-8a02-46be-93aa-0d77e56ab69a","Type":"ContainerStarted","Data":"803a1c8eb71e5340d4fbe7e11f6a35833a46e8db755c6a17c187a0ae27d51725"} Jul 10 19:05:04.056224 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:04.056189 2560 generic.go:356] "Generic (PLEG): container finished" podID="e004b747-8a02-46be-93aa-0d77e56ab69a" containerID="803a1c8eb71e5340d4fbe7e11f6a35833a46e8db755c6a17c187a0ae27d51725" exitCode=0 Jul 10 19:05:04.056695 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:04.056277 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-kwh88" event={"ID":"e004b747-8a02-46be-93aa-0d77e56ab69a","Type":"ContainerDied","Data":"803a1c8eb71e5340d4fbe7e11f6a35833a46e8db755c6a17c187a0ae27d51725"} Jul 10 19:05:05.062069 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:05.062020 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-kwh88" event={"ID":"e004b747-8a02-46be-93aa-0d77e56ab69a","Type":"ContainerStarted","Data":"7633c556ff24b904ae6678f0c20bd36aac62af11f3d72cd82e56c3d51c933e8d"} Jul 10 19:05:05.062477 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:05.062168 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-kwh88" Jul 10 19:05:05.087424 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:05.087365 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-kwh88" podStartSLOduration=3.437550972 podStartE2EDuration="8.087346109s" totalImagesPullingTime="3.527450612s" totalInitContainerRuntime="1.122344525s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:04:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:05:05.085240548 +0000 UTC m=+396.415381358" watchObservedRunningTime="2026-07-10 19:05:05.087346109 +0000 UTC m=+396.417486918" Jul 10 19:05:08.796830 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:08.796793 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:08.897039 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:08.897009 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:16.068693 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:16.068654 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-kwh88" Jul 10 19:05:18.820590 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:18.820554 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:19.044344 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:19.044298 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:19.678194 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:19.678151 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:19.947347 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:19.947231 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:20.177009 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:20.176970 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:20.379410 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:20.379307 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:20.642050 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:20.641937 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:22.446108 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:22.446032 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:22.615707 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:22.615677 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:23.829481 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:23.829425 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:24.007900 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:24.007844 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:26.724261 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:26.724223 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:27.200921 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:27.200892 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b69967895-zbggq"] Jul 10 19:05:27.203299 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:27.203268 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b69967895-zbggq" Jul 10 19:05:27.209413 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:27.209391 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 10 19:05:27.209522 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:27.209429 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 10 19:05:27.214112 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:27.214096 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 10 19:05:27.214895 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:27.214876 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-ht7f4\"" Jul 10 19:05:27.214987 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:27.214879 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 10 19:05:27.214987 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:27.214908 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 10 19:05:27.241278 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:27.241251 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b69967895-zbggq"] Jul 10 19:05:27.297913 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:27.297873 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxz5t\" (UniqueName: \"kubernetes.io/projected/60a4337a-8004-43ae-b41f-62b2ab657468-kube-api-access-rxz5t\") pod \"lws-controller-manager-5b69967895-zbggq\" (UID: \"60a4337a-8004-43ae-b41f-62b2ab657468\") " pod="openshift-lws-operator/lws-controller-manager-5b69967895-zbggq" Jul 10 19:05:27.298112 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:27.297982 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/60a4337a-8004-43ae-b41f-62b2ab657468-manager-config\") pod \"lws-controller-manager-5b69967895-zbggq\" (UID: \"60a4337a-8004-43ae-b41f-62b2ab657468\") " pod="openshift-lws-operator/lws-controller-manager-5b69967895-zbggq" Jul 10 19:05:27.298198 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:27.298120 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/60a4337a-8004-43ae-b41f-62b2ab657468-metrics-cert\") pod \"lws-controller-manager-5b69967895-zbggq\" (UID: \"60a4337a-8004-43ae-b41f-62b2ab657468\") " pod="openshift-lws-operator/lws-controller-manager-5b69967895-zbggq" Jul 10 19:05:27.298198 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:27.298171 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60a4337a-8004-43ae-b41f-62b2ab657468-cert\") pod \"lws-controller-manager-5b69967895-zbggq\" (UID: \"60a4337a-8004-43ae-b41f-62b2ab657468\") " pod="openshift-lws-operator/lws-controller-manager-5b69967895-zbggq" Jul 10 19:05:27.399328 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:27.399287 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxz5t\" (UniqueName: \"kubernetes.io/projected/60a4337a-8004-43ae-b41f-62b2ab657468-kube-api-access-rxz5t\") pod \"lws-controller-manager-5b69967895-zbggq\" (UID: \"60a4337a-8004-43ae-b41f-62b2ab657468\") " pod="openshift-lws-operator/lws-controller-manager-5b69967895-zbggq" Jul 10 19:05:27.399563 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:27.399356 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/60a4337a-8004-43ae-b41f-62b2ab657468-manager-config\") pod \"lws-controller-manager-5b69967895-zbggq\" (UID: \"60a4337a-8004-43ae-b41f-62b2ab657468\") " pod="openshift-lws-operator/lws-controller-manager-5b69967895-zbggq" Jul 10 19:05:27.399563 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:27.399417 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/60a4337a-8004-43ae-b41f-62b2ab657468-metrics-cert\") pod \"lws-controller-manager-5b69967895-zbggq\" (UID: \"60a4337a-8004-43ae-b41f-62b2ab657468\") " pod="openshift-lws-operator/lws-controller-manager-5b69967895-zbggq" Jul 10 19:05:27.399563 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:27.399482 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60a4337a-8004-43ae-b41f-62b2ab657468-cert\") pod \"lws-controller-manager-5b69967895-zbggq\" (UID: \"60a4337a-8004-43ae-b41f-62b2ab657468\") " pod="openshift-lws-operator/lws-controller-manager-5b69967895-zbggq" Jul 10 19:05:27.400147 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:27.400116 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/60a4337a-8004-43ae-b41f-62b2ab657468-manager-config\") pod \"lws-controller-manager-5b69967895-zbggq\" (UID: \"60a4337a-8004-43ae-b41f-62b2ab657468\") " pod="openshift-lws-operator/lws-controller-manager-5b69967895-zbggq" Jul 10 19:05:27.402430 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:27.402397 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/60a4337a-8004-43ae-b41f-62b2ab657468-metrics-cert\") pod \"lws-controller-manager-5b69967895-zbggq\" (UID: \"60a4337a-8004-43ae-b41f-62b2ab657468\") " pod="openshift-lws-operator/lws-controller-manager-5b69967895-zbggq" Jul 10 19:05:27.402674 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:27.402560 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60a4337a-8004-43ae-b41f-62b2ab657468-cert\") pod \"lws-controller-manager-5b69967895-zbggq\" (UID: \"60a4337a-8004-43ae-b41f-62b2ab657468\") " pod="openshift-lws-operator/lws-controller-manager-5b69967895-zbggq" Jul 10 19:05:27.433550 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:27.433523 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxz5t\" (UniqueName: \"kubernetes.io/projected/60a4337a-8004-43ae-b41f-62b2ab657468-kube-api-access-rxz5t\") pod \"lws-controller-manager-5b69967895-zbggq\" (UID: \"60a4337a-8004-43ae-b41f-62b2ab657468\") " pod="openshift-lws-operator/lws-controller-manager-5b69967895-zbggq" Jul 10 19:05:27.513275 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:27.513185 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b69967895-zbggq" Jul 10 19:05:27.709403 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:27.709375 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b69967895-zbggq"] Jul 10 19:05:27.710980 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:05:27.710956 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60a4337a_8004_43ae_b41f_62b2ab657468.slice/crio-cfd829100ebc6c4fd4fb7b1389867944d2d12f4cdd6a6d60106d2b4b4104e0c0 WatchSource:0}: Error finding container cfd829100ebc6c4fd4fb7b1389867944d2d12f4cdd6a6d60106d2b4b4104e0c0: Status 404 returned error can't find the container with id cfd829100ebc6c4fd4fb7b1389867944d2d12f4cdd6a6d60106d2b4b4104e0c0 Jul 10 19:05:28.166199 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:28.166161 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b69967895-zbggq" event={"ID":"60a4337a-8004-43ae-b41f-62b2ab657468","Type":"ContainerStarted","Data":"cfd829100ebc6c4fd4fb7b1389867944d2d12f4cdd6a6d60106d2b4b4104e0c0"} Jul 10 19:05:28.588064 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:28.587990 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:28.812330 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:28.812291 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:29.939157 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:29.939119 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:29.971428 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:29.971396 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh"] Jul 10 19:05:29.975091 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:29.975070 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" Jul 10 19:05:29.980336 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:29.980278 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 19:05:29.980336 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:29.980331 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 19:05:29.980648 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:29.980627 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 19:05:29.980850 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:29.980824 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-cm6xx\"" Jul 10 19:05:29.992113 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:29.992075 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh"] Jul 10 19:05:30.021604 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:30.021565 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gzgzh\" (UID: \"a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" Jul 10 19:05:30.021785 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:30.021619 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gzgzh\" (UID: \"a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" Jul 10 19:05:30.021785 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:30.021716 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gzgzh\" (UID: \"a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" Jul 10 19:05:30.021785 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:30.021772 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gzgzh\" (UID: \"a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" Jul 10 19:05:30.021956 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:30.021843 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gzgzh\" (UID: \"a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" Jul 10 19:05:30.021956 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:30.021870 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gzgzh\" (UID: \"a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" Jul 10 19:05:30.021956 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:30.021898 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gtm5\" (UniqueName: \"kubernetes.io/projected/a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3-kube-api-access-4gtm5\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gzgzh\" (UID: \"a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" Jul 10 19:05:30.021956 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:30.021943 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gzgzh\" (UID: \"a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" Jul 10 19:05:30.022169 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:30.021978 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gzgzh\" (UID: \"a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" Jul 10 19:05:30.123304 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:30.123254 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gzgzh\" (UID: \"a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" Jul 10 19:05:30.123304 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:30.123304 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gzgzh\" (UID: \"a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" Jul 10 19:05:30.123613 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:30.123331 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gtm5\" (UniqueName: \"kubernetes.io/projected/a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3-kube-api-access-4gtm5\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gzgzh\" (UID: \"a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" Jul 10 19:05:30.123613 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:30.123364 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gzgzh\" (UID: \"a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" Jul 10 19:05:30.123613 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:30.123387 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gzgzh\" (UID: \"a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" Jul 10 19:05:30.123613 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:30.123465 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gzgzh\" (UID: \"a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" Jul 10 19:05:30.123613 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:30.123494 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gzgzh\" (UID: \"a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" Jul 10 19:05:30.123882 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:30.123731 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gzgzh\" (UID: \"a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" Jul 10 19:05:30.123882 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:30.123813 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gzgzh\" (UID: \"a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" Jul 10 19:05:30.123882 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:30.123843 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gzgzh\" (UID: \"a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" Jul 10 19:05:30.124117 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:30.123886 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gzgzh\" (UID: \"a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" Jul 10 19:05:30.124117 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:30.124101 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gzgzh\" (UID: \"a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" Jul 10 19:05:30.124233 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:30.124205 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gzgzh\" (UID: \"a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" Jul 10 19:05:30.124550 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:30.124524 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gzgzh\" (UID: \"a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" Jul 10 19:05:30.126147 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:30.126122 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gzgzh\" (UID: \"a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" Jul 10 19:05:30.126344 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:30.126320 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gzgzh\" (UID: \"a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" Jul 10 19:05:30.143397 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:30.143367 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gzgzh\" (UID: \"a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" Jul 10 19:05:30.155591 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:30.155564 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gtm5\" (UniqueName: \"kubernetes.io/projected/a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3-kube-api-access-4gtm5\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gzgzh\" (UID: \"a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" Jul 10 19:05:30.295547 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:30.288575 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" Jul 10 19:05:30.437917 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:30.437877 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:30.877216 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:30.877116 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:30.883058 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:05:30.883026 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda068c7cb_6506_42f4_bcb6_b49f0ec0d6c3.slice/crio-f40545232494bc10d526c43e6b419e0117fc2c3513efa2176c6b3be2e424d3a8 WatchSource:0}: Error finding container f40545232494bc10d526c43e6b419e0117fc2c3513efa2176c6b3be2e424d3a8: Status 404 returned error can't find the container with id f40545232494bc10d526c43e6b419e0117fc2c3513efa2176c6b3be2e424d3a8 Jul 10 19:05:30.888884 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:30.888860 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh"] Jul 10 19:05:31.181416 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:31.181371 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" event={"ID":"a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3","Type":"ContainerStarted","Data":"f40545232494bc10d526c43e6b419e0117fc2c3513efa2176c6b3be2e424d3a8"} Jul 10 19:05:31.182802 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:31.182777 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b69967895-zbggq" event={"ID":"60a4337a-8004-43ae-b41f-62b2ab657468","Type":"ContainerStarted","Data":"998a50b2d25cb19ff6827aaddbc90f48c49f639e1ab37dbac1fd87e0e61e0736"} Jul 10 19:05:31.182992 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:31.182975 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5b69967895-zbggq" Jul 10 19:05:31.214820 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:31.214771 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5b69967895-zbggq" podStartSLOduration=1.257388828 podStartE2EDuration="4.214754622s" totalImagesPullingTime="2.957365794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:05:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:05:31.213666049 +0000 UTC m=+422.543806880" watchObservedRunningTime="2026-07-10 19:05:31.214754622 +0000 UTC m=+422.544895432" Jul 10 19:05:31.276276 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:31.276240 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-pr25j"] Jul 10 19:05:31.278787 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:31.278771 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-pr25j" Jul 10 19:05:31.282287 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:31.282265 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 10 19:05:31.282287 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:31.282272 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-c8d7b\"" Jul 10 19:05:31.282680 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:31.282666 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 10 19:05:31.282776 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:31.282755 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 10 19:05:31.309173 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:31.309148 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-pr25j"] Jul 10 19:05:31.335930 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:31.335891 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fb9e1aa3-6f16-42eb-a72d-aefe00037d58-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-pr25j\" (UID: \"fb9e1aa3-6f16-42eb-a72d-aefe00037d58\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-pr25j" Jul 10 19:05:31.335930 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:31.335934 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb9e1aa3-6f16-42eb-a72d-aefe00037d58-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-pr25j\" (UID: \"fb9e1aa3-6f16-42eb-a72d-aefe00037d58\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-pr25j" Jul 10 19:05:31.336133 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:31.336026 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fb9e1aa3-6f16-42eb-a72d-aefe00037d58-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-pr25j\" (UID: \"fb9e1aa3-6f16-42eb-a72d-aefe00037d58\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-pr25j" Jul 10 19:05:31.336133 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:31.336070 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fb9e1aa3-6f16-42eb-a72d-aefe00037d58-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-pr25j\" (UID: \"fb9e1aa3-6f16-42eb-a72d-aefe00037d58\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-pr25j" Jul 10 19:05:31.336133 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:31.336090 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7ldv\" (UniqueName: \"kubernetes.io/projected/fb9e1aa3-6f16-42eb-a72d-aefe00037d58-kube-api-access-d7ldv\") pod \"istiod-openshift-gateway-94bb8fbfd-pr25j\" (UID: \"fb9e1aa3-6f16-42eb-a72d-aefe00037d58\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-pr25j" Jul 10 19:05:31.336133 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:31.336119 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fb9e1aa3-6f16-42eb-a72d-aefe00037d58-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-pr25j\" (UID: \"fb9e1aa3-6f16-42eb-a72d-aefe00037d58\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-pr25j" Jul 10 19:05:31.336258 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:31.336185 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb9e1aa3-6f16-42eb-a72d-aefe00037d58-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-pr25j\" (UID: \"fb9e1aa3-6f16-42eb-a72d-aefe00037d58\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-pr25j" Jul 10 19:05:31.437563 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:31.437464 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fb9e1aa3-6f16-42eb-a72d-aefe00037d58-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-pr25j\" (UID: \"fb9e1aa3-6f16-42eb-a72d-aefe00037d58\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-pr25j" Jul 10 19:05:31.437563 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:31.437512 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7ldv\" (UniqueName: \"kubernetes.io/projected/fb9e1aa3-6f16-42eb-a72d-aefe00037d58-kube-api-access-d7ldv\") pod \"istiod-openshift-gateway-94bb8fbfd-pr25j\" (UID: \"fb9e1aa3-6f16-42eb-a72d-aefe00037d58\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-pr25j" Jul 10 19:05:31.437563 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:31.437556 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fb9e1aa3-6f16-42eb-a72d-aefe00037d58-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-pr25j\" (UID: \"fb9e1aa3-6f16-42eb-a72d-aefe00037d58\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-pr25j" Jul 10 19:05:31.437847 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:31.437584 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb9e1aa3-6f16-42eb-a72d-aefe00037d58-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-pr25j\" (UID: \"fb9e1aa3-6f16-42eb-a72d-aefe00037d58\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-pr25j" Jul 10 19:05:31.437847 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:31.437629 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fb9e1aa3-6f16-42eb-a72d-aefe00037d58-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-pr25j\" (UID: \"fb9e1aa3-6f16-42eb-a72d-aefe00037d58\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-pr25j" Jul 10 19:05:31.437847 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:31.437663 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb9e1aa3-6f16-42eb-a72d-aefe00037d58-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-pr25j\" (UID: \"fb9e1aa3-6f16-42eb-a72d-aefe00037d58\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-pr25j" Jul 10 19:05:31.437847 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:31.437742 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fb9e1aa3-6f16-42eb-a72d-aefe00037d58-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-pr25j\" (UID: \"fb9e1aa3-6f16-42eb-a72d-aefe00037d58\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-pr25j" Jul 10 19:05:31.438762 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:31.438732 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fb9e1aa3-6f16-42eb-a72d-aefe00037d58-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-pr25j\" (UID: \"fb9e1aa3-6f16-42eb-a72d-aefe00037d58\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-pr25j" Jul 10 19:05:31.440652 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:31.440587 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb9e1aa3-6f16-42eb-a72d-aefe00037d58-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-pr25j\" (UID: \"fb9e1aa3-6f16-42eb-a72d-aefe00037d58\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-pr25j" Jul 10 19:05:31.440877 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:31.440858 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fb9e1aa3-6f16-42eb-a72d-aefe00037d58-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-pr25j\" (UID: \"fb9e1aa3-6f16-42eb-a72d-aefe00037d58\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-pr25j" Jul 10 19:05:31.440941 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:31.440876 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fb9e1aa3-6f16-42eb-a72d-aefe00037d58-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-pr25j\" (UID: \"fb9e1aa3-6f16-42eb-a72d-aefe00037d58\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-pr25j" Jul 10 19:05:31.440941 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:31.440924 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fb9e1aa3-6f16-42eb-a72d-aefe00037d58-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-pr25j\" (UID: \"fb9e1aa3-6f16-42eb-a72d-aefe00037d58\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-pr25j" Jul 10 19:05:31.448103 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:31.448078 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb9e1aa3-6f16-42eb-a72d-aefe00037d58-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-pr25j\" (UID: \"fb9e1aa3-6f16-42eb-a72d-aefe00037d58\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-pr25j" Jul 10 19:05:31.452790 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:31.452743 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7ldv\" (UniqueName: \"kubernetes.io/projected/fb9e1aa3-6f16-42eb-a72d-aefe00037d58-kube-api-access-d7ldv\") pod \"istiod-openshift-gateway-94bb8fbfd-pr25j\" (UID: \"fb9e1aa3-6f16-42eb-a72d-aefe00037d58\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-pr25j" Jul 10 19:05:31.588656 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:31.588613 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-pr25j" Jul 10 19:05:31.772081 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:31.772049 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-pr25j"] Jul 10 19:05:31.821127 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:05:31.821083 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb9e1aa3_6f16_42eb_a72d_aefe00037d58.slice/crio-5c6d94db2eacf3ef1d8ee70633993db0803e1de2d6b66d68bfdcc81ef7219bc0 WatchSource:0}: Error finding container 5c6d94db2eacf3ef1d8ee70633993db0803e1de2d6b66d68bfdcc81ef7219bc0: Status 404 returned error can't find the container with id 5c6d94db2eacf3ef1d8ee70633993db0803e1de2d6b66d68bfdcc81ef7219bc0 Jul 10 19:05:32.191781 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:32.191728 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-pr25j" event={"ID":"fb9e1aa3-6f16-42eb-a72d-aefe00037d58","Type":"ContainerStarted","Data":"5c6d94db2eacf3ef1d8ee70633993db0803e1de2d6b66d68bfdcc81ef7219bc0"} Jul 10 19:05:34.497095 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:34.497043 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:34.613289 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:34.613244 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:34.697293 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:34.697228 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:34.962320 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:34.962280 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:35.210154 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:35.210123 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:35.733417 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:35.733372 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:05:35.733705 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:35.733478 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:05:35.739523 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:35.739501 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:05:35.739608 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:35.739560 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:05:35.739608 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:35.739589 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:05:36.216602 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:36.216560 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-pr25j" event={"ID":"fb9e1aa3-6f16-42eb-a72d-aefe00037d58","Type":"ContainerStarted","Data":"53401783ca816e3dce2d0b50fc54290d363afd1759acdc824a349dbd8bc1be74"} Jul 10 19:05:36.216873 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:36.216847 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-pr25j" Jul 10 19:05:36.218746 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:36.218715 2560 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-pr25j container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 19:05:36.218862 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:36.218768 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-pr25j" podUID="fb9e1aa3-6f16-42eb-a72d-aefe00037d58" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 19:05:36.219275 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:36.219244 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" event={"ID":"a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3","Type":"ContainerStarted","Data":"3c98edffd8f30dd87f3a4d84909e88217b7eaaab1068fa10b8edb914e6da0b89"} Jul 10 19:05:36.265938 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:36.265870 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-pr25j" podStartSLOduration=1.3559629530000001 podStartE2EDuration="5.265848909s" totalImagesPullingTime="3.909885956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:05:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:05:36.262975383 +0000 UTC m=+427.593116229" watchObservedRunningTime="2026-07-10 19:05:36.265848909 +0000 UTC m=+427.595989765" Jul 10 19:05:36.289739 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:36.289698 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" Jul 10 19:05:36.291083 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:36.291055 2560 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" start-of-body= Jul 10 19:05:36.291249 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:36.291117 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" podUID="a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" Jul 10 19:05:36.303799 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:36.303742 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" podStartSLOduration=2.449782839 podStartE2EDuration="7.30372082s" totalImagesPullingTime="4.853937981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:05:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:05:36.301947443 +0000 UTC m=+427.632088286" watchObservedRunningTime="2026-07-10 19:05:36.30372082 +0000 UTC m=+427.633861631" Jul 10 19:05:36.782243 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:36.782212 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:36.873391 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:36.873359 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:36.976602 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:36.976571 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:37.225298 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:37.225262 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-pr25j" Jul 10 19:05:37.294057 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:37.294016 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" Jul 10 19:05:38.229025 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:38.228994 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" Jul 10 19:05:38.230076 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:38.230049 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gzgzh" Jul 10 19:05:42.194007 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:05:42.193974 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5b69967895-zbggq" Jul 10 19:06:00.299073 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:06:00.298994 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:00.366546 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:06:00.366510 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:20.003114 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:06:20.003077 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:20.094928 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:06:20.094880 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:20.261233 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:06:20.261143 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:20.772664 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:06:20.772595 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:20.838283 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:06:20.838252 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:22.726662 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:06:22.726628 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:22.790943 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:06:22.790912 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:43.211503 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:06:43.211436 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:46.656655 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:06:46.656623 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:46.770880 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:06:46.770839 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:52.022088 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:06:52.022048 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:52.239670 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:06:52.239634 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:53.982294 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:06:53.982258 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:54.077013 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:06:54.076979 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:54.596315 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:06:54.596281 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:54.867046 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:06:54.866963 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:54.991637 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:06:54.991601 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:57.227918 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:06:57.227874 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:57.334502 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:06:57.334466 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:59.220147 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:06:59.220103 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:59.321865 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:06:59.321824 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:00.836872 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:00.836819 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:01.053539 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:01.053505 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:02.597790 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:02.597754 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:02.697889 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:02.697854 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:04.657466 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:04.657401 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:04.745999 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:04.745961 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:04.806794 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:04.806753 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:08.649244 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:08.649210 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:08.747041 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:08.747007 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:15.220716 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:15.220681 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:15.320200 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:15.320166 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:17.822765 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:17.822733 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:17.923813 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:17.923779 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:21.612152 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:21.612069 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:21.712138 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:21.712103 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:30.812324 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:30.812280 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:30.872218 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:30.872181 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:36.614386 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:36.614345 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:36.718663 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:36.718620 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:37.217057 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:37.217026 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:37.327583 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:37.327545 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:52.432075 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.432037 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42"] Jul 10 19:07:52.434768 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.434739 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:07:52.437583 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.437563 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4dmkb\"" Jul 10 19:07:52.446876 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.446854 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42"] Jul 10 19:07:52.459107 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.459083 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:52.554415 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.554371 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df98b8de-e87e-4a94-adb1-2daf956c61c0-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-82b42\" (UID: \"df98b8de-e87e-4a94-adb1-2daf956c61c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:07:52.554598 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.554421 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df98b8de-e87e-4a94-adb1-2daf956c61c0-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-82b42\" (UID: \"df98b8de-e87e-4a94-adb1-2daf956c61c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:07:52.554598 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.554506 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df98b8de-e87e-4a94-adb1-2daf956c61c0-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-82b42\" (UID: \"df98b8de-e87e-4a94-adb1-2daf956c61c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:07:52.554598 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.554560 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df98b8de-e87e-4a94-adb1-2daf956c61c0-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-82b42\" (UID: \"df98b8de-e87e-4a94-adb1-2daf956c61c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:07:52.554598 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.554589 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df98b8de-e87e-4a94-adb1-2daf956c61c0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-82b42\" (UID: \"df98b8de-e87e-4a94-adb1-2daf956c61c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:07:52.554753 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.554618 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df98b8de-e87e-4a94-adb1-2daf956c61c0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-82b42\" (UID: \"df98b8de-e87e-4a94-adb1-2daf956c61c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:07:52.554753 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.554691 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df98b8de-e87e-4a94-adb1-2daf956c61c0-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-82b42\" (UID: \"df98b8de-e87e-4a94-adb1-2daf956c61c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:07:52.554829 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.554768 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df98b8de-e87e-4a94-adb1-2daf956c61c0-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-82b42\" (UID: \"df98b8de-e87e-4a94-adb1-2daf956c61c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:07:52.554829 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.554799 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jhjh\" (UniqueName: \"kubernetes.io/projected/df98b8de-e87e-4a94-adb1-2daf956c61c0-kube-api-access-4jhjh\") pod \"maas-default-gateway-openshift-default-56654dcf7-82b42\" (UID: \"df98b8de-e87e-4a94-adb1-2daf956c61c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:07:52.585588 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.585554 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:52.655724 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.655690 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df98b8de-e87e-4a94-adb1-2daf956c61c0-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-82b42\" (UID: \"df98b8de-e87e-4a94-adb1-2daf956c61c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:07:52.655918 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.655748 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df98b8de-e87e-4a94-adb1-2daf956c61c0-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-82b42\" (UID: \"df98b8de-e87e-4a94-adb1-2daf956c61c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:07:52.655918 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.655767 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jhjh\" (UniqueName: \"kubernetes.io/projected/df98b8de-e87e-4a94-adb1-2daf956c61c0-kube-api-access-4jhjh\") pod \"maas-default-gateway-openshift-default-56654dcf7-82b42\" (UID: \"df98b8de-e87e-4a94-adb1-2daf956c61c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:07:52.656049 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.655906 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df98b8de-e87e-4a94-adb1-2daf956c61c0-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-82b42\" (UID: \"df98b8de-e87e-4a94-adb1-2daf956c61c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:07:52.656049 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.655952 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df98b8de-e87e-4a94-adb1-2daf956c61c0-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-82b42\" (UID: \"df98b8de-e87e-4a94-adb1-2daf956c61c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:07:52.656049 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.656003 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df98b8de-e87e-4a94-adb1-2daf956c61c0-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-82b42\" (UID: \"df98b8de-e87e-4a94-adb1-2daf956c61c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:07:52.656049 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.656031 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df98b8de-e87e-4a94-adb1-2daf956c61c0-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-82b42\" (UID: \"df98b8de-e87e-4a94-adb1-2daf956c61c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:07:52.656323 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.656077 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df98b8de-e87e-4a94-adb1-2daf956c61c0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-82b42\" (UID: \"df98b8de-e87e-4a94-adb1-2daf956c61c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:07:52.656323 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.656113 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df98b8de-e87e-4a94-adb1-2daf956c61c0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-82b42\" (UID: \"df98b8de-e87e-4a94-adb1-2daf956c61c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:07:52.656323 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.656179 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df98b8de-e87e-4a94-adb1-2daf956c61c0-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-82b42\" (UID: \"df98b8de-e87e-4a94-adb1-2daf956c61c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:07:52.656518 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.656319 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df98b8de-e87e-4a94-adb1-2daf956c61c0-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-82b42\" (UID: \"df98b8de-e87e-4a94-adb1-2daf956c61c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:07:52.656518 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.656386 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df98b8de-e87e-4a94-adb1-2daf956c61c0-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-82b42\" (UID: \"df98b8de-e87e-4a94-adb1-2daf956c61c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:07:52.656630 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.656511 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df98b8de-e87e-4a94-adb1-2daf956c61c0-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-82b42\" (UID: \"df98b8de-e87e-4a94-adb1-2daf956c61c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:07:52.656745 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.656727 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df98b8de-e87e-4a94-adb1-2daf956c61c0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-82b42\" (UID: \"df98b8de-e87e-4a94-adb1-2daf956c61c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:07:52.658266 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.658239 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df98b8de-e87e-4a94-adb1-2daf956c61c0-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-82b42\" (UID: \"df98b8de-e87e-4a94-adb1-2daf956c61c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:07:52.658682 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.658663 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df98b8de-e87e-4a94-adb1-2daf956c61c0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-82b42\" (UID: \"df98b8de-e87e-4a94-adb1-2daf956c61c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:07:52.665957 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.665907 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df98b8de-e87e-4a94-adb1-2daf956c61c0-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-82b42\" (UID: \"df98b8de-e87e-4a94-adb1-2daf956c61c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:07:52.666098 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.666076 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jhjh\" (UniqueName: \"kubernetes.io/projected/df98b8de-e87e-4a94-adb1-2daf956c61c0-kube-api-access-4jhjh\") pod \"maas-default-gateway-openshift-default-56654dcf7-82b42\" (UID: \"df98b8de-e87e-4a94-adb1-2daf956c61c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:07:52.747316 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.747221 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:07:52.880311 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.880285 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42"] Jul 10 19:07:52.882237 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:07:52.882211 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf98b8de_e87e_4a94_adb1_2daf956c61c0.slice/crio-94354088f2ca3bc01cff0bf81e7b47ae4b547b8fba992511b0b592d60b2af917 WatchSource:0}: Error finding container 94354088f2ca3bc01cff0bf81e7b47ae4b547b8fba992511b0b592d60b2af917: Status 404 returned error can't find the container with id 94354088f2ca3bc01cff0bf81e7b47ae4b547b8fba992511b0b592d60b2af917 Jul 10 19:07:52.884191 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.884159 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:07:52.884282 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.884219 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:07:52.884282 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:52.884246 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:07:53.857454 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:53.857413 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" event={"ID":"df98b8de-e87e-4a94-adb1-2daf956c61c0","Type":"ContainerStarted","Data":"428ab28f465e68687bf74b1ee1904c6a9428c7d55a39aa2dbd8eea849d2f739c"} Jul 10 19:07:53.857939 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:53.857493 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" event={"ID":"df98b8de-e87e-4a94-adb1-2daf956c61c0","Type":"ContainerStarted","Data":"94354088f2ca3bc01cff0bf81e7b47ae4b547b8fba992511b0b592d60b2af917"} Jul 10 19:07:53.882452 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:53.882397 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" podStartSLOduration=1.882382025 podStartE2EDuration="1.882382025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:07:53.881491738 +0000 UTC m=+565.211632569" watchObservedRunningTime="2026-07-10 19:07:53.882382025 +0000 UTC m=+565.212522835" Jul 10 19:07:54.747435 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:54.747393 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:07:54.752009 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:54.751988 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:07:54.863216 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:54.863186 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:07:54.864061 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:54.864045 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:07:54.867061 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:54.867031 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:54.971958 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:54.971891 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:55.025579 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:55.025494 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:55.699328 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:55.699293 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:55.756887 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:07:55.756858 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:15.233730 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:08:15.233693 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:16.028461 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:08:16.028415 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:16.132435 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:08:16.132404 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:16.480157 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:08:16.480129 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:28.730769 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:08:28.730735 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:28.912671 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:08:28.912629 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:29.132817 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:08:29.132738 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l52t_2a22956a-4e96-41de-945c-a851b587a442/ovn-acl-logging/0.log" Jul 10 19:08:29.142532 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:08:29.142510 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l52t_2a22956a-4e96-41de-945c-a851b587a442/ovn-acl-logging/0.log" Jul 10 19:08:29.464028 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:08:29.463992 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:29.565176 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:08:29.565141 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:34.972332 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:08:34.972296 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:35.072354 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:08:35.072316 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:35.259857 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:08:35.259761 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:38.153987 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:08:38.153952 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:38.257359 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:08:38.257327 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:07.099984 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:07.099944 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:07.155042 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:07.155005 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:10.946437 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:10.946406 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:11.041497 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:11.041428 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:11.440368 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:11.440334 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:11.640872 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:11.640840 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:11.954384 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:11.954352 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:12.016589 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:12.016548 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-lvdc7"] Jul 10 19:09:12.019434 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:12.019410 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-lvdc7" Jul 10 19:09:12.022326 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:12.022311 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 10 19:09:12.033570 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:12.033550 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-lvdc7"] Jul 10 19:09:12.041663 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:12.041637 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:12.053061 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:12.053040 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-sklxc"] Jul 10 19:09:12.055715 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:12.055700 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-sklxc" Jul 10 19:09:12.060316 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:12.060298 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-66s48\"" Jul 10 19:09:12.068056 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:12.068036 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-sklxc"] Jul 10 19:09:12.105430 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:12.105396 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpr7k\" (UniqueName: \"kubernetes.io/projected/0a6619a8-29b5-4241-bdb1-8a44bf57b1da-kube-api-access-kpr7k\") pod \"payload-pre-processing-7964457696-lvdc7\" (UID: \"0a6619a8-29b5-4241-bdb1-8a44bf57b1da\") " pod="openshift-ingress/payload-pre-processing-7964457696-lvdc7" Jul 10 19:09:12.105596 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:12.105532 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a6619a8-29b5-4241-bdb1-8a44bf57b1da-plugins-config-volume\") pod \"payload-pre-processing-7964457696-lvdc7\" (UID: \"0a6619a8-29b5-4241-bdb1-8a44bf57b1da\") " pod="openshift-ingress/payload-pre-processing-7964457696-lvdc7" Jul 10 19:09:12.206522 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:12.206412 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhz2h\" (UniqueName: \"kubernetes.io/projected/ef212931-5043-4f3a-804e-d2b071ee7a14-kube-api-access-fhz2h\") pod \"payload-processing-58f86c7798-sklxc\" (UID: \"ef212931-5043-4f3a-804e-d2b071ee7a14\") " pod="openshift-ingress/payload-processing-58f86c7798-sklxc" Jul 10 19:09:12.206522 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:12.206494 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a6619a8-29b5-4241-bdb1-8a44bf57b1da-plugins-config-volume\") pod \"payload-pre-processing-7964457696-lvdc7\" (UID: \"0a6619a8-29b5-4241-bdb1-8a44bf57b1da\") " pod="openshift-ingress/payload-pre-processing-7964457696-lvdc7" Jul 10 19:09:12.206734 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:12.206534 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpr7k\" (UniqueName: \"kubernetes.io/projected/0a6619a8-29b5-4241-bdb1-8a44bf57b1da-kube-api-access-kpr7k\") pod \"payload-pre-processing-7964457696-lvdc7\" (UID: \"0a6619a8-29b5-4241-bdb1-8a44bf57b1da\") " pod="openshift-ingress/payload-pre-processing-7964457696-lvdc7" Jul 10 19:09:12.206734 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:12.206562 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef212931-5043-4f3a-804e-d2b071ee7a14-plugins-config-volume\") pod \"payload-processing-58f86c7798-sklxc\" (UID: \"ef212931-5043-4f3a-804e-d2b071ee7a14\") " pod="openshift-ingress/payload-processing-58f86c7798-sklxc" Jul 10 19:09:12.207114 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:12.207094 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a6619a8-29b5-4241-bdb1-8a44bf57b1da-plugins-config-volume\") pod \"payload-pre-processing-7964457696-lvdc7\" (UID: \"0a6619a8-29b5-4241-bdb1-8a44bf57b1da\") " pod="openshift-ingress/payload-pre-processing-7964457696-lvdc7" Jul 10 19:09:12.215674 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:12.215651 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpr7k\" (UniqueName: \"kubernetes.io/projected/0a6619a8-29b5-4241-bdb1-8a44bf57b1da-kube-api-access-kpr7k\") pod \"payload-pre-processing-7964457696-lvdc7\" (UID: \"0a6619a8-29b5-4241-bdb1-8a44bf57b1da\") " pod="openshift-ingress/payload-pre-processing-7964457696-lvdc7" Jul 10 19:09:12.307962 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:12.307923 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhz2h\" (UniqueName: \"kubernetes.io/projected/ef212931-5043-4f3a-804e-d2b071ee7a14-kube-api-access-fhz2h\") pod \"payload-processing-58f86c7798-sklxc\" (UID: \"ef212931-5043-4f3a-804e-d2b071ee7a14\") " pod="openshift-ingress/payload-processing-58f86c7798-sklxc" Jul 10 19:09:12.308159 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:12.307986 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef212931-5043-4f3a-804e-d2b071ee7a14-plugins-config-volume\") pod \"payload-processing-58f86c7798-sklxc\" (UID: \"ef212931-5043-4f3a-804e-d2b071ee7a14\") " pod="openshift-ingress/payload-processing-58f86c7798-sklxc" Jul 10 19:09:12.308780 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:12.308760 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef212931-5043-4f3a-804e-d2b071ee7a14-plugins-config-volume\") pod \"payload-processing-58f86c7798-sklxc\" (UID: \"ef212931-5043-4f3a-804e-d2b071ee7a14\") " pod="openshift-ingress/payload-processing-58f86c7798-sklxc" Jul 10 19:09:12.317712 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:12.317690 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhz2h\" (UniqueName: \"kubernetes.io/projected/ef212931-5043-4f3a-804e-d2b071ee7a14-kube-api-access-fhz2h\") pod \"payload-processing-58f86c7798-sklxc\" (UID: \"ef212931-5043-4f3a-804e-d2b071ee7a14\") " pod="openshift-ingress/payload-processing-58f86c7798-sklxc" Jul 10 19:09:12.329673 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:12.329652 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-lvdc7" Jul 10 19:09:12.365763 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:12.365727 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-sklxc" Jul 10 19:09:12.480758 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:12.480734 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-lvdc7"] Jul 10 19:09:12.481956 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:09:12.481931 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a6619a8_29b5_4241_bdb1_8a44bf57b1da.slice/crio-38158f08a19a1b1179b63fc6af4ae62571d0e8d2d7946603ff135be4f22c33c9 WatchSource:0}: Error finding container 38158f08a19a1b1179b63fc6af4ae62571d0e8d2d7946603ff135be4f22c33c9: Status 404 returned error can't find the container with id 38158f08a19a1b1179b63fc6af4ae62571d0e8d2d7946603ff135be4f22c33c9 Jul 10 19:09:12.511886 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:12.511862 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-sklxc"] Jul 10 19:09:12.513405 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:09:12.513379 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef212931_5043_4f3a_804e_d2b071ee7a14.slice/crio-00e37032dd86187aee6925d931a8b141398e6037c3b6cccd7acf2875f6a1fc29 WatchSource:0}: Error finding container 00e37032dd86187aee6925d931a8b141398e6037c3b6cccd7acf2875f6a1fc29: Status 404 returned error can't find the container with id 00e37032dd86187aee6925d931a8b141398e6037c3b6cccd7acf2875f6a1fc29 Jul 10 19:09:13.254813 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:13.254785 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-lvdc7" event={"ID":"0a6619a8-29b5-4241-bdb1-8a44bf57b1da","Type":"ContainerStarted","Data":"38158f08a19a1b1179b63fc6af4ae62571d0e8d2d7946603ff135be4f22c33c9"} Jul 10 19:09:13.255221 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:13.254913 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-sklxc" event={"ID":"ef212931-5043-4f3a-804e-d2b071ee7a14","Type":"ContainerStarted","Data":"00e37032dd86187aee6925d931a8b141398e6037c3b6cccd7acf2875f6a1fc29"} Jul 10 19:09:14.261427 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:14.261377 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-lvdc7" event={"ID":"0a6619a8-29b5-4241-bdb1-8a44bf57b1da","Type":"ContainerStarted","Data":"39b0e351e7d1dfd3a837c7bd029d07042d9e8c950d78d2077f64b67f1bc34cca"} Jul 10 19:09:14.261889 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:14.261462 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-lvdc7" Jul 10 19:09:14.262925 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:14.262902 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-sklxc" event={"ID":"ef212931-5043-4f3a-804e-d2b071ee7a14","Type":"ContainerStarted","Data":"f66426799c2bdd5b18719b2184b6e872771051d849666fcb88e87c5a3644b3ec"} Jul 10 19:09:14.263063 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:14.263042 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-sklxc" Jul 10 19:09:14.279818 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:14.279774 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-lvdc7" podStartSLOduration=1.951983305 podStartE2EDuration="3.27975865s" totalImagesPullingTime="1.327775345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:09:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:09:14.278109111 +0000 UTC m=+645.608249949" watchObservedRunningTime="2026-07-10 19:09:14.27975865 +0000 UTC m=+645.609899460" Jul 10 19:09:14.294834 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:14.294784 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-sklxc" podStartSLOduration=0.995497438 podStartE2EDuration="2.294769825s" totalImagesPullingTime="1.299272387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:09:14.293391255 +0000 UTC m=+645.623532087" watchObservedRunningTime="2026-07-10 19:09:14.294769825 +0000 UTC m=+645.624910894" Jul 10 19:09:20.293236 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:20.293201 2560 generic.go:356] "Generic (PLEG): container finished" podID="df98b8de-e87e-4a94-adb1-2daf956c61c0" containerID="428ab28f465e68687bf74b1ee1904c6a9428c7d55a39aa2dbd8eea849d2f739c" exitCode=0 Jul 10 19:09:20.293717 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:20.293255 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" event={"ID":"df98b8de-e87e-4a94-adb1-2daf956c61c0","Type":"ContainerDied","Data":"428ab28f465e68687bf74b1ee1904c6a9428c7d55a39aa2dbd8eea849d2f739c"} Jul 10 19:09:20.293717 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:20.293658 2560 scope.go:117] "RemoveContainer" containerID="428ab28f465e68687bf74b1ee1904c6a9428c7d55a39aa2dbd8eea849d2f739c" Jul 10 19:09:20.294649 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:20.294621 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:09:20.294723 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:20.294691 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:09:20.294768 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:20.294721 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:09:21.299750 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:21.299713 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" event={"ID":"df98b8de-e87e-4a94-adb1-2daf956c61c0","Type":"ContainerStarted","Data":"674f486f2b383ec6d03788caa1f4acc9180776f5303ce0cc04e87a2ca77ab201"} Jul 10 19:09:21.666112 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:21.666078 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:21.747925 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:21.747877 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:09:21.748935 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:21.748909 2560 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-82b42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" start-of-body= Jul 10 19:09:21.749005 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:21.748961 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" podUID="df98b8de-e87e-4a94-adb1-2daf956c61c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" Jul 10 19:09:22.748064 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:22.748027 2560 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-82b42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" start-of-body= Jul 10 19:09:22.748482 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:22.748086 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" podUID="df98b8de-e87e-4a94-adb1-2daf956c61c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" Jul 10 19:09:23.138983 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:23.138882 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:23.747821 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:23.747789 2560 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-82b42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" start-of-body= Jul 10 19:09:23.748075 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:23.747845 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" podUID="df98b8de-e87e-4a94-adb1-2daf956c61c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" Jul 10 19:09:24.748333 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:24.748294 2560 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-82b42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" start-of-body= Jul 10 19:09:24.748856 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:24.748359 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" podUID="df98b8de-e87e-4a94-adb1-2daf956c61c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.30:15021/healthz/ready\": dial tcp 10.132.0.30:15021: connect: connection refused" Jul 10 19:09:24.863191 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:24.863149 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:09:25.268983 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:25.268953 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-sklxc" Jul 10 19:09:25.269144 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:25.269131 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-lvdc7" Jul 10 19:09:25.751518 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:25.751479 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:09:25.752355 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:25.752337 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-82b42" Jul 10 19:09:26.407074 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:26.407038 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:28.490203 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:28.490170 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:35.771617 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:35.771584 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:40.790656 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:40.790623 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-4vftx"] Jul 10 19:09:40.793859 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:40.793839 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4vftx" Jul 10 19:09:40.796922 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:40.796901 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 19:09:40.798498 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:40.798481 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 10 19:09:40.798612 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:40.798520 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 19:09:40.798612 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:40.798556 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-plhxh\"" Jul 10 19:09:40.805081 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:40.805056 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-4vftx"] Jul 10 19:09:40.824103 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:40.824071 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:40.917884 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:40.917854 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96835982-08fa-4605-8290-ffd6d56343e7-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4vftx\" (UID: \"96835982-08fa-4605-8290-ffd6d56343e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4vftx" Jul 10 19:09:40.918070 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:40.917907 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96835982-08fa-4605-8290-ffd6d56343e7-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4vftx\" (UID: \"96835982-08fa-4605-8290-ffd6d56343e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4vftx" Jul 10 19:09:40.918070 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:40.917990 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96835982-08fa-4605-8290-ffd6d56343e7-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4vftx\" (UID: \"96835982-08fa-4605-8290-ffd6d56343e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4vftx" Jul 10 19:09:40.918204 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:40.918066 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhtxj\" (UniqueName: \"kubernetes.io/projected/96835982-08fa-4605-8290-ffd6d56343e7-kube-api-access-fhtxj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4vftx\" (UID: \"96835982-08fa-4605-8290-ffd6d56343e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4vftx" Jul 10 19:09:40.918204 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:40.918114 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96835982-08fa-4605-8290-ffd6d56343e7-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4vftx\" (UID: \"96835982-08fa-4605-8290-ffd6d56343e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4vftx" Jul 10 19:09:40.918204 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:40.918172 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96835982-08fa-4605-8290-ffd6d56343e7-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4vftx\" (UID: \"96835982-08fa-4605-8290-ffd6d56343e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4vftx" Jul 10 19:09:41.018851 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:41.018813 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96835982-08fa-4605-8290-ffd6d56343e7-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4vftx\" (UID: \"96835982-08fa-4605-8290-ffd6d56343e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4vftx" Jul 10 19:09:41.019046 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:41.018865 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96835982-08fa-4605-8290-ffd6d56343e7-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4vftx\" (UID: \"96835982-08fa-4605-8290-ffd6d56343e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4vftx" Jul 10 19:09:41.019046 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:41.018923 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96835982-08fa-4605-8290-ffd6d56343e7-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4vftx\" (UID: \"96835982-08fa-4605-8290-ffd6d56343e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4vftx" Jul 10 19:09:41.019169 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:41.019071 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96835982-08fa-4605-8290-ffd6d56343e7-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4vftx\" (UID: \"96835982-08fa-4605-8290-ffd6d56343e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4vftx" Jul 10 19:09:41.019169 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:41.019119 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96835982-08fa-4605-8290-ffd6d56343e7-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4vftx\" (UID: \"96835982-08fa-4605-8290-ffd6d56343e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4vftx" Jul 10 19:09:41.019272 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:41.019172 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhtxj\" (UniqueName: \"kubernetes.io/projected/96835982-08fa-4605-8290-ffd6d56343e7-kube-api-access-fhtxj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4vftx\" (UID: \"96835982-08fa-4605-8290-ffd6d56343e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4vftx" Jul 10 19:09:41.019393 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:41.019354 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96835982-08fa-4605-8290-ffd6d56343e7-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4vftx\" (UID: \"96835982-08fa-4605-8290-ffd6d56343e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4vftx" Jul 10 19:09:41.019577 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:41.019393 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96835982-08fa-4605-8290-ffd6d56343e7-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4vftx\" (UID: \"96835982-08fa-4605-8290-ffd6d56343e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4vftx" Jul 10 19:09:41.019577 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:41.019466 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96835982-08fa-4605-8290-ffd6d56343e7-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4vftx\" (UID: \"96835982-08fa-4605-8290-ffd6d56343e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4vftx" Jul 10 19:09:41.021127 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:41.021104 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96835982-08fa-4605-8290-ffd6d56343e7-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4vftx\" (UID: \"96835982-08fa-4605-8290-ffd6d56343e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4vftx" Jul 10 19:09:41.021398 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:41.021382 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96835982-08fa-4605-8290-ffd6d56343e7-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4vftx\" (UID: \"96835982-08fa-4605-8290-ffd6d56343e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4vftx" Jul 10 19:09:41.028413 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:41.028392 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhtxj\" (UniqueName: \"kubernetes.io/projected/96835982-08fa-4605-8290-ffd6d56343e7-kube-api-access-fhtxj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4vftx\" (UID: \"96835982-08fa-4605-8290-ffd6d56343e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4vftx" Jul 10 19:09:41.104740 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:41.104641 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4vftx" Jul 10 19:09:41.236203 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:41.236156 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-4vftx"] Jul 10 19:09:41.238922 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:09:41.238894 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96835982_08fa_4605_8290_ffd6d56343e7.slice/crio-ad9d8479d99d2fcdf919c07a533d1c6889280b558a733882c73631cbc6b361d2 WatchSource:0}: Error finding container ad9d8479d99d2fcdf919c07a533d1c6889280b558a733882c73631cbc6b361d2: Status 404 returned error can't find the container with id ad9d8479d99d2fcdf919c07a533d1c6889280b558a733882c73631cbc6b361d2 Jul 10 19:09:41.240334 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:41.240312 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 19:09:41.395306 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:41.395224 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4vftx" event={"ID":"96835982-08fa-4605-8290-ffd6d56343e7","Type":"ContainerStarted","Data":"ad9d8479d99d2fcdf919c07a533d1c6889280b558a733882c73631cbc6b361d2"} Jul 10 19:09:42.401314 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:42.401273 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4vftx" event={"ID":"96835982-08fa-4605-8290-ffd6d56343e7","Type":"ContainerStarted","Data":"bf1cca1ce67562e07ee7bcaa378f1827a728917d103f197b022950785231579e"} Jul 10 19:09:42.421270 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:42.421219 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4vftx" podStartSLOduration=1.436396333 podStartE2EDuration="2.421201925s" totalImagesPullingTime="984.805592ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:09:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:09:42.420146593 +0000 UTC m=+673.750287404" watchObservedRunningTime="2026-07-10 19:09:42.421201925 +0000 UTC m=+673.751342742" Jul 10 19:09:51.105803 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:51.105746 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4vftx" Jul 10 19:09:51.106228 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:51.105860 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4vftx" Jul 10 19:09:51.117817 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:51.117792 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4vftx" Jul 10 19:09:51.453730 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:51.453699 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4vftx" Jul 10 19:09:52.939574 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:52.939544 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:53.039736 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:53.039701 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:59.835950 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:09:59.835917 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:10:03.386263 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:03.386225 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j"] Jul 10 19:10:03.389258 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:03.389239 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j" Jul 10 19:10:03.392165 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:03.392147 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 10 19:10:03.400661 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:03.400640 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j"] Jul 10 19:10:03.441701 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:03.441662 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a3b9969-6253-4250-a8c3-d87cbb56579a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j\" (UID: \"8a3b9969-6253-4250-a8c3-d87cbb56579a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j" Jul 10 19:10:03.441870 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:03.441740 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a3b9969-6253-4250-a8c3-d87cbb56579a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j\" (UID: \"8a3b9969-6253-4250-a8c3-d87cbb56579a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j" Jul 10 19:10:03.441870 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:03.441816 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a3b9969-6253-4250-a8c3-d87cbb56579a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j\" (UID: \"8a3b9969-6253-4250-a8c3-d87cbb56579a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j" Jul 10 19:10:03.441870 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:03.441861 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x9bm\" (UniqueName: \"kubernetes.io/projected/8a3b9969-6253-4250-a8c3-d87cbb56579a-kube-api-access-6x9bm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j\" (UID: \"8a3b9969-6253-4250-a8c3-d87cbb56579a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j" Jul 10 19:10:03.442005 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:03.441944 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a3b9969-6253-4250-a8c3-d87cbb56579a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j\" (UID: \"8a3b9969-6253-4250-a8c3-d87cbb56579a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j" Jul 10 19:10:03.442005 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:03.441997 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a3b9969-6253-4250-a8c3-d87cbb56579a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j\" (UID: \"8a3b9969-6253-4250-a8c3-d87cbb56579a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j" Jul 10 19:10:03.486151 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:03.486122 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:10:03.542808 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:03.542767 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a3b9969-6253-4250-a8c3-d87cbb56579a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j\" (UID: \"8a3b9969-6253-4250-a8c3-d87cbb56579a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j" Jul 10 19:10:03.542808 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:03.542810 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x9bm\" (UniqueName: \"kubernetes.io/projected/8a3b9969-6253-4250-a8c3-d87cbb56579a-kube-api-access-6x9bm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j\" (UID: \"8a3b9969-6253-4250-a8c3-d87cbb56579a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j" Jul 10 19:10:03.543050 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:03.542865 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a3b9969-6253-4250-a8c3-d87cbb56579a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j\" (UID: \"8a3b9969-6253-4250-a8c3-d87cbb56579a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j" Jul 10 19:10:03.543050 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:03.542917 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a3b9969-6253-4250-a8c3-d87cbb56579a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j\" (UID: \"8a3b9969-6253-4250-a8c3-d87cbb56579a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j" Jul 10 19:10:03.543050 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:03.542969 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a3b9969-6253-4250-a8c3-d87cbb56579a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j\" (UID: \"8a3b9969-6253-4250-a8c3-d87cbb56579a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j" Jul 10 19:10:03.543198 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:03.543123 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a3b9969-6253-4250-a8c3-d87cbb56579a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j\" (UID: \"8a3b9969-6253-4250-a8c3-d87cbb56579a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j" Jul 10 19:10:03.543252 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:03.543233 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a3b9969-6253-4250-a8c3-d87cbb56579a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j\" (UID: \"8a3b9969-6253-4250-a8c3-d87cbb56579a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j" Jul 10 19:10:03.543309 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:03.543264 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a3b9969-6253-4250-a8c3-d87cbb56579a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j\" (UID: \"8a3b9969-6253-4250-a8c3-d87cbb56579a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j" Jul 10 19:10:03.543309 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:03.543291 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a3b9969-6253-4250-a8c3-d87cbb56579a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j\" (UID: \"8a3b9969-6253-4250-a8c3-d87cbb56579a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j" Jul 10 19:10:03.545255 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:03.545223 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a3b9969-6253-4250-a8c3-d87cbb56579a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j\" (UID: \"8a3b9969-6253-4250-a8c3-d87cbb56579a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j" Jul 10 19:10:03.545539 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:03.545512 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a3b9969-6253-4250-a8c3-d87cbb56579a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j\" (UID: \"8a3b9969-6253-4250-a8c3-d87cbb56579a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j" Jul 10 19:10:03.551745 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:03.551722 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x9bm\" (UniqueName: \"kubernetes.io/projected/8a3b9969-6253-4250-a8c3-d87cbb56579a-kube-api-access-6x9bm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j\" (UID: \"8a3b9969-6253-4250-a8c3-d87cbb56579a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j" Jul 10 19:10:03.699576 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:03.699544 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j" Jul 10 19:10:03.829919 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:03.829857 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j"] Jul 10 19:10:03.832227 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:10:03.832200 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a3b9969_6253_4250_a8c3_d87cbb56579a.slice/crio-20f1f73f830dda501202621ec1a849ef74ee3064b3255885e3f17561a45e00b6 WatchSource:0}: Error finding container 20f1f73f830dda501202621ec1a849ef74ee3064b3255885e3f17561a45e00b6: Status 404 returned error can't find the container with id 20f1f73f830dda501202621ec1a849ef74ee3064b3255885e3f17561a45e00b6 Jul 10 19:10:04.505174 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:04.505137 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j" event={"ID":"8a3b9969-6253-4250-a8c3-d87cbb56579a","Type":"ContainerStarted","Data":"94850bef916170aa74c4ab56533e4fce78479c66b76c3f11535a40d65807dde5"} Jul 10 19:10:04.505567 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:04.505194 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j" event={"ID":"8a3b9969-6253-4250-a8c3-d87cbb56579a","Type":"ContainerStarted","Data":"20f1f73f830dda501202621ec1a849ef74ee3064b3255885e3f17561a45e00b6"} Jul 10 19:10:04.526588 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:04.526536 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j" podStartSLOduration=1.335949217 podStartE2EDuration="1.526514742s" totalImagesPullingTime="190.565525ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:10:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:10:04.526052632 +0000 UTC m=+695.856193442" watchObservedRunningTime="2026-07-10 19:10:04.526514742 +0000 UTC m=+695.856655551" Jul 10 19:10:13.700591 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:13.700558 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j" Jul 10 19:10:13.701152 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:13.700626 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j" Jul 10 19:10:13.712752 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:13.712728 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j" Jul 10 19:10:14.562801 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:14.562766 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j" Jul 10 19:10:20.126175 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:20.126109 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:10:27.745953 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:27.745872 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:10:45.175696 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:10:45.175661 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:11:13.472536 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:11:13.472501 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:11:13.536162 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:11:13.536120 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:11:14.584579 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:11:14.584542 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:11:14.684580 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:11:14.684545 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:12:24.087738 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:12:24.087700 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:12:25.436275 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:12:25.436240 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:12.219135 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:13:12.219097 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:12.321225 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:13:12.321187 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:12.437720 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:13:12.437696 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:18.658347 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:13:18.658309 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:18.717249 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:13:18.717219 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:18.784753 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:13:18.784713 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:18.885503 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:13:18.885466 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:18.985583 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:13:18.985503 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:25.691540 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:13:25.691425 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:25.791573 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:13:25.791543 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:29.165890 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:13:29.165856 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l52t_2a22956a-4e96-41de-945c-a851b587a442/ovn-acl-logging/0.log" Jul 10 19:13:29.172538 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:13:29.172517 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l52t_2a22956a-4e96-41de-945c-a851b587a442/ovn-acl-logging/0.log" Jul 10 19:13:43.499587 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:13:43.499557 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:43.568582 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:13:43.568539 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:15:00.151356 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:15:00.151272 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728515-l42tt"] Jul 10 19:15:00.153526 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:15:00.153509 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728515-l42tt" Jul 10 19:15:00.156408 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:15:00.156385 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fsnnw\"" Jul 10 19:15:00.163401 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:15:00.163377 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728515-l42tt" Jul 10 19:15:00.168655 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:15:00.168617 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728515-l42tt"] Jul 10 19:15:00.300681 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:15:00.300651 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728515-l42tt"] Jul 10 19:15:00.303291 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:15:00.303253 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83fa6ee5_9b25_4431_be98_9d359778d5da.slice/crio-c0f32f3e60ddae8df9b9d15c1889ac65bd452008cb3e47ea0815e3fa0c9ae388 WatchSource:0}: Error finding container c0f32f3e60ddae8df9b9d15c1889ac65bd452008cb3e47ea0815e3fa0c9ae388: Status 404 returned error can't find the container with id c0f32f3e60ddae8df9b9d15c1889ac65bd452008cb3e47ea0815e3fa0c9ae388 Jul 10 19:15:00.305212 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:15:00.305192 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 19:15:00.925884 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:15:00.925850 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728515-l42tt" event={"ID":"83fa6ee5-9b25-4431-be98-9d359778d5da","Type":"ContainerStarted","Data":"c0f32f3e60ddae8df9b9d15c1889ac65bd452008cb3e47ea0815e3fa0c9ae388"} Jul 10 19:15:03.943375 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:15:03.943341 2560 generic.go:356] "Generic (PLEG): container finished" podID="83fa6ee5-9b25-4431-be98-9d359778d5da" containerID="ff5fb6cd10c86748df47a4243f69f2879d677700f4e9df1fb0e2cbfc23eda189" exitCode=0 Jul 10 19:15:03.943734 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:15:03.943384 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728515-l42tt" event={"ID":"83fa6ee5-9b25-4431-be98-9d359778d5da","Type":"ContainerDied","Data":"ff5fb6cd10c86748df47a4243f69f2879d677700f4e9df1fb0e2cbfc23eda189"} Jul 10 19:15:05.079830 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:15:05.079805 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728515-l42tt" Jul 10 19:15:05.955066 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:15:05.955036 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728515-l42tt" Jul 10 19:15:05.955066 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:15:05.955053 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728515-l42tt" event={"ID":"83fa6ee5-9b25-4431-be98-9d359778d5da","Type":"ContainerDied","Data":"c0f32f3e60ddae8df9b9d15c1889ac65bd452008cb3e47ea0815e3fa0c9ae388"} Jul 10 19:15:05.955334 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:15:05.955098 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0f32f3e60ddae8df9b9d15c1889ac65bd452008cb3e47ea0815e3fa0c9ae388" Jul 10 19:17:39.025040 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:17:39.025002 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:17:40.150582 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:17:40.150543 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:18:04.661634 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:18:04.661554 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:18:08.692908 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:18:08.692865 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:18:29.199986 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:18:29.199955 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l52t_2a22956a-4e96-41de-945c-a851b587a442/ovn-acl-logging/0.log" Jul 10 19:18:29.203080 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:18:29.203059 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l52t_2a22956a-4e96-41de-945c-a851b587a442/ovn-acl-logging/0.log" Jul 10 19:19:39.040660 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:19:39.040585 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:23:29.232357 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:23:29.232331 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l52t_2a22956a-4e96-41de-945c-a851b587a442/ovn-acl-logging/0.log" Jul 10 19:23:29.233860 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:23:29.232686 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l52t_2a22956a-4e96-41de-945c-a851b587a442/ovn-acl-logging/0.log" Jul 10 19:23:57.837006 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:23:57.836924 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:24:04.589068 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:04.589030 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:24:12.884704 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:12.884644 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:24:17.937388 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:17.937351 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:24:20.735093 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:20.735052 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:24:20.830585 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:20.830555 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:24:21.164506 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:21.164478 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:24:27.200200 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.200163 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:24:27.276371 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.276338 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:24:27.387708 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.387670 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66"] Jul 10 19:24:27.388148 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.388128 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83fa6ee5-9b25-4431-be98-9d359778d5da" containerName="cleanup" Jul 10 19:24:27.388148 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.388149 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="83fa6ee5-9b25-4431-be98-9d359778d5da" containerName="cleanup" Jul 10 19:24:27.388305 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.388243 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="83fa6ee5-9b25-4431-be98-9d359778d5da" containerName="cleanup" Jul 10 19:24:27.390600 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.390581 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:27.393455 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.393422 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6e61e5c8-openshift-default-dockercfg-52zq9\"" Jul 10 19:24:27.419352 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.419332 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66"] Jul 10 19:24:27.427709 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.427679 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:24:27.486307 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.486216 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf5dab9-d8a6-4603-8518-e627e97cf0c2-credential-socket\") pod \"e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:27.486515 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.486309 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf5dab9-d8a6-4603-8518-e627e97cf0c2-istio-data\") pod \"e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:27.486664 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.486548 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf5dab9-d8a6-4603-8518-e627e97cf0c2-istio-podinfo\") pod \"e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:27.486755 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.486680 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf5dab9-d8a6-4603-8518-e627e97cf0c2-workload-certs\") pod \"e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:27.486831 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.486749 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf5dab9-d8a6-4603-8518-e627e97cf0c2-workload-socket\") pod \"e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:27.486913 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.486877 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf5dab9-d8a6-4603-8518-e627e97cf0c2-istio-token\") pod \"e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:27.487024 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.486983 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf5dab9-d8a6-4603-8518-e627e97cf0c2-istiod-ca-cert\") pod \"e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:27.487111 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.487053 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf5dab9-d8a6-4603-8518-e627e97cf0c2-istio-envoy\") pod \"e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:27.487196 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.487109 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqjhv\" (UniqueName: \"kubernetes.io/projected/abf5dab9-d8a6-4603-8518-e627e97cf0c2-kube-api-access-pqjhv\") pod \"e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:27.524353 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.524319 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:24:27.587660 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.587632 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf5dab9-d8a6-4603-8518-e627e97cf0c2-istio-token\") pod \"e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:27.587763 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.587677 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf5dab9-d8a6-4603-8518-e627e97cf0c2-istiod-ca-cert\") pod \"e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:27.587763 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.587700 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf5dab9-d8a6-4603-8518-e627e97cf0c2-istio-envoy\") pod \"e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:27.587763 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.587725 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqjhv\" (UniqueName: \"kubernetes.io/projected/abf5dab9-d8a6-4603-8518-e627e97cf0c2-kube-api-access-pqjhv\") pod \"e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:27.587763 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.587755 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf5dab9-d8a6-4603-8518-e627e97cf0c2-credential-socket\") pod \"e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:27.587943 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.587780 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf5dab9-d8a6-4603-8518-e627e97cf0c2-istio-data\") pod \"e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:27.587943 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.587833 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf5dab9-d8a6-4603-8518-e627e97cf0c2-istio-podinfo\") pod \"e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:27.587943 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.587855 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf5dab9-d8a6-4603-8518-e627e97cf0c2-workload-certs\") pod \"e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:27.587943 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.587880 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf5dab9-d8a6-4603-8518-e627e97cf0c2-workload-socket\") pod \"e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:27.588187 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.588168 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf5dab9-d8a6-4603-8518-e627e97cf0c2-credential-socket\") pod \"e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:27.588252 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.588230 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf5dab9-d8a6-4603-8518-e627e97cf0c2-istio-data\") pod \"e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:27.588317 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.588301 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf5dab9-d8a6-4603-8518-e627e97cf0c2-workload-socket\") pod \"e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:27.588369 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.588332 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf5dab9-d8a6-4603-8518-e627e97cf0c2-workload-certs\") pod \"e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:27.588421 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.588404 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf5dab9-d8a6-4603-8518-e627e97cf0c2-istiod-ca-cert\") pod \"e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:27.590039 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.590012 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf5dab9-d8a6-4603-8518-e627e97cf0c2-istio-envoy\") pod \"e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:27.590184 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.590169 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf5dab9-d8a6-4603-8518-e627e97cf0c2-istio-podinfo\") pod \"e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:27.596117 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.596095 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqjhv\" (UniqueName: \"kubernetes.io/projected/abf5dab9-d8a6-4603-8518-e627e97cf0c2-kube-api-access-pqjhv\") pod \"e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:27.596204 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.596185 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf5dab9-d8a6-4603-8518-e627e97cf0c2-istio-token\") pod \"e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:27.700683 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.700659 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:27.838854 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.838824 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66"] Jul 10 19:24:27.839654 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:24:27.839628 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf5dab9_d8a6_4603_8518_e627e97cf0c2.slice/crio-0872a060df8121622bdd972da9040ff4f1d907f413a39a501acbc48cbf2e9bca WatchSource:0}: Error finding container 0872a060df8121622bdd972da9040ff4f1d907f413a39a501acbc48cbf2e9bca: Status 404 returned error can't find the container with id 0872a060df8121622bdd972da9040ff4f1d907f413a39a501acbc48cbf2e9bca Jul 10 19:24:27.841818 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.841783 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:24:27.841915 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.841864 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:24:27.841915 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:27.841908 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:24:28.627139 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:28.627105 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:24:28.677202 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:28.677172 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" event={"ID":"abf5dab9-d8a6-4603-8518-e627e97cf0c2","Type":"ContainerStarted","Data":"713f64cca978571d11146613c49e31e7ad628ab2a3f0789230f9d9ab1f7f32f2"} Jul 10 19:24:28.677344 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:28.677224 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" event={"ID":"abf5dab9-d8a6-4603-8518-e627e97cf0c2","Type":"ContainerStarted","Data":"0872a060df8121622bdd972da9040ff4f1d907f413a39a501acbc48cbf2e9bca"} Jul 10 19:24:28.700991 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:28.700966 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:28.706521 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:28.706497 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:28.725903 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:28.725881 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt"] Jul 10 19:24:28.728421 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:28.728405 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt" Jul 10 19:24:28.731413 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:28.731389 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 19:24:28.731413 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:28.731406 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-44lh8\"" Jul 10 19:24:28.732686 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:28.732669 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 19:24:28.736348 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:28.736331 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6e61e5c8-serving-cert\"" Jul 10 19:24:28.759714 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:28.759672 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" podStartSLOduration=1.759660381 podStartE2EDuration="1.759660381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:24:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:24:28.756250376 +0000 UTC m=+1560.086391186" watchObservedRunningTime="2026-07-10 19:24:28.759660381 +0000 UTC m=+1560.089801189" Jul 10 19:24:28.769841 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:28.769815 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt"] Jul 10 19:24:28.783056 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:28.783032 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:24:28.799229 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:28.799209 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4937a540-5113-4360-abb6-dad54a3d1cf9-openshift-service-ca\") pod \"maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt\" (UID: \"4937a540-5113-4360-abb6-dad54a3d1cf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt" Jul 10 19:24:28.799317 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:28.799266 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4937a540-5113-4360-abb6-dad54a3d1cf9-maas-api-tls\") pod \"maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt\" (UID: \"4937a540-5113-4360-abb6-dad54a3d1cf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt" Jul 10 19:24:28.799397 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:28.799345 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwhpv\" (UniqueName: \"kubernetes.io/projected/4937a540-5113-4360-abb6-dad54a3d1cf9-kube-api-access-wwhpv\") pod \"maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt\" (UID: \"4937a540-5113-4360-abb6-dad54a3d1cf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt" Jul 10 19:24:28.900423 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:28.900337 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4937a540-5113-4360-abb6-dad54a3d1cf9-openshift-service-ca\") pod \"maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt\" (UID: \"4937a540-5113-4360-abb6-dad54a3d1cf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt" Jul 10 19:24:28.902285 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:28.901129 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4937a540-5113-4360-abb6-dad54a3d1cf9-maas-api-tls\") pod \"maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt\" (UID: \"4937a540-5113-4360-abb6-dad54a3d1cf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt" Jul 10 19:24:28.902285 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:28.901151 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4937a540-5113-4360-abb6-dad54a3d1cf9-openshift-service-ca\") pod \"maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt\" (UID: \"4937a540-5113-4360-abb6-dad54a3d1cf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt" Jul 10 19:24:28.902285 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:28.901249 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwhpv\" (UniqueName: \"kubernetes.io/projected/4937a540-5113-4360-abb6-dad54a3d1cf9-kube-api-access-wwhpv\") pod \"maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt\" (UID: \"4937a540-5113-4360-abb6-dad54a3d1cf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt" Jul 10 19:24:28.902285 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:24:28.901673 2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-6e61e5c8-serving-cert: secret "maas-api-e2e-ait-6e61e5c8-serving-cert" not found Jul 10 19:24:28.902285 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:24:28.901727 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4937a540-5113-4360-abb6-dad54a3d1cf9-maas-api-tls podName:4937a540-5113-4360-abb6-dad54a3d1cf9 nodeName:}" failed. No retries permitted until 2026-07-10 19:24:29.401709825 +0000 UTC m=+1560.731850620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4937a540-5113-4360-abb6-dad54a3d1cf9-maas-api-tls") pod "maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt" (UID: "4937a540-5113-4360-abb6-dad54a3d1cf9") : secret "maas-api-e2e-ait-6e61e5c8-serving-cert" not found Jul 10 19:24:28.907579 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:28.907547 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:24:28.913765 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:28.913744 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwhpv\" (UniqueName: \"kubernetes.io/projected/4937a540-5113-4360-abb6-dad54a3d1cf9-kube-api-access-wwhpv\") pod \"maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt\" (UID: \"4937a540-5113-4360-abb6-dad54a3d1cf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt" Jul 10 19:24:29.021898 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:29.021871 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:24:29.344272 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:29.344238 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:24:29.406484 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:29.406427 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4937a540-5113-4360-abb6-dad54a3d1cf9-maas-api-tls\") pod \"maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt\" (UID: \"4937a540-5113-4360-abb6-dad54a3d1cf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt" Jul 10 19:24:29.408651 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:29.408630 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4937a540-5113-4360-abb6-dad54a3d1cf9-maas-api-tls\") pod \"maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt\" (UID: \"4937a540-5113-4360-abb6-dad54a3d1cf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt" Jul 10 19:24:29.443588 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:29.443561 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:24:29.643909 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:29.643835 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-44lh8\"" Jul 10 19:24:29.651999 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:29.651976 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt" Jul 10 19:24:29.682350 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:29.682322 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:29.683399 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:29.683383 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:29.795852 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:29.795829 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt"] Jul 10 19:24:29.797993 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:24:29.797965 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4937a540_5113_4360_abb6_dad54a3d1cf9.slice/crio-1346742f896820424316c194778a5efb5d37f31425608b026cac7ca0e60f02d8 WatchSource:0}: Error finding container 1346742f896820424316c194778a5efb5d37f31425608b026cac7ca0e60f02d8: Status 404 returned error can't find the container with id 1346742f896820424316c194778a5efb5d37f31425608b026cac7ca0e60f02d8 Jul 10 19:24:29.799509 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:29.799481 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 19:24:30.689602 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:30.689566 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt" event={"ID":"4937a540-5113-4360-abb6-dad54a3d1cf9","Type":"ContainerStarted","Data":"1346742f896820424316c194778a5efb5d37f31425608b026cac7ca0e60f02d8"} Jul 10 19:24:31.990527 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:31.990492 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:24:32.700912 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:32.700874 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt" event={"ID":"4937a540-5113-4360-abb6-dad54a3d1cf9","Type":"ContainerStarted","Data":"c075a561cab0ece514719aafa42c390dd2c0aa686f63baf8b12d65a5505d34f9"} Jul 10 19:24:32.701098 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:32.700958 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt" Jul 10 19:24:32.729065 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:32.729024 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt" podStartSLOduration=2.680446984 podStartE2EDuration="4.729011399s" totalImagesPullingTime="2.048564415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:24:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:24:32.72831402 +0000 UTC m=+1564.058454853" watchObservedRunningTime="2026-07-10 19:24:32.729011399 +0000 UTC m=+1564.059152207" Jul 10 19:24:38.712245 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:38.712214 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt" Jul 10 19:24:40.742036 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:40.741977 2560 generic.go:356] "Generic (PLEG): container finished" podID="abf5dab9-d8a6-4603-8518-e627e97cf0c2" containerID="713f64cca978571d11146613c49e31e7ad628ab2a3f0789230f9d9ab1f7f32f2" exitCode=0 Jul 10 19:24:40.742373 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:40.742046 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" event={"ID":"abf5dab9-d8a6-4603-8518-e627e97cf0c2","Type":"ContainerDied","Data":"713f64cca978571d11146613c49e31e7ad628ab2a3f0789230f9d9ab1f7f32f2"} Jul 10 19:24:40.742476 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:40.742433 2560 scope.go:117] "RemoveContainer" containerID="713f64cca978571d11146613c49e31e7ad628ab2a3f0789230f9d9ab1f7f32f2" Jul 10 19:24:40.743493 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:40.743465 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:24:40.743668 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:40.743545 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:24:40.743668 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:40.743596 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:24:41.748373 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:41.748342 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" event={"ID":"abf5dab9-d8a6-4603-8518-e627e97cf0c2","Type":"ContainerStarted","Data":"defd503a85902f90b35377812a1eb634d733e930da1d8b2a0195df335f2e00a1"} Jul 10 19:24:42.701023 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:42.700988 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:42.702118 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:42.702091 2560 patch_prober.go:28] interesting pod/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 10 19:24:42.702241 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:42.702144 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" podUID="abf5dab9-d8a6-4603-8518-e627e97cf0c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 10 19:24:43.701086 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:43.701050 2560 patch_prober.go:28] interesting pod/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 10 19:24:43.701465 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:43.701110 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" podUID="abf5dab9-d8a6-4603-8518-e627e97cf0c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 10 19:24:44.682802 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:44.682770 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:44.701553 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:44.701520 2560 patch_prober.go:28] interesting pod/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 10 19:24:44.701952 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:44.701577 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" podUID="abf5dab9-d8a6-4603-8518-e627e97cf0c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 10 19:24:45.701238 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:45.701202 2560 patch_prober.go:28] interesting pod/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 10 19:24:45.701410 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:45.701255 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" podUID="abf5dab9-d8a6-4603-8518-e627e97cf0c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 10 19:24:46.704640 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:46.704604 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:24:46.705466 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:24:46.705421 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:25:35.964157 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:35.964071 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:36.034853 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:36.034822 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:44.148522 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:44.148490 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt"] Jul 10 19:25:44.149085 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:44.148714 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt" podUID="4937a540-5113-4360-abb6-dad54a3d1cf9" containerName="maas-api" containerID="cri-o://c075a561cab0ece514719aafa42c390dd2c0aa686f63baf8b12d65a5505d34f9" gracePeriod=30 Jul 10 19:25:44.171726 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:44.171682 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:44.396345 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:44.396314 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt" Jul 10 19:25:44.562108 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:44.562076 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4937a540-5113-4360-abb6-dad54a3d1cf9-maas-api-tls\") pod \"4937a540-5113-4360-abb6-dad54a3d1cf9\" (UID: \"4937a540-5113-4360-abb6-dad54a3d1cf9\") " Jul 10 19:25:44.562283 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:44.562184 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwhpv\" (UniqueName: \"kubernetes.io/projected/4937a540-5113-4360-abb6-dad54a3d1cf9-kube-api-access-wwhpv\") pod \"4937a540-5113-4360-abb6-dad54a3d1cf9\" (UID: \"4937a540-5113-4360-abb6-dad54a3d1cf9\") " Jul 10 19:25:44.562283 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:44.562231 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4937a540-5113-4360-abb6-dad54a3d1cf9-openshift-service-ca\") pod \"4937a540-5113-4360-abb6-dad54a3d1cf9\" (UID: \"4937a540-5113-4360-abb6-dad54a3d1cf9\") " Jul 10 19:25:44.562648 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:44.562616 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4937a540-5113-4360-abb6-dad54a3d1cf9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4937a540-5113-4360-abb6-dad54a3d1cf9" (UID: "4937a540-5113-4360-abb6-dad54a3d1cf9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:25:44.564265 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:44.564236 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4937a540-5113-4360-abb6-dad54a3d1cf9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4937a540-5113-4360-abb6-dad54a3d1cf9" (UID: "4937a540-5113-4360-abb6-dad54a3d1cf9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:25:44.564384 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:44.564285 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4937a540-5113-4360-abb6-dad54a3d1cf9-kube-api-access-wwhpv" (OuterVolumeSpecName: "kube-api-access-wwhpv") pod "4937a540-5113-4360-abb6-dad54a3d1cf9" (UID: "4937a540-5113-4360-abb6-dad54a3d1cf9"). InnerVolumeSpecName "kube-api-access-wwhpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:25:44.663834 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:44.663810 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwhpv\" (UniqueName: \"kubernetes.io/projected/4937a540-5113-4360-abb6-dad54a3d1cf9-kube-api-access-wwhpv\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:25:44.663834 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:44.663834 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4937a540-5113-4360-abb6-dad54a3d1cf9-openshift-service-ca\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:25:44.663969 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:44.663844 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4937a540-5113-4360-abb6-dad54a3d1cf9-maas-api-tls\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:25:45.064899 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:45.064864 2560 generic.go:356] "Generic (PLEG): container finished" podID="4937a540-5113-4360-abb6-dad54a3d1cf9" containerID="c075a561cab0ece514719aafa42c390dd2c0aa686f63baf8b12d65a5505d34f9" exitCode=0 Jul 10 19:25:45.065061 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:45.064928 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt" Jul 10 19:25:45.065061 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:45.064950 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt" event={"ID":"4937a540-5113-4360-abb6-dad54a3d1cf9","Type":"ContainerDied","Data":"c075a561cab0ece514719aafa42c390dd2c0aa686f63baf8b12d65a5505d34f9"} Jul 10 19:25:45.065061 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:45.065036 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt" event={"ID":"4937a540-5113-4360-abb6-dad54a3d1cf9","Type":"ContainerDied","Data":"1346742f896820424316c194778a5efb5d37f31425608b026cac7ca0e60f02d8"} Jul 10 19:25:45.065160 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:45.065064 2560 scope.go:117] "RemoveContainer" containerID="c075a561cab0ece514719aafa42c390dd2c0aa686f63baf8b12d65a5505d34f9" Jul 10 19:25:45.074222 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:45.074195 2560 scope.go:117] "RemoveContainer" containerID="c075a561cab0ece514719aafa42c390dd2c0aa686f63baf8b12d65a5505d34f9" Jul 10 19:25:45.074488 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:25:45.074469 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c075a561cab0ece514719aafa42c390dd2c0aa686f63baf8b12d65a5505d34f9\": container with ID starting with c075a561cab0ece514719aafa42c390dd2c0aa686f63baf8b12d65a5505d34f9 not found: ID does not exist" containerID="c075a561cab0ece514719aafa42c390dd2c0aa686f63baf8b12d65a5505d34f9" Jul 10 19:25:45.074546 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:45.074495 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c075a561cab0ece514719aafa42c390dd2c0aa686f63baf8b12d65a5505d34f9"} err="failed to get container status \"c075a561cab0ece514719aafa42c390dd2c0aa686f63baf8b12d65a5505d34f9\": rpc error: code = NotFound desc = could not find container \"c075a561cab0ece514719aafa42c390dd2c0aa686f63baf8b12d65a5505d34f9\": container with ID starting with c075a561cab0ece514719aafa42c390dd2c0aa686f63baf8b12d65a5505d34f9 not found: ID does not exist" Jul 10 19:25:45.086337 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:45.086316 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt"] Jul 10 19:25:45.091770 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:45.091750 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e61e5c8-5cc877ccb7-ztnpt"] Jul 10 19:25:45.253991 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:45.253957 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4937a540-5113-4360-abb6-dad54a3d1cf9" path="/var/lib/kubelet/pods/4937a540-5113-4360-abb6-dad54a3d1cf9/volumes" Jul 10 19:25:48.801419 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:48.801386 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:54.867618 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:54.867577 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:54.968234 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:54.968193 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:58.969495 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:58.969460 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66"] Jul 10 19:25:58.969877 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:58.969663 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" podUID="abf5dab9-d8a6-4603-8518-e627e97cf0c2" containerName="istio-proxy" containerID="cri-o://defd503a85902f90b35377812a1eb634d733e930da1d8b2a0195df335f2e00a1" gracePeriod=30 Jul 10 19:25:58.994906 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:58.994871 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:59.272591 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.272504 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8"] Jul 10 19:25:59.273086 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.273070 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4937a540-5113-4360-abb6-dad54a3d1cf9" containerName="maas-api" Jul 10 19:25:59.273166 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.273088 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4937a540-5113-4360-abb6-dad54a3d1cf9" containerName="maas-api" Jul 10 19:25:59.273206 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.273192 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4937a540-5113-4360-abb6-dad54a3d1cf9" containerName="maas-api" Jul 10 19:25:59.277986 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.277966 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:25:59.280853 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.280836 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6402da18-openshift-default-dockercfg-xzdg6\"" Jul 10 19:25:59.291294 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.291270 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8"] Jul 10 19:25:59.315803 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.315760 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:59.392112 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.392076 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68df8fbe-2268-46eb-8aa2-228b05b40555-istio-data\") pod \"e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:25:59.392375 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.392355 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68df8fbe-2268-46eb-8aa2-228b05b40555-istio-envoy\") pod \"e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:25:59.392628 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.392611 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68df8fbe-2268-46eb-8aa2-228b05b40555-workload-socket\") pod \"e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:25:59.392783 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.392767 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68df8fbe-2268-46eb-8aa2-228b05b40555-credential-socket\") pod \"e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:25:59.392906 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.392893 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68df8fbe-2268-46eb-8aa2-228b05b40555-istio-podinfo\") pod \"e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:25:59.393034 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.393021 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9tr7\" (UniqueName: \"kubernetes.io/projected/68df8fbe-2268-46eb-8aa2-228b05b40555-kube-api-access-z9tr7\") pod \"e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:25:59.393159 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.393144 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68df8fbe-2268-46eb-8aa2-228b05b40555-istio-token\") pod \"e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:25:59.393350 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.393314 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68df8fbe-2268-46eb-8aa2-228b05b40555-istiod-ca-cert\") pod \"e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:25:59.393591 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.393538 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68df8fbe-2268-46eb-8aa2-228b05b40555-workload-certs\") pod \"e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:25:59.494813 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.494779 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68df8fbe-2268-46eb-8aa2-228b05b40555-istio-data\") pod \"e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:25:59.494956 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.494819 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68df8fbe-2268-46eb-8aa2-228b05b40555-istio-envoy\") pod \"e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:25:59.494956 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.494895 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68df8fbe-2268-46eb-8aa2-228b05b40555-workload-socket\") pod \"e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:25:59.494956 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.494930 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68df8fbe-2268-46eb-8aa2-228b05b40555-credential-socket\") pod \"e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:25:59.495105 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.494974 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68df8fbe-2268-46eb-8aa2-228b05b40555-istio-podinfo\") pod \"e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:25:59.495105 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.495007 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9tr7\" (UniqueName: \"kubernetes.io/projected/68df8fbe-2268-46eb-8aa2-228b05b40555-kube-api-access-z9tr7\") pod \"e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:25:59.495105 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.495033 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68df8fbe-2268-46eb-8aa2-228b05b40555-istio-token\") pod \"e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:25:59.495105 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.495086 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68df8fbe-2268-46eb-8aa2-228b05b40555-istiod-ca-cert\") pod \"e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:25:59.495303 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.495117 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68df8fbe-2268-46eb-8aa2-228b05b40555-workload-certs\") pod \"e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:25:59.495303 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.495217 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68df8fbe-2268-46eb-8aa2-228b05b40555-istio-data\") pod \"e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:25:59.495577 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.495543 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68df8fbe-2268-46eb-8aa2-228b05b40555-workload-socket\") pod \"e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:25:59.495688 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.495596 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68df8fbe-2268-46eb-8aa2-228b05b40555-workload-certs\") pod \"e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:25:59.495737 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.495704 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68df8fbe-2268-46eb-8aa2-228b05b40555-credential-socket\") pod \"e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:25:59.495818 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.495800 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68df8fbe-2268-46eb-8aa2-228b05b40555-istiod-ca-cert\") pod \"e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:25:59.496988 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.496962 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68df8fbe-2268-46eb-8aa2-228b05b40555-istio-envoy\") pod \"e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:25:59.497637 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.497613 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68df8fbe-2268-46eb-8aa2-228b05b40555-istio-podinfo\") pod \"e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:25:59.504282 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.504262 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68df8fbe-2268-46eb-8aa2-228b05b40555-istio-token\") pod \"e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:25:59.504813 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.504794 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9tr7\" (UniqueName: \"kubernetes.io/projected/68df8fbe-2268-46eb-8aa2-228b05b40555-kube-api-access-z9tr7\") pod \"e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:25:59.588703 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.588643 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:25:59.743306 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.743279 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8"] Jul 10 19:25:59.745156 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:25:59.745121 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68df8fbe_2268_46eb_8aa2_228b05b40555.slice/crio-f9377222924f3168e65abe97419b6e46ad7e8a379af5b4c6f0f01e19e429c788 WatchSource:0}: Error finding container f9377222924f3168e65abe97419b6e46ad7e8a379af5b4c6f0f01e19e429c788: Status 404 returned error can't find the container with id f9377222924f3168e65abe97419b6e46ad7e8a379af5b4c6f0f01e19e429c788 Jul 10 19:25:59.747233 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.747202 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:25:59.747315 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.747279 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:25:59.747315 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:25:59.747308 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:26:00.142688 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:00.142650 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" event={"ID":"68df8fbe-2268-46eb-8aa2-228b05b40555","Type":"ContainerStarted","Data":"46663d3586e6ede7143121d8d29e1b312b82593bc967995ba9e106b02e1c5fe4"} Jul 10 19:26:00.143088 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:00.142713 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" event={"ID":"68df8fbe-2268-46eb-8aa2-228b05b40555","Type":"ContainerStarted","Data":"f9377222924f3168e65abe97419b6e46ad7e8a379af5b4c6f0f01e19e429c788"} Jul 10 19:26:00.170210 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:00.170149 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" podStartSLOduration=1.170130432 podStartE2EDuration="1.170130432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:25:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:26:00.168339189 +0000 UTC m=+1651.498479998" watchObservedRunningTime="2026-07-10 19:26:00.170130432 +0000 UTC m=+1651.500271241" Jul 10 19:26:00.493256 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:00.493227 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:26:00.577493 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:00.577433 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:26:00.583332 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:00.583307 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6402da18-64db9cf95d-ptk56"] Jul 10 19:26:00.587234 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:00.587215 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6402da18-64db9cf95d-ptk56" Jul 10 19:26:00.589523 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:00.589500 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:26:00.590222 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:00.590034 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6402da18-serving-cert\"" Jul 10 19:26:00.590222 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:00.590096 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 19:26:00.590222 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:00.590115 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-44lh8\"" Jul 10 19:26:00.590222 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:00.590100 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 19:26:00.595096 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:00.595066 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:26:00.597909 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:00.597888 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6402da18-64db9cf95d-ptk56"] Jul 10 19:26:00.647995 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:00.647966 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:26:00.705518 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:00.705481 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c9b7a1-67d4-473d-8c52-0d30a4a89247-maas-api-tls\") pod \"maas-api-e2e-ait-6402da18-64db9cf95d-ptk56\" (UID: \"c9c9b7a1-67d4-473d-8c52-0d30a4a89247\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6402da18-64db9cf95d-ptk56" Jul 10 19:26:00.705683 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:00.705577 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c9b7a1-67d4-473d-8c52-0d30a4a89247-openshift-service-ca\") pod \"maas-api-e2e-ait-6402da18-64db9cf95d-ptk56\" (UID: \"c9c9b7a1-67d4-473d-8c52-0d30a4a89247\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6402da18-64db9cf95d-ptk56" Jul 10 19:26:00.705683 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:00.705627 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc8wn\" (UniqueName: \"kubernetes.io/projected/c9c9b7a1-67d4-473d-8c52-0d30a4a89247-kube-api-access-wc8wn\") pod \"maas-api-e2e-ait-6402da18-64db9cf95d-ptk56\" (UID: \"c9c9b7a1-67d4-473d-8c52-0d30a4a89247\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6402da18-64db9cf95d-ptk56" Jul 10 19:26:00.806466 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:00.806384 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c9b7a1-67d4-473d-8c52-0d30a4a89247-maas-api-tls\") pod \"maas-api-e2e-ait-6402da18-64db9cf95d-ptk56\" (UID: \"c9c9b7a1-67d4-473d-8c52-0d30a4a89247\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6402da18-64db9cf95d-ptk56" Jul 10 19:26:00.806466 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:00.806427 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c9b7a1-67d4-473d-8c52-0d30a4a89247-openshift-service-ca\") pod \"maas-api-e2e-ait-6402da18-64db9cf95d-ptk56\" (UID: \"c9c9b7a1-67d4-473d-8c52-0d30a4a89247\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6402da18-64db9cf95d-ptk56" Jul 10 19:26:00.806627 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:00.806487 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc8wn\" (UniqueName: \"kubernetes.io/projected/c9c9b7a1-67d4-473d-8c52-0d30a4a89247-kube-api-access-wc8wn\") pod \"maas-api-e2e-ait-6402da18-64db9cf95d-ptk56\" (UID: \"c9c9b7a1-67d4-473d-8c52-0d30a4a89247\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6402da18-64db9cf95d-ptk56" Jul 10 19:26:00.806678 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:26:00.806665 2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-6402da18-serving-cert: secret "maas-api-e2e-ait-6402da18-serving-cert" not found Jul 10 19:26:00.806726 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:26:00.806718 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9c9b7a1-67d4-473d-8c52-0d30a4a89247-maas-api-tls podName:c9c9b7a1-67d4-473d-8c52-0d30a4a89247 nodeName:}" failed. No retries permitted until 2026-07-10 19:26:01.306704213 +0000 UTC m=+1652.636845000 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c9c9b7a1-67d4-473d-8c52-0d30a4a89247-maas-api-tls") pod "maas-api-e2e-ait-6402da18-64db9cf95d-ptk56" (UID: "c9c9b7a1-67d4-473d-8c52-0d30a4a89247") : secret "maas-api-e2e-ait-6402da18-serving-cert" not found Jul 10 19:26:00.807048 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:00.807033 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c9b7a1-67d4-473d-8c52-0d30a4a89247-openshift-service-ca\") pod \"maas-api-e2e-ait-6402da18-64db9cf95d-ptk56\" (UID: \"c9c9b7a1-67d4-473d-8c52-0d30a4a89247\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6402da18-64db9cf95d-ptk56" Jul 10 19:26:00.815713 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:00.815689 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc8wn\" (UniqueName: \"kubernetes.io/projected/c9c9b7a1-67d4-473d-8c52-0d30a4a89247-kube-api-access-wc8wn\") pod \"maas-api-e2e-ait-6402da18-64db9cf95d-ptk56\" (UID: \"c9c9b7a1-67d4-473d-8c52-0d30a4a89247\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6402da18-64db9cf95d-ptk56" Jul 10 19:26:00.907213 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:00.907188 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:26:01.144907 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:01.144823 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:26:01.147687 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:01.147653 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:26:01.148566 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:01.148549 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:26:01.245194 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:01.245151 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:26:01.311908 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:01.311876 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c9b7a1-67d4-473d-8c52-0d30a4a89247-maas-api-tls\") pod \"maas-api-e2e-ait-6402da18-64db9cf95d-ptk56\" (UID: \"c9c9b7a1-67d4-473d-8c52-0d30a4a89247\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6402da18-64db9cf95d-ptk56" Jul 10 19:26:01.318045 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:01.318010 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c9b7a1-67d4-473d-8c52-0d30a4a89247-maas-api-tls\") pod \"maas-api-e2e-ait-6402da18-64db9cf95d-ptk56\" (UID: \"c9c9b7a1-67d4-473d-8c52-0d30a4a89247\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6402da18-64db9cf95d-ptk56" Jul 10 19:26:01.500466 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:01.500415 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6402da18-64db9cf95d-ptk56" Jul 10 19:26:01.630664 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:01.630642 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6402da18-64db9cf95d-ptk56"] Jul 10 19:26:01.632488 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:26:01.632429 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c9b7a1_67d4_473d_8c52_0d30a4a89247.slice/crio-82e7ab38a497a7410e0929fc249128ec4d90fb40ab421a2a6b949b97e84370d6 WatchSource:0}: Error finding container 82e7ab38a497a7410e0929fc249128ec4d90fb40ab421a2a6b949b97e84370d6: Status 404 returned error can't find the container with id 82e7ab38a497a7410e0929fc249128ec4d90fb40ab421a2a6b949b97e84370d6 Jul 10 19:26:01.707494 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:01.707432 2560 patch_prober.go:28] interesting pod/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 19:26:01.707623 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:01.707512 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" podUID="abf5dab9-d8a6-4603-8518-e627e97cf0c2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 19:26:02.154862 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:02.154828 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6402da18-64db9cf95d-ptk56" event={"ID":"c9c9b7a1-67d4-473d-8c52-0d30a4a89247","Type":"ContainerStarted","Data":"14250d229d209886ed9926cf0ec8fbe023774cad11f952a42405a5c9dd456cc2"} Jul 10 19:26:02.155298 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:02.154882 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6402da18-64db9cf95d-ptk56" event={"ID":"c9c9b7a1-67d4-473d-8c52-0d30a4a89247","Type":"ContainerStarted","Data":"82e7ab38a497a7410e0929fc249128ec4d90fb40ab421a2a6b949b97e84370d6"} Jul 10 19:26:02.155298 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:02.155135 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6402da18-64db9cf95d-ptk56" Jul 10 19:26:02.175456 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:02.175401 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6402da18-64db9cf95d-ptk56" podStartSLOduration=1.782242336 podStartE2EDuration="2.175386574s" totalImagesPullingTime="393.144238ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:26:02.173176701 +0000 UTC m=+1653.503317510" watchObservedRunningTime="2026-07-10 19:26:02.175386574 +0000 UTC m=+1653.505527382" Jul 10 19:26:04.170952 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:04.170926 2560 generic.go:356] "Generic (PLEG): container finished" podID="abf5dab9-d8a6-4603-8518-e627e97cf0c2" containerID="defd503a85902f90b35377812a1eb634d733e930da1d8b2a0195df335f2e00a1" exitCode=0 Jul 10 19:26:04.171278 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:04.171004 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" event={"ID":"abf5dab9-d8a6-4603-8518-e627e97cf0c2","Type":"ContainerDied","Data":"defd503a85902f90b35377812a1eb634d733e930da1d8b2a0195df335f2e00a1"} Jul 10 19:26:04.171278 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:04.171081 2560 scope.go:117] "RemoveContainer" containerID="713f64cca978571d11146613c49e31e7ad628ab2a3f0789230f9d9ab1f7f32f2" Jul 10 19:26:04.260154 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:04.260132 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:26:04.346985 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:04.345776 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf5dab9-d8a6-4603-8518-e627e97cf0c2-istio-data\") pod \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " Jul 10 19:26:04.346985 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:04.345829 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf5dab9-d8a6-4603-8518-e627e97cf0c2-workload-certs\") pod \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " Jul 10 19:26:04.346985 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:04.345859 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf5dab9-d8a6-4603-8518-e627e97cf0c2-workload-socket\") pod \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " Jul 10 19:26:04.346985 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:04.345889 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf5dab9-d8a6-4603-8518-e627e97cf0c2-istio-podinfo\") pod \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " Jul 10 19:26:04.346985 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:04.345981 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf5dab9-d8a6-4603-8518-e627e97cf0c2-istiod-ca-cert\") pod \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " Jul 10 19:26:04.346985 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:04.346010 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf5dab9-d8a6-4603-8518-e627e97cf0c2-istio-token\") pod \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " Jul 10 19:26:04.346985 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:04.346090 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf5dab9-d8a6-4603-8518-e627e97cf0c2-credential-socket\") pod \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " Jul 10 19:26:04.346985 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:04.346119 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqjhv\" (UniqueName: \"kubernetes.io/projected/abf5dab9-d8a6-4603-8518-e627e97cf0c2-kube-api-access-pqjhv\") pod \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " Jul 10 19:26:04.346985 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:04.346167 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf5dab9-d8a6-4603-8518-e627e97cf0c2-istio-envoy\") pod \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\" (UID: \"abf5dab9-d8a6-4603-8518-e627e97cf0c2\") " Jul 10 19:26:04.349600 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:04.348818 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abf5dab9-d8a6-4603-8518-e627e97cf0c2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "abf5dab9-d8a6-4603-8518-e627e97cf0c2" (UID: "abf5dab9-d8a6-4603-8518-e627e97cf0c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:26:04.349600 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:04.349000 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf5dab9-d8a6-4603-8518-e627e97cf0c2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "abf5dab9-d8a6-4603-8518-e627e97cf0c2" (UID: "abf5dab9-d8a6-4603-8518-e627e97cf0c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:26:04.349600 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:04.349467 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf5dab9-d8a6-4603-8518-e627e97cf0c2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "abf5dab9-d8a6-4603-8518-e627e97cf0c2" (UID: "abf5dab9-d8a6-4603-8518-e627e97cf0c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:26:04.349600 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:04.349565 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf5dab9-d8a6-4603-8518-e627e97cf0c2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "abf5dab9-d8a6-4603-8518-e627e97cf0c2" (UID: "abf5dab9-d8a6-4603-8518-e627e97cf0c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:26:04.349962 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:04.349928 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf5dab9-d8a6-4603-8518-e627e97cf0c2-istio-data" (OuterVolumeSpecName: "istio-data") pod "abf5dab9-d8a6-4603-8518-e627e97cf0c2" (UID: "abf5dab9-d8a6-4603-8518-e627e97cf0c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:26:04.351879 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:04.351851 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abf5dab9-d8a6-4603-8518-e627e97cf0c2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "abf5dab9-d8a6-4603-8518-e627e97cf0c2" (UID: "abf5dab9-d8a6-4603-8518-e627e97cf0c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 19:26:04.351968 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:04.351877 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf5dab9-d8a6-4603-8518-e627e97cf0c2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "abf5dab9-d8a6-4603-8518-e627e97cf0c2" (UID: "abf5dab9-d8a6-4603-8518-e627e97cf0c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:26:04.353452 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:04.353401 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf5dab9-d8a6-4603-8518-e627e97cf0c2-istio-token" (OuterVolumeSpecName: "istio-token") pod "abf5dab9-d8a6-4603-8518-e627e97cf0c2" (UID: "abf5dab9-d8a6-4603-8518-e627e97cf0c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:26:04.353727 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:04.353697 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf5dab9-d8a6-4603-8518-e627e97cf0c2-kube-api-access-pqjhv" (OuterVolumeSpecName: "kube-api-access-pqjhv") pod "abf5dab9-d8a6-4603-8518-e627e97cf0c2" (UID: "abf5dab9-d8a6-4603-8518-e627e97cf0c2"). InnerVolumeSpecName "kube-api-access-pqjhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:26:04.447401 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:04.447369 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf5dab9-d8a6-4603-8518-e627e97cf0c2-istiod-ca-cert\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:26:04.447401 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:04.447398 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf5dab9-d8a6-4603-8518-e627e97cf0c2-istio-token\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:26:04.447401 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:04.447406 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf5dab9-d8a6-4603-8518-e627e97cf0c2-credential-socket\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:26:04.447634 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:04.447415 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqjhv\" (UniqueName: \"kubernetes.io/projected/abf5dab9-d8a6-4603-8518-e627e97cf0c2-kube-api-access-pqjhv\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:26:04.447634 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:04.447424 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf5dab9-d8a6-4603-8518-e627e97cf0c2-istio-envoy\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:26:04.447634 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:04.447433 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf5dab9-d8a6-4603-8518-e627e97cf0c2-istio-data\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:26:04.447634 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:04.447454 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf5dab9-d8a6-4603-8518-e627e97cf0c2-workload-certs\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:26:04.447634 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:04.447463 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf5dab9-d8a6-4603-8518-e627e97cf0c2-workload-socket\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:26:04.447634 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:04.447471 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf5dab9-d8a6-4603-8518-e627e97cf0c2-istio-podinfo\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:26:05.177358 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:05.177318 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" event={"ID":"abf5dab9-d8a6-4603-8518-e627e97cf0c2","Type":"ContainerDied","Data":"0872a060df8121622bdd972da9040ff4f1d907f413a39a501acbc48cbf2e9bca"} Jul 10 19:26:05.177358 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:05.177352 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66" Jul 10 19:26:05.177864 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:05.177404 2560 scope.go:117] "RemoveContainer" containerID="defd503a85902f90b35377812a1eb634d733e930da1d8b2a0195df335f2e00a1" Jul 10 19:26:05.200798 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:05.200776 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66"] Jul 10 19:26:05.204261 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:05.204241 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6e61e5c8-openshift-default-c64f9bdcc-w8x66"] Jul 10 19:26:05.252600 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:05.252575 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abf5dab9-d8a6-4603-8518-e627e97cf0c2" path="/var/lib/kubelet/pods/abf5dab9-d8a6-4603-8518-e627e97cf0c2/volumes" Jul 10 19:26:08.167634 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:08.167597 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6402da18-64db9cf95d-ptk56" Jul 10 19:26:10.521302 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:10.521265 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:26:10.622505 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:10.622471 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:26:11.214131 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:11.214098 2560 generic.go:356] "Generic (PLEG): container finished" podID="68df8fbe-2268-46eb-8aa2-228b05b40555" containerID="46663d3586e6ede7143121d8d29e1b312b82593bc967995ba9e106b02e1c5fe4" exitCode=0 Jul 10 19:26:11.214300 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:11.214172 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" event={"ID":"68df8fbe-2268-46eb-8aa2-228b05b40555","Type":"ContainerDied","Data":"46663d3586e6ede7143121d8d29e1b312b82593bc967995ba9e106b02e1c5fe4"} Jul 10 19:26:11.214583 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:11.214568 2560 scope.go:117] "RemoveContainer" containerID="46663d3586e6ede7143121d8d29e1b312b82593bc967995ba9e106b02e1c5fe4" Jul 10 19:26:11.215502 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:11.215474 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:26:11.215568 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:11.215550 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:26:11.215618 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:11.215596 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:26:12.222480 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:12.222420 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" event={"ID":"68df8fbe-2268-46eb-8aa2-228b05b40555","Type":"ContainerStarted","Data":"cd55f31078506601e2eefe794d13d671c216b89d6bbb75099a03e1e9cefd6a87"} Jul 10 19:26:12.588997 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:12.588906 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:26:12.589943 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:12.589920 2560 patch_prober.go:28] interesting pod/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Jul 10 19:26:12.590043 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:12.589976 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" podUID="68df8fbe-2268-46eb-8aa2-228b05b40555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Jul 10 19:26:13.589633 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:13.589603 2560 patch_prober.go:28] interesting pod/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Jul 10 19:26:13.590005 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:13.589663 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" podUID="68df8fbe-2268-46eb-8aa2-228b05b40555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Jul 10 19:26:13.887502 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:13.887469 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:26:14.589408 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:14.589376 2560 patch_prober.go:28] interesting pod/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Jul 10 19:26:14.589607 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:14.589424 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" podUID="68df8fbe-2268-46eb-8aa2-228b05b40555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Jul 10 19:26:15.589175 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:15.589142 2560 patch_prober.go:28] interesting pod/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Jul 10 19:26:15.589577 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:15.589208 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" podUID="68df8fbe-2268-46eb-8aa2-228b05b40555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Jul 10 19:26:16.148249 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:16.148215 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:26:16.592379 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:16.592351 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:26:16.593323 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:26:16.593301 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:27:15.987339 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:15.987303 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6402da18-64db9cf95d-ptk56"] Jul 10 19:27:15.987953 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:15.987562 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6402da18-64db9cf95d-ptk56" podUID="c9c9b7a1-67d4-473d-8c52-0d30a4a89247" containerName="maas-api" containerID="cri-o://14250d229d209886ed9926cf0ec8fbe023774cad11f952a42405a5c9dd456cc2" gracePeriod=30 Jul 10 19:27:16.013991 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:16.013955 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:16.228155 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:16.228131 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6402da18-64db9cf95d-ptk56" Jul 10 19:27:16.290993 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:16.290919 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c9b7a1-67d4-473d-8c52-0d30a4a89247-openshift-service-ca\") pod \"c9c9b7a1-67d4-473d-8c52-0d30a4a89247\" (UID: \"c9c9b7a1-67d4-473d-8c52-0d30a4a89247\") " Jul 10 19:27:16.291130 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:16.291039 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wc8wn\" (UniqueName: \"kubernetes.io/projected/c9c9b7a1-67d4-473d-8c52-0d30a4a89247-kube-api-access-wc8wn\") pod \"c9c9b7a1-67d4-473d-8c52-0d30a4a89247\" (UID: \"c9c9b7a1-67d4-473d-8c52-0d30a4a89247\") " Jul 10 19:27:16.291130 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:16.291109 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c9b7a1-67d4-473d-8c52-0d30a4a89247-maas-api-tls\") pod \"c9c9b7a1-67d4-473d-8c52-0d30a4a89247\" (UID: \"c9c9b7a1-67d4-473d-8c52-0d30a4a89247\") " Jul 10 19:27:16.291245 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:16.291226 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9c9b7a1-67d4-473d-8c52-0d30a4a89247-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c9c9b7a1-67d4-473d-8c52-0d30a4a89247" (UID: "c9c9b7a1-67d4-473d-8c52-0d30a4a89247"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:27:16.291555 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:16.291535 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c9b7a1-67d4-473d-8c52-0d30a4a89247-openshift-service-ca\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:27:16.293046 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:16.293024 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9c9b7a1-67d4-473d-8c52-0d30a4a89247-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c9c9b7a1-67d4-473d-8c52-0d30a4a89247" (UID: "c9c9b7a1-67d4-473d-8c52-0d30a4a89247"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:27:16.293120 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:16.293108 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c9b7a1-67d4-473d-8c52-0d30a4a89247-kube-api-access-wc8wn" (OuterVolumeSpecName: "kube-api-access-wc8wn") pod "c9c9b7a1-67d4-473d-8c52-0d30a4a89247" (UID: "c9c9b7a1-67d4-473d-8c52-0d30a4a89247"). InnerVolumeSpecName "kube-api-access-wc8wn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:27:16.392608 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:16.392575 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c9b7a1-67d4-473d-8c52-0d30a4a89247-maas-api-tls\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:27:16.392608 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:16.392606 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc8wn\" (UniqueName: \"kubernetes.io/projected/c9c9b7a1-67d4-473d-8c52-0d30a4a89247-kube-api-access-wc8wn\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:27:16.549897 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:16.549824 2560 generic.go:356] "Generic (PLEG): container finished" podID="c9c9b7a1-67d4-473d-8c52-0d30a4a89247" containerID="14250d229d209886ed9926cf0ec8fbe023774cad11f952a42405a5c9dd456cc2" exitCode=0 Jul 10 19:27:16.549897 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:16.549869 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6402da18-64db9cf95d-ptk56" Jul 10 19:27:16.549897 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:16.549885 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6402da18-64db9cf95d-ptk56" event={"ID":"c9c9b7a1-67d4-473d-8c52-0d30a4a89247","Type":"ContainerDied","Data":"14250d229d209886ed9926cf0ec8fbe023774cad11f952a42405a5c9dd456cc2"} Jul 10 19:27:16.550160 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:16.549941 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6402da18-64db9cf95d-ptk56" event={"ID":"c9c9b7a1-67d4-473d-8c52-0d30a4a89247","Type":"ContainerDied","Data":"82e7ab38a497a7410e0929fc249128ec4d90fb40ab421a2a6b949b97e84370d6"} Jul 10 19:27:16.550160 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:16.549975 2560 scope.go:117] "RemoveContainer" containerID="14250d229d209886ed9926cf0ec8fbe023774cad11f952a42405a5c9dd456cc2" Jul 10 19:27:16.558942 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:16.558923 2560 scope.go:117] "RemoveContainer" containerID="14250d229d209886ed9926cf0ec8fbe023774cad11f952a42405a5c9dd456cc2" Jul 10 19:27:16.559195 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:27:16.559177 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14250d229d209886ed9926cf0ec8fbe023774cad11f952a42405a5c9dd456cc2\": container with ID starting with 14250d229d209886ed9926cf0ec8fbe023774cad11f952a42405a5c9dd456cc2 not found: ID does not exist" containerID="14250d229d209886ed9926cf0ec8fbe023774cad11f952a42405a5c9dd456cc2" Jul 10 19:27:16.559246 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:16.559201 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14250d229d209886ed9926cf0ec8fbe023774cad11f952a42405a5c9dd456cc2"} err="failed to get container status \"14250d229d209886ed9926cf0ec8fbe023774cad11f952a42405a5c9dd456cc2\": rpc error: code = NotFound desc = could not find container \"14250d229d209886ed9926cf0ec8fbe023774cad11f952a42405a5c9dd456cc2\": container with ID starting with 14250d229d209886ed9926cf0ec8fbe023774cad11f952a42405a5c9dd456cc2 not found: ID does not exist" Jul 10 19:27:16.571315 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:16.571295 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6402da18-64db9cf95d-ptk56"] Jul 10 19:27:16.577185 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:16.577166 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6402da18-64db9cf95d-ptk56"] Jul 10 19:27:17.259503 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:17.259470 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9c9b7a1-67d4-473d-8c52-0d30a4a89247" path="/var/lib/kubelet/pods/c9c9b7a1-67d4-473d-8c52-0d30a4a89247/volumes" Jul 10 19:27:20.590925 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:20.590887 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:26.904158 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:26.904120 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:26.967256 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:26.967211 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:31.554352 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:31.554320 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8"] Jul 10 19:27:31.554715 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:31.554518 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" podUID="68df8fbe-2268-46eb-8aa2-228b05b40555" containerName="istio-proxy" containerID="cri-o://cd55f31078506601e2eefe794d13d671c216b89d6bbb75099a03e1e9cefd6a87" gracePeriod=30 Jul 10 19:27:31.579149 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:31.579121 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:31.594246 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:31.594222 2560 patch_prober.go:28] interesting pod/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 19:27:31.594368 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:31.594284 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" podUID="68df8fbe-2268-46eb-8aa2-228b05b40555" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 19:27:31.835208 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:31.835129 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr"] Jul 10 19:27:31.835686 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:31.835667 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9c9b7a1-67d4-473d-8c52-0d30a4a89247" containerName="maas-api" Jul 10 19:27:31.835686 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:31.835687 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9c9b7a1-67d4-473d-8c52-0d30a4a89247" containerName="maas-api" Jul 10 19:27:31.835788 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:31.835696 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abf5dab9-d8a6-4603-8518-e627e97cf0c2" containerName="istio-proxy" Jul 10 19:27:31.835788 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:31.835702 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="abf5dab9-d8a6-4603-8518-e627e97cf0c2" containerName="istio-proxy" Jul 10 19:27:31.835849 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:31.835800 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9c9b7a1-67d4-473d-8c52-0d30a4a89247" containerName="maas-api" Jul 10 19:27:31.835849 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:31.835810 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="abf5dab9-d8a6-4603-8518-e627e97cf0c2" containerName="istio-proxy" Jul 10 19:27:31.838883 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:31.838864 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:31.841721 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:31.841701 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-2397a366-openshift-default-dockercfg-6fvx6\"" Jul 10 19:27:31.850752 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:31.850733 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr"] Jul 10 19:27:31.868402 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:31.868374 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:31.935568 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:31.935541 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96f76ad3-a4e4-4249-8f1d-380134981727-istio-envoy\") pod \"e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:31.935696 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:31.935590 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f76ad3-a4e4-4249-8f1d-380134981727-credential-socket\") pod \"e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:31.935696 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:31.935622 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9bv6\" (UniqueName: \"kubernetes.io/projected/96f76ad3-a4e4-4249-8f1d-380134981727-kube-api-access-l9bv6\") pod \"e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:31.935696 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:31.935639 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96f76ad3-a4e4-4249-8f1d-380134981727-istio-data\") pod \"e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:31.935696 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:31.935677 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96f76ad3-a4e4-4249-8f1d-380134981727-istio-podinfo\") pod \"e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:31.935696 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:31.935695 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f76ad3-a4e4-4249-8f1d-380134981727-workload-socket\") pod \"e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:31.935896 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:31.935730 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96f76ad3-a4e4-4249-8f1d-380134981727-istiod-ca-cert\") pod \"e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:31.935896 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:31.935770 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96f76ad3-a4e4-4249-8f1d-380134981727-istio-token\") pod \"e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:31.935896 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:31.935801 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96f76ad3-a4e4-4249-8f1d-380134981727-workload-certs\") pod \"e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:32.036222 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:32.036184 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f76ad3-a4e4-4249-8f1d-380134981727-credential-socket\") pod \"e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:32.036222 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:32.036222 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9bv6\" (UniqueName: \"kubernetes.io/projected/96f76ad3-a4e4-4249-8f1d-380134981727-kube-api-access-l9bv6\") pod \"e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:32.036512 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:32.036240 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96f76ad3-a4e4-4249-8f1d-380134981727-istio-data\") pod \"e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:32.036512 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:32.036297 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96f76ad3-a4e4-4249-8f1d-380134981727-istio-podinfo\") pod \"e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:32.036512 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:32.036321 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f76ad3-a4e4-4249-8f1d-380134981727-workload-socket\") pod \"e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:32.036512 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:32.036364 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96f76ad3-a4e4-4249-8f1d-380134981727-istiod-ca-cert\") pod \"e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:32.036512 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:32.036422 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96f76ad3-a4e4-4249-8f1d-380134981727-istio-token\") pod \"e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:32.036512 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:32.036467 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96f76ad3-a4e4-4249-8f1d-380134981727-workload-certs\") pod \"e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:32.036512 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:32.036503 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96f76ad3-a4e4-4249-8f1d-380134981727-istio-envoy\") pod \"e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:32.037229 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:32.036723 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96f76ad3-a4e4-4249-8f1d-380134981727-istio-data\") pod \"e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:32.037229 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:32.036769 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f76ad3-a4e4-4249-8f1d-380134981727-credential-socket\") pod \"e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:32.037229 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:32.036957 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96f76ad3-a4e4-4249-8f1d-380134981727-workload-certs\") pod \"e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:32.037394 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:32.037249 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96f76ad3-a4e4-4249-8f1d-380134981727-istiod-ca-cert\") pod \"e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:32.037394 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:32.037375 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f76ad3-a4e4-4249-8f1d-380134981727-workload-socket\") pod \"e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:32.038735 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:32.038715 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96f76ad3-a4e4-4249-8f1d-380134981727-istio-envoy\") pod \"e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:32.039084 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:32.039067 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96f76ad3-a4e4-4249-8f1d-380134981727-istio-podinfo\") pod \"e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:32.045179 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:32.045153 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9bv6\" (UniqueName: \"kubernetes.io/projected/96f76ad3-a4e4-4249-8f1d-380134981727-kube-api-access-l9bv6\") pod \"e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:32.045725 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:32.045704 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96f76ad3-a4e4-4249-8f1d-380134981727-istio-token\") pod \"e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:32.150311 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:32.150216 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:32.291130 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:32.291102 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr"] Jul 10 19:27:32.292767 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:27:32.292741 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96f76ad3_a4e4_4249_8f1d_380134981727.slice/crio-932b093562b79741083fb2b36137614431e1185c5979a4f6b4431c1bafe98bcc WatchSource:0}: Error finding container 932b093562b79741083fb2b36137614431e1185c5979a4f6b4431c1bafe98bcc: Status 404 returned error can't find the container with id 932b093562b79741083fb2b36137614431e1185c5979a4f6b4431c1bafe98bcc Jul 10 19:27:32.294997 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:32.294956 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:27:32.295070 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:32.295050 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:27:32.295119 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:32.295092 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:27:32.634125 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:32.634089 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" event={"ID":"96f76ad3-a4e4-4249-8f1d-380134981727","Type":"ContainerStarted","Data":"e701754b7ae2dc73524d47fe397124e83d8e4227f5a9c6450fc56243dcd9b173"} Jul 10 19:27:32.634598 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:32.634139 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" event={"ID":"96f76ad3-a4e4-4249-8f1d-380134981727","Type":"ContainerStarted","Data":"932b093562b79741083fb2b36137614431e1185c5979a4f6b4431c1bafe98bcc"} Jul 10 19:27:32.656944 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:32.656893 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" podStartSLOduration=1.656878171 podStartE2EDuration="1.656878171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:27:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:27:32.654026926 +0000 UTC m=+1743.984167758" watchObservedRunningTime="2026-07-10 19:27:32.656878171 +0000 UTC m=+1743.987018980" Jul 10 19:27:32.971154 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:32.971119 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:33.045255 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:33.045217 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:33.045597 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:33.045575 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m"] Jul 10 19:27:33.046030 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:33.046015 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abf5dab9-d8a6-4603-8518-e627e97cf0c2" containerName="istio-proxy" Jul 10 19:27:33.046030 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:33.046031 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="abf5dab9-d8a6-4603-8518-e627e97cf0c2" containerName="istio-proxy" Jul 10 19:27:33.046133 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:33.046123 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="abf5dab9-d8a6-4603-8518-e627e97cf0c2" containerName="istio-proxy" Jul 10 19:27:33.049395 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:33.049378 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m" Jul 10 19:27:33.052366 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:33.052346 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-2397a366-serving-cert\"" Jul 10 19:27:33.052497 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:33.052352 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 19:27:33.052716 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:33.052702 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 19:27:33.052893 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:33.052880 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-44lh8\"" Jul 10 19:27:33.062551 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:33.062528 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m"] Jul 10 19:27:33.140035 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:33.139997 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:33.150476 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:33.150435 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4-openshift-service-ca\") pod \"maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m\" (UID: \"e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m" Jul 10 19:27:33.150590 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:33.150523 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwcdj\" (UniqueName: \"kubernetes.io/projected/e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4-kube-api-access-qwcdj\") pod \"maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m\" (UID: \"e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m" Jul 10 19:27:33.150590 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:33.150555 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:33.150590 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:33.150574 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4-maas-api-tls\") pod \"maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m\" (UID: \"e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m" Jul 10 19:27:33.155097 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:33.155076 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:33.251472 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:33.251341 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4-openshift-service-ca\") pod \"maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m\" (UID: \"e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m" Jul 10 19:27:33.251472 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:33.251421 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwcdj\" (UniqueName: \"kubernetes.io/projected/e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4-kube-api-access-qwcdj\") pod \"maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m\" (UID: \"e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m" Jul 10 19:27:33.251709 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:33.251499 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4-maas-api-tls\") pod \"maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m\" (UID: \"e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m" Jul 10 19:27:33.251709 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:27:33.251650 2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-2397a366-serving-cert: secret "maas-api-e2e-derive-2397a366-serving-cert" not found Jul 10 19:27:33.251830 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:27:33.251712 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4-maas-api-tls podName:e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4 nodeName:}" failed. No retries permitted until 2026-07-10 19:27:33.751691768 +0000 UTC m=+1745.081832561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4-maas-api-tls") pod "maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m" (UID: "e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4") : secret "maas-api-e2e-derive-2397a366-serving-cert" not found Jul 10 19:27:33.251990 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:33.251973 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4-openshift-service-ca\") pod \"maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m\" (UID: \"e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m" Jul 10 19:27:33.274649 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:33.274617 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwcdj\" (UniqueName: \"kubernetes.io/projected/e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4-kube-api-access-qwcdj\") pod \"maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m\" (UID: \"e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m" Jul 10 19:27:33.453817 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:33.453783 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:33.639689 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:33.639612 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:33.640542 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:33.640526 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:33.644795 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:33.644770 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:33.746258 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:33.746229 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:33.756678 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:33.756656 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4-maas-api-tls\") pod \"maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m\" (UID: \"e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m" Jul 10 19:27:33.758999 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:33.758980 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4-maas-api-tls\") pod \"maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m\" (UID: \"e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m" Jul 10 19:27:33.959804 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:33.959771 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m" Jul 10 19:27:34.092196 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:34.092171 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m"] Jul 10 19:27:34.093704 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:27:34.093670 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ae08c2_1f7f_4fe2_897a_fa6b433b25f4.slice/crio-b34d32dc1eac911d4c1d324e80ecce740c07ea187ed3fef05ad96db259d30044 WatchSource:0}: Error finding container b34d32dc1eac911d4c1d324e80ecce740c07ea187ed3fef05ad96db259d30044: Status 404 returned error can't find the container with id b34d32dc1eac911d4c1d324e80ecce740c07ea187ed3fef05ad96db259d30044 Jul 10 19:27:34.645168 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:34.645141 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m" event={"ID":"e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4","Type":"ContainerStarted","Data":"b34d32dc1eac911d4c1d324e80ecce740c07ea187ed3fef05ad96db259d30044"} Jul 10 19:27:35.651556 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:35.651512 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m" event={"ID":"e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4","Type":"ContainerStarted","Data":"e25a155fb092600a80f16f0e8c5a37ba3582be4a4113be7ce22e9b28adfc5ea9"} Jul 10 19:27:35.652016 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:35.651689 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m" Jul 10 19:27:35.670913 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:35.670866 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m" podStartSLOduration=2.17553564 podStartE2EDuration="2.670852134s" totalImagesPullingTime="495.316494ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:27:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:27:35.669623661 +0000 UTC m=+1746.999764495" watchObservedRunningTime="2026-07-10 19:27:35.670852134 +0000 UTC m=+1747.000992942" Jul 10 19:27:36.823149 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:36.823087 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:27:36.888645 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:36.888616 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68df8fbe-2268-46eb-8aa2-228b05b40555-istio-token\") pod \"68df8fbe-2268-46eb-8aa2-228b05b40555\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " Jul 10 19:27:36.888645 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:36.888650 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9tr7\" (UniqueName: \"kubernetes.io/projected/68df8fbe-2268-46eb-8aa2-228b05b40555-kube-api-access-z9tr7\") pod \"68df8fbe-2268-46eb-8aa2-228b05b40555\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " Jul 10 19:27:36.888853 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:36.888712 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68df8fbe-2268-46eb-8aa2-228b05b40555-credential-socket\") pod \"68df8fbe-2268-46eb-8aa2-228b05b40555\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " Jul 10 19:27:36.888853 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:36.888753 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68df8fbe-2268-46eb-8aa2-228b05b40555-istio-data\") pod \"68df8fbe-2268-46eb-8aa2-228b05b40555\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " Jul 10 19:27:36.888853 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:36.888781 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68df8fbe-2268-46eb-8aa2-228b05b40555-istio-envoy\") pod \"68df8fbe-2268-46eb-8aa2-228b05b40555\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " Jul 10 19:27:36.888853 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:36.888827 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68df8fbe-2268-46eb-8aa2-228b05b40555-istio-podinfo\") pod \"68df8fbe-2268-46eb-8aa2-228b05b40555\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " Jul 10 19:27:36.888853 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:36.888851 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68df8fbe-2268-46eb-8aa2-228b05b40555-workload-certs\") pod \"68df8fbe-2268-46eb-8aa2-228b05b40555\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " Jul 10 19:27:36.889051 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:36.888886 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68df8fbe-2268-46eb-8aa2-228b05b40555-istiod-ca-cert\") pod \"68df8fbe-2268-46eb-8aa2-228b05b40555\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " Jul 10 19:27:36.889051 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:36.888959 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68df8fbe-2268-46eb-8aa2-228b05b40555-workload-socket\") pod \"68df8fbe-2268-46eb-8aa2-228b05b40555\" (UID: \"68df8fbe-2268-46eb-8aa2-228b05b40555\") " Jul 10 19:27:36.889051 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:36.889026 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68df8fbe-2268-46eb-8aa2-228b05b40555-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "68df8fbe-2268-46eb-8aa2-228b05b40555" (UID: "68df8fbe-2268-46eb-8aa2-228b05b40555"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:27:36.889217 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:36.889203 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68df8fbe-2268-46eb-8aa2-228b05b40555-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "68df8fbe-2268-46eb-8aa2-228b05b40555" (UID: "68df8fbe-2268-46eb-8aa2-228b05b40555"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:27:36.889385 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:36.889363 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68df8fbe-2268-46eb-8aa2-228b05b40555-credential-socket\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:27:36.889595 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:36.889578 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68df8fbe-2268-46eb-8aa2-228b05b40555-workload-certs\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:27:36.889726 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:36.889432 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68df8fbe-2268-46eb-8aa2-228b05b40555-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "68df8fbe-2268-46eb-8aa2-228b05b40555" (UID: "68df8fbe-2268-46eb-8aa2-228b05b40555"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:27:36.889852 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:36.889587 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68df8fbe-2268-46eb-8aa2-228b05b40555-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "68df8fbe-2268-46eb-8aa2-228b05b40555" (UID: "68df8fbe-2268-46eb-8aa2-228b05b40555"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:27:36.890292 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:36.890263 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68df8fbe-2268-46eb-8aa2-228b05b40555-istio-data" (OuterVolumeSpecName: "istio-data") pod "68df8fbe-2268-46eb-8aa2-228b05b40555" (UID: "68df8fbe-2268-46eb-8aa2-228b05b40555"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:27:36.891378 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:36.891354 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68df8fbe-2268-46eb-8aa2-228b05b40555-kube-api-access-z9tr7" (OuterVolumeSpecName: "kube-api-access-z9tr7") pod "68df8fbe-2268-46eb-8aa2-228b05b40555" (UID: "68df8fbe-2268-46eb-8aa2-228b05b40555"). InnerVolumeSpecName "kube-api-access-z9tr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:27:36.891627 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:36.891608 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68df8fbe-2268-46eb-8aa2-228b05b40555-istio-token" (OuterVolumeSpecName: "istio-token") pod "68df8fbe-2268-46eb-8aa2-228b05b40555" (UID: "68df8fbe-2268-46eb-8aa2-228b05b40555"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:27:36.892257 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:36.892236 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68df8fbe-2268-46eb-8aa2-228b05b40555-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "68df8fbe-2268-46eb-8aa2-228b05b40555" (UID: "68df8fbe-2268-46eb-8aa2-228b05b40555"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 19:27:36.892416 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:36.892394 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68df8fbe-2268-46eb-8aa2-228b05b40555-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "68df8fbe-2268-46eb-8aa2-228b05b40555" (UID: "68df8fbe-2268-46eb-8aa2-228b05b40555"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:27:36.991047 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:36.991012 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68df8fbe-2268-46eb-8aa2-228b05b40555-istio-token\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:27:36.991047 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:36.991039 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9tr7\" (UniqueName: \"kubernetes.io/projected/68df8fbe-2268-46eb-8aa2-228b05b40555-kube-api-access-z9tr7\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:27:36.991047 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:36.991049 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68df8fbe-2268-46eb-8aa2-228b05b40555-istio-data\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:27:36.991266 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:36.991057 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68df8fbe-2268-46eb-8aa2-228b05b40555-istio-envoy\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:27:36.991266 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:36.991065 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68df8fbe-2268-46eb-8aa2-228b05b40555-istio-podinfo\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:27:36.991266 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:36.991072 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68df8fbe-2268-46eb-8aa2-228b05b40555-istiod-ca-cert\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:27:36.991266 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:36.991083 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68df8fbe-2268-46eb-8aa2-228b05b40555-workload-socket\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:27:37.662593 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:37.662557 2560 generic.go:356] "Generic (PLEG): container finished" podID="68df8fbe-2268-46eb-8aa2-228b05b40555" containerID="cd55f31078506601e2eefe794d13d671c216b89d6bbb75099a03e1e9cefd6a87" exitCode=0 Jul 10 19:27:37.662751 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:37.662608 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" Jul 10 19:27:37.662751 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:37.662634 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" event={"ID":"68df8fbe-2268-46eb-8aa2-228b05b40555","Type":"ContainerDied","Data":"cd55f31078506601e2eefe794d13d671c216b89d6bbb75099a03e1e9cefd6a87"} Jul 10 19:27:37.662751 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:37.662688 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8" event={"ID":"68df8fbe-2268-46eb-8aa2-228b05b40555","Type":"ContainerDied","Data":"f9377222924f3168e65abe97419b6e46ad7e8a379af5b4c6f0f01e19e429c788"} Jul 10 19:27:37.662751 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:37.662712 2560 scope.go:117] "RemoveContainer" containerID="cd55f31078506601e2eefe794d13d671c216b89d6bbb75099a03e1e9cefd6a87" Jul 10 19:27:37.671399 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:37.671183 2560 scope.go:117] "RemoveContainer" containerID="46663d3586e6ede7143121d8d29e1b312b82593bc967995ba9e106b02e1c5fe4" Jul 10 19:27:37.678882 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:37.678852 2560 scope.go:117] "RemoveContainer" containerID="cd55f31078506601e2eefe794d13d671c216b89d6bbb75099a03e1e9cefd6a87" Jul 10 19:27:37.679200 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:27:37.679164 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd55f31078506601e2eefe794d13d671c216b89d6bbb75099a03e1e9cefd6a87\": container with ID starting with cd55f31078506601e2eefe794d13d671c216b89d6bbb75099a03e1e9cefd6a87 not found: ID does not exist" containerID="cd55f31078506601e2eefe794d13d671c216b89d6bbb75099a03e1e9cefd6a87" Jul 10 19:27:37.679320 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:37.679203 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd55f31078506601e2eefe794d13d671c216b89d6bbb75099a03e1e9cefd6a87"} err="failed to get container status \"cd55f31078506601e2eefe794d13d671c216b89d6bbb75099a03e1e9cefd6a87\": rpc error: code = NotFound desc = could not find container \"cd55f31078506601e2eefe794d13d671c216b89d6bbb75099a03e1e9cefd6a87\": container with ID starting with cd55f31078506601e2eefe794d13d671c216b89d6bbb75099a03e1e9cefd6a87 not found: ID does not exist" Jul 10 19:27:37.679320 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:37.679236 2560 scope.go:117] "RemoveContainer" containerID="46663d3586e6ede7143121d8d29e1b312b82593bc967995ba9e106b02e1c5fe4" Jul 10 19:27:37.679539 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:27:37.679516 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46663d3586e6ede7143121d8d29e1b312b82593bc967995ba9e106b02e1c5fe4\": container with ID starting with 46663d3586e6ede7143121d8d29e1b312b82593bc967995ba9e106b02e1c5fe4 not found: ID does not exist" containerID="46663d3586e6ede7143121d8d29e1b312b82593bc967995ba9e106b02e1c5fe4" Jul 10 19:27:37.679625 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:37.679550 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46663d3586e6ede7143121d8d29e1b312b82593bc967995ba9e106b02e1c5fe4"} err="failed to get container status \"46663d3586e6ede7143121d8d29e1b312b82593bc967995ba9e106b02e1c5fe4\": rpc error: code = NotFound desc = could not find container \"46663d3586e6ede7143121d8d29e1b312b82593bc967995ba9e106b02e1c5fe4\": container with ID starting with 46663d3586e6ede7143121d8d29e1b312b82593bc967995ba9e106b02e1c5fe4 not found: ID does not exist" Jul 10 19:27:37.681471 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:37.681454 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8"] Jul 10 19:27:37.686102 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:37.686078 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6402da18-openshift-default-69dfbcfc95-59jp8"] Jul 10 19:27:37.950335 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:37.950242 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:38.043399 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:38.042055 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:39.252465 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:39.252420 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68df8fbe-2268-46eb-8aa2-228b05b40555" path="/var/lib/kubelet/pods/68df8fbe-2268-46eb-8aa2-228b05b40555/volumes" Jul 10 19:27:41.401208 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:41.401174 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:41.663378 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:41.663347 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m" Jul 10 19:27:43.696296 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:43.696270 2560 generic.go:356] "Generic (PLEG): container finished" podID="96f76ad3-a4e4-4249-8f1d-380134981727" containerID="e701754b7ae2dc73524d47fe397124e83d8e4227f5a9c6450fc56243dcd9b173" exitCode=0 Jul 10 19:27:43.696657 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:43.696333 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" event={"ID":"96f76ad3-a4e4-4249-8f1d-380134981727","Type":"ContainerDied","Data":"e701754b7ae2dc73524d47fe397124e83d8e4227f5a9c6450fc56243dcd9b173"} Jul 10 19:27:43.696725 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:43.696671 2560 scope.go:117] "RemoveContainer" containerID="e701754b7ae2dc73524d47fe397124e83d8e4227f5a9c6450fc56243dcd9b173" Jul 10 19:27:43.697628 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:43.697598 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:27:43.697718 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:43.697668 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:27:43.697718 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:43.697699 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:27:44.703256 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:44.703218 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" event={"ID":"96f76ad3-a4e4-4249-8f1d-380134981727","Type":"ContainerStarted","Data":"b09c1ae97e7250913e232f567931ed34f4c49c8395fe50be0d9184f00266abfc"} Jul 10 19:27:45.151216 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:45.151127 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:45.152221 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:45.152194 2560 patch_prober.go:28] interesting pod/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 10 19:27:45.152358 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:45.152249 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" podUID="96f76ad3-a4e4-4249-8f1d-380134981727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 10 19:27:46.151062 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:46.151027 2560 patch_prober.go:28] interesting pod/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 10 19:27:46.151481 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:46.151096 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" podUID="96f76ad3-a4e4-4249-8f1d-380134981727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 10 19:27:47.150568 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:47.150531 2560 patch_prober.go:28] interesting pod/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 10 19:27:47.150859 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:47.150602 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" podUID="96f76ad3-a4e4-4249-8f1d-380134981727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 10 19:27:48.150671 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:48.150635 2560 patch_prober.go:28] interesting pod/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 10 19:27:48.151051 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:48.150686 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" podUID="96f76ad3-a4e4-4249-8f1d-380134981727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 10 19:27:48.640486 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:48.640456 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:49.151212 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:49.151173 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:49.154337 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:49.154315 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:49.155283 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:49.155264 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:27:49.253955 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:49.253925 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:55.445205 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:55.445156 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:55.546782 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:27:55.546746 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:29.273648 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:29.273622 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l52t_2a22956a-4e96-41de-945c-a851b587a442/ovn-acl-logging/0.log" Jul 10 19:28:29.275022 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:29.275001 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l52t_2a22956a-4e96-41de-945c-a851b587a442/ovn-acl-logging/0.log" Jul 10 19:28:37.933699 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:37.933651 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m"] Jul 10 19:28:37.934193 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:37.933891 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m" podUID="e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4" containerName="maas-api" containerID="cri-o://e25a155fb092600a80f16f0e8c5a37ba3582be4a4113be7ce22e9b28adfc5ea9" gracePeriod=30 Jul 10 19:28:37.956195 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:37.956167 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:38.184278 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:38.184212 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m" Jul 10 19:28:38.259701 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:38.259670 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4-openshift-service-ca\") pod \"e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4\" (UID: \"e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4\") " Jul 10 19:28:38.259876 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:38.259752 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4-maas-api-tls\") pod \"e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4\" (UID: \"e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4\") " Jul 10 19:28:38.259876 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:38.259781 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwcdj\" (UniqueName: \"kubernetes.io/projected/e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4-kube-api-access-qwcdj\") pod \"e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4\" (UID: \"e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4\") " Jul 10 19:28:38.260003 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:38.259983 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4" (UID: "e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:28:38.261817 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:38.261791 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4" (UID: "e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:28:38.261817 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:38.261806 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4-kube-api-access-qwcdj" (OuterVolumeSpecName: "kube-api-access-qwcdj") pod "e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4" (UID: "e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4"). InnerVolumeSpecName "kube-api-access-qwcdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:28:38.361284 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:38.361235 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwcdj\" (UniqueName: \"kubernetes.io/projected/e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4-kube-api-access-qwcdj\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:28:38.361284 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:38.361282 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4-openshift-service-ca\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:28:38.361284 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:38.361296 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4-maas-api-tls\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:28:38.965575 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:38.965540 2560 generic.go:356] "Generic (PLEG): container finished" podID="e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4" containerID="e25a155fb092600a80f16f0e8c5a37ba3582be4a4113be7ce22e9b28adfc5ea9" exitCode=0 Jul 10 19:28:38.966007 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:38.965586 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m" Jul 10 19:28:38.966007 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:38.965637 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m" event={"ID":"e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4","Type":"ContainerDied","Data":"e25a155fb092600a80f16f0e8c5a37ba3582be4a4113be7ce22e9b28adfc5ea9"} Jul 10 19:28:38.966007 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:38.965697 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m" event={"ID":"e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4","Type":"ContainerDied","Data":"b34d32dc1eac911d4c1d324e80ecce740c07ea187ed3fef05ad96db259d30044"} Jul 10 19:28:38.966007 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:38.965732 2560 scope.go:117] "RemoveContainer" containerID="e25a155fb092600a80f16f0e8c5a37ba3582be4a4113be7ce22e9b28adfc5ea9" Jul 10 19:28:38.974957 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:38.974935 2560 scope.go:117] "RemoveContainer" containerID="e25a155fb092600a80f16f0e8c5a37ba3582be4a4113be7ce22e9b28adfc5ea9" Jul 10 19:28:38.975203 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:28:38.975186 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e25a155fb092600a80f16f0e8c5a37ba3582be4a4113be7ce22e9b28adfc5ea9\": container with ID starting with e25a155fb092600a80f16f0e8c5a37ba3582be4a4113be7ce22e9b28adfc5ea9 not found: ID does not exist" containerID="e25a155fb092600a80f16f0e8c5a37ba3582be4a4113be7ce22e9b28adfc5ea9" Jul 10 19:28:38.975310 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:38.975214 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25a155fb092600a80f16f0e8c5a37ba3582be4a4113be7ce22e9b28adfc5ea9"} err="failed to get container status \"e25a155fb092600a80f16f0e8c5a37ba3582be4a4113be7ce22e9b28adfc5ea9\": rpc error: code = NotFound desc = could not find container \"e25a155fb092600a80f16f0e8c5a37ba3582be4a4113be7ce22e9b28adfc5ea9\": container with ID starting with e25a155fb092600a80f16f0e8c5a37ba3582be4a4113be7ce22e9b28adfc5ea9 not found: ID does not exist" Jul 10 19:28:38.988070 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:38.988048 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m"] Jul 10 19:28:38.992021 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:38.992001 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2397a366-7bfb76bc6b-f692m"] Jul 10 19:28:39.255689 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:39.253646 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4" path="/var/lib/kubelet/pods/e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4/volumes" Jul 10 19:28:42.503813 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:42.503756 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:48.576050 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:48.576022 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:48.633536 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:48.633508 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:52.724208 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:52.724178 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr"] Jul 10 19:28:52.724646 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:52.724399 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" podUID="96f76ad3-a4e4-4249-8f1d-380134981727" containerName="istio-proxy" containerID="cri-o://b09c1ae97e7250913e232f567931ed34f4c49c8395fe50be0d9184f00266abfc" gracePeriod=30 Jul 10 19:28:52.819494 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:52.819463 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:53.300968 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:53.300931 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:53.399347 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:53.399312 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:53.499257 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:53.499222 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:54.976603 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:54.976568 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:57.970251 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:57.970230 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:28:58.042076 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.042045 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96f76ad3-a4e4-4249-8f1d-380134981727-istio-data\") pod \"96f76ad3-a4e4-4249-8f1d-380134981727\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " Jul 10 19:28:58.042201 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.042097 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96f76ad3-a4e4-4249-8f1d-380134981727-istio-envoy\") pod \"96f76ad3-a4e4-4249-8f1d-380134981727\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " Jul 10 19:28:58.042201 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.042120 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96f76ad3-a4e4-4249-8f1d-380134981727-istiod-ca-cert\") pod \"96f76ad3-a4e4-4249-8f1d-380134981727\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " Jul 10 19:28:58.042201 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.042186 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96f76ad3-a4e4-4249-8f1d-380134981727-workload-certs\") pod \"96f76ad3-a4e4-4249-8f1d-380134981727\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " Jul 10 19:28:58.042340 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.042228 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f76ad3-a4e4-4249-8f1d-380134981727-workload-socket\") pod \"96f76ad3-a4e4-4249-8f1d-380134981727\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " Jul 10 19:28:58.042340 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.042270 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96f76ad3-a4e4-4249-8f1d-380134981727-istio-token\") pod \"96f76ad3-a4e4-4249-8f1d-380134981727\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " Jul 10 19:28:58.042340 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.042317 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f76ad3-a4e4-4249-8f1d-380134981727-credential-socket\") pod \"96f76ad3-a4e4-4249-8f1d-380134981727\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " Jul 10 19:28:58.042501 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.042371 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96f76ad3-a4e4-4249-8f1d-380134981727-istio-podinfo\") pod \"96f76ad3-a4e4-4249-8f1d-380134981727\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " Jul 10 19:28:58.042501 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.042422 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9bv6\" (UniqueName: \"kubernetes.io/projected/96f76ad3-a4e4-4249-8f1d-380134981727-kube-api-access-l9bv6\") pod \"96f76ad3-a4e4-4249-8f1d-380134981727\" (UID: \"96f76ad3-a4e4-4249-8f1d-380134981727\") " Jul 10 19:28:58.042868 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.042806 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f76ad3-a4e4-4249-8f1d-380134981727-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "96f76ad3-a4e4-4249-8f1d-380134981727" (UID: "96f76ad3-a4e4-4249-8f1d-380134981727"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:28:58.042868 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.042734 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96f76ad3-a4e4-4249-8f1d-380134981727-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "96f76ad3-a4e4-4249-8f1d-380134981727" (UID: "96f76ad3-a4e4-4249-8f1d-380134981727"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:28:58.042868 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.042864 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f76ad3-a4e4-4249-8f1d-380134981727-istio-data" (OuterVolumeSpecName: "istio-data") pod "96f76ad3-a4e4-4249-8f1d-380134981727" (UID: "96f76ad3-a4e4-4249-8f1d-380134981727"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:28:58.043153 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.043049 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f76ad3-a4e4-4249-8f1d-380134981727-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "96f76ad3-a4e4-4249-8f1d-380134981727" (UID: "96f76ad3-a4e4-4249-8f1d-380134981727"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:28:58.043153 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.043081 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f76ad3-a4e4-4249-8f1d-380134981727-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "96f76ad3-a4e4-4249-8f1d-380134981727" (UID: "96f76ad3-a4e4-4249-8f1d-380134981727"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:28:58.044418 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.044397 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f76ad3-a4e4-4249-8f1d-380134981727-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "96f76ad3-a4e4-4249-8f1d-380134981727" (UID: "96f76ad3-a4e4-4249-8f1d-380134981727"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:28:58.044947 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.044923 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96f76ad3-a4e4-4249-8f1d-380134981727-kube-api-access-l9bv6" (OuterVolumeSpecName: "kube-api-access-l9bv6") pod "96f76ad3-a4e4-4249-8f1d-380134981727" (UID: "96f76ad3-a4e4-4249-8f1d-380134981727"). InnerVolumeSpecName "kube-api-access-l9bv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:28:58.045029 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.044999 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96f76ad3-a4e4-4249-8f1d-380134981727-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "96f76ad3-a4e4-4249-8f1d-380134981727" (UID: "96f76ad3-a4e4-4249-8f1d-380134981727"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 19:28:58.045118 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.045097 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96f76ad3-a4e4-4249-8f1d-380134981727-istio-token" (OuterVolumeSpecName: "istio-token") pod "96f76ad3-a4e4-4249-8f1d-380134981727" (UID: "96f76ad3-a4e4-4249-8f1d-380134981727"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:28:58.062284 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.062261 2560 generic.go:356] "Generic (PLEG): container finished" podID="96f76ad3-a4e4-4249-8f1d-380134981727" containerID="b09c1ae97e7250913e232f567931ed34f4c49c8395fe50be0d9184f00266abfc" exitCode=0 Jul 10 19:28:58.062388 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.062292 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" event={"ID":"96f76ad3-a4e4-4249-8f1d-380134981727","Type":"ContainerDied","Data":"b09c1ae97e7250913e232f567931ed34f4c49c8395fe50be0d9184f00266abfc"} Jul 10 19:28:58.062388 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.062311 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" Jul 10 19:28:58.062388 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.062334 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr" event={"ID":"96f76ad3-a4e4-4249-8f1d-380134981727","Type":"ContainerDied","Data":"932b093562b79741083fb2b36137614431e1185c5979a4f6b4431c1bafe98bcc"} Jul 10 19:28:58.062388 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.062355 2560 scope.go:117] "RemoveContainer" containerID="b09c1ae97e7250913e232f567931ed34f4c49c8395fe50be0d9184f00266abfc" Jul 10 19:28:58.071692 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.071677 2560 scope.go:117] "RemoveContainer" containerID="e701754b7ae2dc73524d47fe397124e83d8e4227f5a9c6450fc56243dcd9b173" Jul 10 19:28:58.080638 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.079746 2560 scope.go:117] "RemoveContainer" containerID="b09c1ae97e7250913e232f567931ed34f4c49c8395fe50be0d9184f00266abfc" Jul 10 19:28:58.080638 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:28:58.079986 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b09c1ae97e7250913e232f567931ed34f4c49c8395fe50be0d9184f00266abfc\": container with ID starting with b09c1ae97e7250913e232f567931ed34f4c49c8395fe50be0d9184f00266abfc not found: ID does not exist" containerID="b09c1ae97e7250913e232f567931ed34f4c49c8395fe50be0d9184f00266abfc" Jul 10 19:28:58.080638 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.080014 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b09c1ae97e7250913e232f567931ed34f4c49c8395fe50be0d9184f00266abfc"} err="failed to get container status \"b09c1ae97e7250913e232f567931ed34f4c49c8395fe50be0d9184f00266abfc\": rpc error: code = NotFound desc = could not find container \"b09c1ae97e7250913e232f567931ed34f4c49c8395fe50be0d9184f00266abfc\": container with ID starting with b09c1ae97e7250913e232f567931ed34f4c49c8395fe50be0d9184f00266abfc not found: ID does not exist" Jul 10 19:28:58.080638 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.080042 2560 scope.go:117] "RemoveContainer" containerID="e701754b7ae2dc73524d47fe397124e83d8e4227f5a9c6450fc56243dcd9b173" Jul 10 19:28:58.080638 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:28:58.080264 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e701754b7ae2dc73524d47fe397124e83d8e4227f5a9c6450fc56243dcd9b173\": container with ID starting with e701754b7ae2dc73524d47fe397124e83d8e4227f5a9c6450fc56243dcd9b173 not found: ID does not exist" containerID="e701754b7ae2dc73524d47fe397124e83d8e4227f5a9c6450fc56243dcd9b173" Jul 10 19:28:58.080638 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.080291 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e701754b7ae2dc73524d47fe397124e83d8e4227f5a9c6450fc56243dcd9b173"} err="failed to get container status \"e701754b7ae2dc73524d47fe397124e83d8e4227f5a9c6450fc56243dcd9b173\": rpc error: code = NotFound desc = could not find container \"e701754b7ae2dc73524d47fe397124e83d8e4227f5a9c6450fc56243dcd9b173\": container with ID starting with e701754b7ae2dc73524d47fe397124e83d8e4227f5a9c6450fc56243dcd9b173 not found: ID does not exist" Jul 10 19:28:58.097168 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.097119 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr"] Jul 10 19:28:58.099665 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.099644 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-2397a366-openshift-default-6ff6f87f7f-gmdmr"] Jul 10 19:28:58.143970 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.143944 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9bv6\" (UniqueName: \"kubernetes.io/projected/96f76ad3-a4e4-4249-8f1d-380134981727-kube-api-access-l9bv6\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:28:58.143970 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.143968 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96f76ad3-a4e4-4249-8f1d-380134981727-istio-data\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:28:58.144101 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.143979 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96f76ad3-a4e4-4249-8f1d-380134981727-istio-envoy\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:28:58.144101 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.143987 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96f76ad3-a4e4-4249-8f1d-380134981727-istiod-ca-cert\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:28:58.144101 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.143994 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96f76ad3-a4e4-4249-8f1d-380134981727-workload-certs\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:28:58.144101 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.144002 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f76ad3-a4e4-4249-8f1d-380134981727-workload-socket\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:28:58.144101 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.144010 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96f76ad3-a4e4-4249-8f1d-380134981727-istio-token\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:28:58.144101 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.144018 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96f76ad3-a4e4-4249-8f1d-380134981727-credential-socket\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:28:58.144101 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:58.144026 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96f76ad3-a4e4-4249-8f1d-380134981727-istio-podinfo\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:28:59.252866 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:59.252837 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96f76ad3-a4e4-4249-8f1d-380134981727" path="/var/lib/kubelet/pods/96f76ad3-a4e4-4249-8f1d-380134981727/volumes" Jul 10 19:28:59.310222 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:28:59.310194 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:29:01.055789 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:29:01.055750 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:29:01.107685 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:29:01.107653 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:29:01.204521 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:29:01.204491 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:29:01.306070 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:29:01.305987 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:29:01.403859 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:29:01.403828 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:29:01.766108 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:29:01.766080 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:29:14.297481 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:29:14.297425 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:29:14.357627 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:29:14.357592 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:29:27.686661 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:29:27.686626 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:29:33.621639 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:29:33.621582 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:29:33.740495 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:29:33.740431 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:29:33.881766 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:29:33.881678 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:29:49.721760 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:29:49.721723 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:29:55.549499 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:29:55.549394 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:29:55.603875 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:29:55.603844 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:29:55.723311 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:29:55.723275 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:29:55.823470 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:29:55.823369 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:00.143110 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:00.143079 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728530-zj5fd"] Jul 10 19:30:00.143554 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:00.143522 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68df8fbe-2268-46eb-8aa2-228b05b40555" containerName="istio-proxy" Jul 10 19:30:00.143554 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:00.143536 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="68df8fbe-2268-46eb-8aa2-228b05b40555" containerName="istio-proxy" Jul 10 19:30:00.143554 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:00.143545 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68df8fbe-2268-46eb-8aa2-228b05b40555" containerName="istio-proxy" Jul 10 19:30:00.143554 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:00.143550 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="68df8fbe-2268-46eb-8aa2-228b05b40555" containerName="istio-proxy" Jul 10 19:30:00.143717 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:00.143581 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96f76ad3-a4e4-4249-8f1d-380134981727" containerName="istio-proxy" Jul 10 19:30:00.143717 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:00.143589 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="96f76ad3-a4e4-4249-8f1d-380134981727" containerName="istio-proxy" Jul 10 19:30:00.143717 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:00.143595 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4" containerName="maas-api" Jul 10 19:30:00.143717 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:00.143600 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4" containerName="maas-api" Jul 10 19:30:00.143717 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:00.143617 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96f76ad3-a4e4-4249-8f1d-380134981727" containerName="istio-proxy" Jul 10 19:30:00.143717 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:00.143623 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="96f76ad3-a4e4-4249-8f1d-380134981727" containerName="istio-proxy" Jul 10 19:30:00.143717 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:00.143698 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="68df8fbe-2268-46eb-8aa2-228b05b40555" containerName="istio-proxy" Jul 10 19:30:00.143717 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:00.143706 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="96f76ad3-a4e4-4249-8f1d-380134981727" containerName="istio-proxy" Jul 10 19:30:00.143717 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:00.143715 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4ae08c2-1f7f-4fe2-897a-fa6b433b25f4" containerName="maas-api" Jul 10 19:30:00.146878 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:00.146862 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728530-zj5fd" Jul 10 19:30:00.149611 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:00.149583 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fsnnw\"" Jul 10 19:30:00.154017 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:00.153996 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728530-zj5fd"] Jul 10 19:30:00.157849 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:00.157829 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728530-zj5fd" Jul 10 19:30:00.287725 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:00.287630 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728530-zj5fd"] Jul 10 19:30:00.290023 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:30:00.289995 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da78f1c_0aff_4a10_adf0_e0186c0c5e34.slice/crio-79439b5551f954f175e82ff6cb62f57c0aaa53f11da11cb55109ebce50934f51 WatchSource:0}: Error finding container 79439b5551f954f175e82ff6cb62f57c0aaa53f11da11cb55109ebce50934f51: Status 404 returned error can't find the container with id 79439b5551f954f175e82ff6cb62f57c0aaa53f11da11cb55109ebce50934f51 Jul 10 19:30:00.360401 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:00.360367 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728530-zj5fd" event={"ID":"0da78f1c-0aff-4a10-adf0-e0186c0c5e34","Type":"ContainerStarted","Data":"79439b5551f954f175e82ff6cb62f57c0aaa53f11da11cb55109ebce50934f51"} Jul 10 19:30:01.366487 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:01.366425 2560 generic.go:356] "Generic (PLEG): container finished" podID="0da78f1c-0aff-4a10-adf0-e0186c0c5e34" containerID="03e32eec0d077ad296086f74f2a0947a90aee79cce788c145efaba86af293404" exitCode=0 Jul 10 19:30:01.366836 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:01.366505 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728530-zj5fd" event={"ID":"0da78f1c-0aff-4a10-adf0-e0186c0c5e34","Type":"ContainerDied","Data":"03e32eec0d077ad296086f74f2a0947a90aee79cce788c145efaba86af293404"} Jul 10 19:30:02.500648 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:02.500620 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728530-zj5fd" Jul 10 19:30:03.377311 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:03.377285 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728530-zj5fd" Jul 10 19:30:03.377500 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:03.377328 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728530-zj5fd" event={"ID":"0da78f1c-0aff-4a10-adf0-e0186c0c5e34","Type":"ContainerDied","Data":"79439b5551f954f175e82ff6cb62f57c0aaa53f11da11cb55109ebce50934f51"} Jul 10 19:30:03.377500 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:03.377369 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79439b5551f954f175e82ff6cb62f57c0aaa53f11da11cb55109ebce50934f51" Jul 10 19:30:06.248808 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:06.248778 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:06.979495 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:06.979455 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:13.062701 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:13.062656 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:13.132082 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:13.132052 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:13.235422 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:13.235391 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:13.350809 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:13.350722 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:14.186887 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:14.186857 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:14.299085 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:14.299052 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:14.398989 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:14.398951 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:15.954625 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:15.954567 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:21.922917 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:21.922880 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:21.982003 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:21.981964 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:22.668926 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:22.668887 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:28.634111 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:28.634077 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:28.734647 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:28.734611 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:28.862488 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:28.862422 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:29.142719 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.142690 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg"] Jul 10 19:30:29.143150 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.143133 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0da78f1c-0aff-4a10-adf0-e0186c0c5e34" containerName="cleanup" Jul 10 19:30:29.143239 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.143152 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0da78f1c-0aff-4a10-adf0-e0186c0c5e34" containerName="cleanup" Jul 10 19:30:29.143294 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.143241 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="68df8fbe-2268-46eb-8aa2-228b05b40555" containerName="istio-proxy" Jul 10 19:30:29.143294 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.143253 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="96f76ad3-a4e4-4249-8f1d-380134981727" containerName="istio-proxy" Jul 10 19:30:29.143294 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.143268 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0da78f1c-0aff-4a10-adf0-e0186c0c5e34" containerName="cleanup" Jul 10 19:30:29.146577 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.146556 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:29.150358 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.150341 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7d2414db-openshift-default-dockercfg-xx5cg\"" Jul 10 19:30:29.161916 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.161892 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg"] Jul 10 19:30:29.170152 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.170132 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:29.245697 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.245659 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-istio-token\") pod \"e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:29.245874 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.245770 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-istio-podinfo\") pod \"e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:29.245874 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.245797 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-istio-data\") pod \"e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:29.245874 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.245823 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-istiod-ca-cert\") pod \"e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:29.245874 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.245862 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj54h\" (UniqueName: \"kubernetes.io/projected/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-kube-api-access-xj54h\") pod \"e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:29.246090 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.245887 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-credential-socket\") pod \"e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:29.246090 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.245920 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-workload-socket\") pod \"e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:29.246090 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.245941 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-istio-envoy\") pod \"e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:29.246090 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.246021 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-workload-certs\") pod \"e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:29.346986 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.346953 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj54h\" (UniqueName: \"kubernetes.io/projected/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-kube-api-access-xj54h\") pod \"e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:29.347157 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.346996 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-credential-socket\") pod \"e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:29.347157 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.347018 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-workload-socket\") pod \"e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:29.347157 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.347124 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-istio-envoy\") pod \"e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:29.347320 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.347201 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-workload-certs\") pod \"e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:29.347320 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.347275 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-istio-token\") pod \"e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:29.347418 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.347329 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-credential-socket\") pod \"e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:29.347497 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.347429 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-istio-podinfo\") pod \"e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:29.347497 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.347471 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-workload-socket\") pod \"e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:29.347497 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.347490 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-istio-data\") pod \"e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:29.347720 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.347517 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-istiod-ca-cert\") pod \"e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:29.347720 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.347544 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-workload-certs\") pod \"e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:29.348032 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.348011 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-istio-data\") pod \"e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:29.348092 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.348033 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-istiod-ca-cert\") pod \"e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:29.349503 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.349481 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-istio-envoy\") pod \"e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:29.349876 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.349855 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-istio-podinfo\") pod \"e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:29.355713 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.355693 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj54h\" (UniqueName: \"kubernetes.io/projected/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-kube-api-access-xj54h\") pod \"e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:29.355803 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.355736 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-istio-token\") pod \"e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:29.456256 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.456230 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:29.592768 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.592742 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg"] Jul 10 19:30:29.594756 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:30:29.594723 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32c278d3_c0d2_487f_8f5a_79ee0e11fa58.slice/crio-7a4ba792d1732000d6060b6a8301ea60f2922684037e94221ba5408a663863c0 WatchSource:0}: Error finding container 7a4ba792d1732000d6060b6a8301ea60f2922684037e94221ba5408a663863c0: Status 404 returned error can't find the container with id 7a4ba792d1732000d6060b6a8301ea60f2922684037e94221ba5408a663863c0 Jul 10 19:30:29.596843 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.596800 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:30:29.596936 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.596870 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:30:29.596936 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.596900 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:30:29.696609 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:29.696575 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:30.512344 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:30.512310 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" event={"ID":"32c278d3-c0d2-487f-8f5a-79ee0e11fa58","Type":"ContainerStarted","Data":"27863b890199e86b74cc594231d6bbc850ae1c8548d358bc9126b859790445dc"} Jul 10 19:30:30.512570 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:30.512368 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" event={"ID":"32c278d3-c0d2-487f-8f5a-79ee0e11fa58","Type":"ContainerStarted","Data":"7a4ba792d1732000d6060b6a8301ea60f2922684037e94221ba5408a663863c0"} Jul 10 19:30:30.534827 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:30.534780 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" podStartSLOduration=1.534764968 podStartE2EDuration="1.534764968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:30:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:30:30.534169451 +0000 UTC m=+1921.864310273" watchObservedRunningTime="2026-07-10 19:30:30.534764968 +0000 UTC m=+1921.864905777" Jul 10 19:30:31.457111 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:31.457081 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:31.461429 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:31.461405 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:31.516983 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:31.516960 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:31.517736 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:31.517715 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:32.450148 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:32.450109 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:32.531724 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:32.531690 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:32.533074 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:32.533048 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8"] Jul 10 19:30:32.536995 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:32.536974 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8" Jul 10 19:30:32.539870 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:32.539848 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7d2414db-serving-cert\"" Jul 10 19:30:32.540023 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:32.539997 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-44lh8\"" Jul 10 19:30:32.540146 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:32.539903 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 19:30:32.541141 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:32.541122 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 19:30:32.546888 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:32.546806 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8"] Jul 10 19:30:32.593031 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:32.592995 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:32.677174 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:32.677137 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e5148c4-78c5-4157-a063-1f61ff3aa2ad-maas-api-tls\") pod \"maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8\" (UID: \"0e5148c4-78c5-4157-a063-1f61ff3aa2ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8" Jul 10 19:30:32.677370 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:32.677185 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e5148c4-78c5-4157-a063-1f61ff3aa2ad-openshift-service-ca\") pod \"maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8\" (UID: \"0e5148c4-78c5-4157-a063-1f61ff3aa2ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8" Jul 10 19:30:32.677370 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:32.677213 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkbfw\" (UniqueName: \"kubernetes.io/projected/0e5148c4-78c5-4157-a063-1f61ff3aa2ad-kube-api-access-rkbfw\") pod \"maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8\" (UID: \"0e5148c4-78c5-4157-a063-1f61ff3aa2ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8" Jul 10 19:30:32.723936 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:32.723837 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:32.778123 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:32.778085 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e5148c4-78c5-4157-a063-1f61ff3aa2ad-maas-api-tls\") pod \"maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8\" (UID: \"0e5148c4-78c5-4157-a063-1f61ff3aa2ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8" Jul 10 19:30:32.778123 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:32.778123 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e5148c4-78c5-4157-a063-1f61ff3aa2ad-openshift-service-ca\") pod \"maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8\" (UID: \"0e5148c4-78c5-4157-a063-1f61ff3aa2ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8" Jul 10 19:30:32.778322 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:32.778146 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkbfw\" (UniqueName: \"kubernetes.io/projected/0e5148c4-78c5-4157-a063-1f61ff3aa2ad-kube-api-access-rkbfw\") pod \"maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8\" (UID: \"0e5148c4-78c5-4157-a063-1f61ff3aa2ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8" Jul 10 19:30:32.778841 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:32.778809 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e5148c4-78c5-4157-a063-1f61ff3aa2ad-openshift-service-ca\") pod \"maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8\" (UID: \"0e5148c4-78c5-4157-a063-1f61ff3aa2ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8" Jul 10 19:30:32.780614 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:32.780594 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e5148c4-78c5-4157-a063-1f61ff3aa2ad-maas-api-tls\") pod \"maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8\" (UID: \"0e5148c4-78c5-4157-a063-1f61ff3aa2ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8" Jul 10 19:30:32.787803 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:32.787778 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkbfw\" (UniqueName: \"kubernetes.io/projected/0e5148c4-78c5-4157-a063-1f61ff3aa2ad-kube-api-access-rkbfw\") pod \"maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8\" (UID: \"0e5148c4-78c5-4157-a063-1f61ff3aa2ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8" Jul 10 19:30:32.852741 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:32.852694 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8" Jul 10 19:30:32.991551 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:32.991520 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8"] Jul 10 19:30:32.992728 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:30:32.992699 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e5148c4_78c5_4157_a063_1f61ff3aa2ad.slice/crio-692c1aa96f0a28c5298a5918b6f1b1b23619ef72555e784275dad6163fd0ebed WatchSource:0}: Error finding container 692c1aa96f0a28c5298a5918b6f1b1b23619ef72555e784275dad6163fd0ebed: Status 404 returned error can't find the container with id 692c1aa96f0a28c5298a5918b6f1b1b23619ef72555e784275dad6163fd0ebed Jul 10 19:30:32.994139 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:32.994122 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 19:30:32.998901 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:32.998879 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:33.099640 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:33.099598 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:33.530086 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:33.530051 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8" event={"ID":"0e5148c4-78c5-4157-a063-1f61ff3aa2ad","Type":"ContainerStarted","Data":"692c1aa96f0a28c5298a5918b6f1b1b23619ef72555e784275dad6163fd0ebed"} Jul 10 19:30:34.537314 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:34.537275 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8" event={"ID":"0e5148c4-78c5-4157-a063-1f61ff3aa2ad","Type":"ContainerStarted","Data":"e7b839a82c16295bc8f7cf2d897ebd5f2cadf3d90e28e1e8a9d15de01b463e15"} Jul 10 19:30:34.537681 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:34.537394 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8" Jul 10 19:30:34.558062 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:34.558018 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8" podStartSLOduration=2.061322708 podStartE2EDuration="2.558006218s" totalImagesPullingTime="496.68351ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:30:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:30:34.557509142 +0000 UTC m=+1925.887649977" watchObservedRunningTime="2026-07-10 19:30:34.558006218 +0000 UTC m=+1925.888147026" Jul 10 19:30:36.920390 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:36.920354 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:37.564147 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:37.564113 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:37.657086 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:37.656985 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:37.975579 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:37.975546 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:40.548246 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:40.548221 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8" Jul 10 19:30:44.072557 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:44.072519 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:44.172577 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:44.172545 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:45.594563 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:45.594532 2560 generic.go:356] "Generic (PLEG): container finished" podID="32c278d3-c0d2-487f-8f5a-79ee0e11fa58" containerID="27863b890199e86b74cc594231d6bbc850ae1c8548d358bc9126b859790445dc" exitCode=0 Jul 10 19:30:45.595053 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:45.594610 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" event={"ID":"32c278d3-c0d2-487f-8f5a-79ee0e11fa58","Type":"ContainerDied","Data":"27863b890199e86b74cc594231d6bbc850ae1c8548d358bc9126b859790445dc"} Jul 10 19:30:45.595126 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:45.595091 2560 scope.go:117] "RemoveContainer" containerID="27863b890199e86b74cc594231d6bbc850ae1c8548d358bc9126b859790445dc" Jul 10 19:30:45.596050 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:45.596024 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:30:45.596118 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:45.596106 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:30:45.596155 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:45.596139 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:30:46.517429 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:46.517394 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:46.601678 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:46.601645 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" event={"ID":"32c278d3-c0d2-487f-8f5a-79ee0e11fa58","Type":"ContainerStarted","Data":"b0eaa3254a71a277fbe54fed0569cedb3fad2625aa2787f07cf0a327479c9690"} Jul 10 19:30:47.456762 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:47.456728 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:47.457938 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:47.457909 2560 patch_prober.go:28] interesting pod/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 10 19:30:47.458046 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:47.457964 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" podUID="32c278d3-c0d2-487f-8f5a-79ee0e11fa58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 10 19:30:48.457600 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:48.457562 2560 patch_prober.go:28] interesting pod/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 10 19:30:48.457975 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:48.457628 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" podUID="32c278d3-c0d2-487f-8f5a-79ee0e11fa58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 10 19:30:49.457337 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:49.457298 2560 patch_prober.go:28] interesting pod/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 10 19:30:49.457565 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:49.457359 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" podUID="32c278d3-c0d2-487f-8f5a-79ee0e11fa58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 10 19:30:50.460491 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:50.460462 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:50.460921 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:50.460820 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:50.461616 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:50.461595 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:30:54.662416 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:54.662378 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8"] Jul 10 19:30:54.663005 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:54.662636 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8" podUID="0e5148c4-78c5-4157-a063-1f61ff3aa2ad" containerName="maas-api" containerID="cri-o://e7b839a82c16295bc8f7cf2d897ebd5f2cadf3d90e28e1e8a9d15de01b463e15" gracePeriod=30 Jul 10 19:30:54.698846 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:54.698815 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:54.919575 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:54.919516 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8" Jul 10 19:30:54.995742 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:54.995715 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e5148c4-78c5-4157-a063-1f61ff3aa2ad-maas-api-tls\") pod \"0e5148c4-78c5-4157-a063-1f61ff3aa2ad\" (UID: \"0e5148c4-78c5-4157-a063-1f61ff3aa2ad\") " Jul 10 19:30:54.995909 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:54.995801 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e5148c4-78c5-4157-a063-1f61ff3aa2ad-openshift-service-ca\") pod \"0e5148c4-78c5-4157-a063-1f61ff3aa2ad\" (UID: \"0e5148c4-78c5-4157-a063-1f61ff3aa2ad\") " Jul 10 19:30:54.995909 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:54.995854 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkbfw\" (UniqueName: \"kubernetes.io/projected/0e5148c4-78c5-4157-a063-1f61ff3aa2ad-kube-api-access-rkbfw\") pod \"0e5148c4-78c5-4157-a063-1f61ff3aa2ad\" (UID: \"0e5148c4-78c5-4157-a063-1f61ff3aa2ad\") " Jul 10 19:30:54.996141 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:54.996120 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e5148c4-78c5-4157-a063-1f61ff3aa2ad-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0e5148c4-78c5-4157-a063-1f61ff3aa2ad" (UID: "0e5148c4-78c5-4157-a063-1f61ff3aa2ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:30:54.997880 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:54.997845 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e5148c4-78c5-4157-a063-1f61ff3aa2ad-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0e5148c4-78c5-4157-a063-1f61ff3aa2ad" (UID: "0e5148c4-78c5-4157-a063-1f61ff3aa2ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:30:54.997998 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:54.997930 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e5148c4-78c5-4157-a063-1f61ff3aa2ad-kube-api-access-rkbfw" (OuterVolumeSpecName: "kube-api-access-rkbfw") pod "0e5148c4-78c5-4157-a063-1f61ff3aa2ad" (UID: "0e5148c4-78c5-4157-a063-1f61ff3aa2ad"). InnerVolumeSpecName "kube-api-access-rkbfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:30:55.096771 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:55.096738 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e5148c4-78c5-4157-a063-1f61ff3aa2ad-openshift-service-ca\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:30:55.096771 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:55.096766 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkbfw\" (UniqueName: \"kubernetes.io/projected/0e5148c4-78c5-4157-a063-1f61ff3aa2ad-kube-api-access-rkbfw\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:30:55.096943 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:55.096778 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e5148c4-78c5-4157-a063-1f61ff3aa2ad-maas-api-tls\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:30:55.648645 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:55.648557 2560 generic.go:356] "Generic (PLEG): container finished" podID="0e5148c4-78c5-4157-a063-1f61ff3aa2ad" containerID="e7b839a82c16295bc8f7cf2d897ebd5f2cadf3d90e28e1e8a9d15de01b463e15" exitCode=0 Jul 10 19:30:55.648645 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:55.648602 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8" Jul 10 19:30:55.648645 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:55.648606 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8" event={"ID":"0e5148c4-78c5-4157-a063-1f61ff3aa2ad","Type":"ContainerDied","Data":"e7b839a82c16295bc8f7cf2d897ebd5f2cadf3d90e28e1e8a9d15de01b463e15"} Jul 10 19:30:55.648889 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:55.648659 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8" event={"ID":"0e5148c4-78c5-4157-a063-1f61ff3aa2ad","Type":"ContainerDied","Data":"692c1aa96f0a28c5298a5918b6f1b1b23619ef72555e784275dad6163fd0ebed"} Jul 10 19:30:55.648889 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:55.648689 2560 scope.go:117] "RemoveContainer" containerID="e7b839a82c16295bc8f7cf2d897ebd5f2cadf3d90e28e1e8a9d15de01b463e15" Jul 10 19:30:55.657122 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:55.657102 2560 scope.go:117] "RemoveContainer" containerID="e7b839a82c16295bc8f7cf2d897ebd5f2cadf3d90e28e1e8a9d15de01b463e15" Jul 10 19:30:55.657350 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:30:55.657334 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7b839a82c16295bc8f7cf2d897ebd5f2cadf3d90e28e1e8a9d15de01b463e15\": container with ID starting with e7b839a82c16295bc8f7cf2d897ebd5f2cadf3d90e28e1e8a9d15de01b463e15 not found: ID does not exist" containerID="e7b839a82c16295bc8f7cf2d897ebd5f2cadf3d90e28e1e8a9d15de01b463e15" Jul 10 19:30:55.657415 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:55.657357 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7b839a82c16295bc8f7cf2d897ebd5f2cadf3d90e28e1e8a9d15de01b463e15"} err="failed to get container status \"e7b839a82c16295bc8f7cf2d897ebd5f2cadf3d90e28e1e8a9d15de01b463e15\": rpc error: code = NotFound desc = could not find container \"e7b839a82c16295bc8f7cf2d897ebd5f2cadf3d90e28e1e8a9d15de01b463e15\": container with ID starting with e7b839a82c16295bc8f7cf2d897ebd5f2cadf3d90e28e1e8a9d15de01b463e15 not found: ID does not exist" Jul 10 19:30:55.671364 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:55.671333 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8"] Jul 10 19:30:55.675209 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:55.675185 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7d2414db-bc89cb995-m4tx8"] Jul 10 19:30:57.252521 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:57.252481 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e5148c4-78c5-4157-a063-1f61ff3aa2ad" path="/var/lib/kubelet/pods/0e5148c4-78c5-4157-a063-1f61ff3aa2ad/volumes" Jul 10 19:30:59.500534 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:30:59.500501 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:05.362310 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:05.362272 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:05.418088 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:05.418058 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:10.182470 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:10.182426 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg"] Jul 10 19:31:10.182887 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:10.182664 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" podUID="32c278d3-c0d2-487f-8f5a-79ee0e11fa58" containerName="istio-proxy" containerID="cri-o://b0eaa3254a71a277fbe54fed0569cedb3fad2625aa2787f07cf0a327479c9690" gracePeriod=30 Jul 10 19:31:10.198336 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:10.198310 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:10.448138 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:10.448041 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:10.579467 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:10.579416 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:10.679046 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:10.679017 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:10.814643 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:10.814567 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:15.428422 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.428397 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:31:15.477535 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.477430 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-istio-envoy\") pod \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " Jul 10 19:31:15.477535 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.477506 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-workload-socket\") pod \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " Jul 10 19:31:15.477756 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.477590 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xj54h\" (UniqueName: \"kubernetes.io/projected/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-kube-api-access-xj54h\") pod \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " Jul 10 19:31:15.477756 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.477624 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-credential-socket\") pod \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " Jul 10 19:31:15.477756 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.477654 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-istio-podinfo\") pod \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " Jul 10 19:31:15.477756 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.477700 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-istio-data\") pod \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " Jul 10 19:31:15.477756 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.477740 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-workload-certs\") pod \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " Jul 10 19:31:15.478020 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.477763 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-istio-token\") pod \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " Jul 10 19:31:15.478020 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.477792 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-istiod-ca-cert\") pod \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\" (UID: \"32c278d3-c0d2-487f-8f5a-79ee0e11fa58\") " Jul 10 19:31:15.478020 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.477805 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "32c278d3-c0d2-487f-8f5a-79ee0e11fa58" (UID: "32c278d3-c0d2-487f-8f5a-79ee0e11fa58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:31:15.478020 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.477928 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "32c278d3-c0d2-487f-8f5a-79ee0e11fa58" (UID: "32c278d3-c0d2-487f-8f5a-79ee0e11fa58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:31:15.478211 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.478040 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "32c278d3-c0d2-487f-8f5a-79ee0e11fa58" (UID: "32c278d3-c0d2-487f-8f5a-79ee0e11fa58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:31:15.478267 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.478221 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-credential-socket\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:31:15.478267 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.478240 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-workload-certs\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:31:15.478267 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.478239 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "32c278d3-c0d2-487f-8f5a-79ee0e11fa58" (UID: "32c278d3-c0d2-487f-8f5a-79ee0e11fa58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:31:15.478267 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.478256 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-workload-socket\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:31:15.478914 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.478875 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-istio-data" (OuterVolumeSpecName: "istio-data") pod "32c278d3-c0d2-487f-8f5a-79ee0e11fa58" (UID: "32c278d3-c0d2-487f-8f5a-79ee0e11fa58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:31:15.479950 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.479922 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-kube-api-access-xj54h" (OuterVolumeSpecName: "kube-api-access-xj54h") pod "32c278d3-c0d2-487f-8f5a-79ee0e11fa58" (UID: "32c278d3-c0d2-487f-8f5a-79ee0e11fa58"). InnerVolumeSpecName "kube-api-access-xj54h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:31:15.480054 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.479992 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "32c278d3-c0d2-487f-8f5a-79ee0e11fa58" (UID: "32c278d3-c0d2-487f-8f5a-79ee0e11fa58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:31:15.480168 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.480145 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "32c278d3-c0d2-487f-8f5a-79ee0e11fa58" (UID: "32c278d3-c0d2-487f-8f5a-79ee0e11fa58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 19:31:15.480308 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.480286 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-istio-token" (OuterVolumeSpecName: "istio-token") pod "32c278d3-c0d2-487f-8f5a-79ee0e11fa58" (UID: "32c278d3-c0d2-487f-8f5a-79ee0e11fa58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:31:15.579408 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.579373 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-istio-envoy\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:31:15.579408 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.579400 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj54h\" (UniqueName: \"kubernetes.io/projected/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-kube-api-access-xj54h\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:31:15.579408 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.579410 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-istio-podinfo\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:31:15.579686 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.579418 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-istio-data\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:31:15.579686 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.579426 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-istio-token\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:31:15.579686 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.579434 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32c278d3-c0d2-487f-8f5a-79ee0e11fa58-istiod-ca-cert\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:31:15.751138 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.751054 2560 generic.go:356] "Generic (PLEG): container finished" podID="32c278d3-c0d2-487f-8f5a-79ee0e11fa58" containerID="b0eaa3254a71a277fbe54fed0569cedb3fad2625aa2787f07cf0a327479c9690" exitCode=0 Jul 10 19:31:15.751138 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.751097 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" event={"ID":"32c278d3-c0d2-487f-8f5a-79ee0e11fa58","Type":"ContainerDied","Data":"b0eaa3254a71a277fbe54fed0569cedb3fad2625aa2787f07cf0a327479c9690"} Jul 10 19:31:15.751138 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.751101 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" Jul 10 19:31:15.751138 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.751138 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg" event={"ID":"32c278d3-c0d2-487f-8f5a-79ee0e11fa58","Type":"ContainerDied","Data":"7a4ba792d1732000d6060b6a8301ea60f2922684037e94221ba5408a663863c0"} Jul 10 19:31:15.751420 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.751160 2560 scope.go:117] "RemoveContainer" containerID="b0eaa3254a71a277fbe54fed0569cedb3fad2625aa2787f07cf0a327479c9690" Jul 10 19:31:15.760883 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.760865 2560 scope.go:117] "RemoveContainer" containerID="27863b890199e86b74cc594231d6bbc850ae1c8548d358bc9126b859790445dc" Jul 10 19:31:15.768505 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.768486 2560 scope.go:117] "RemoveContainer" containerID="b0eaa3254a71a277fbe54fed0569cedb3fad2625aa2787f07cf0a327479c9690" Jul 10 19:31:15.768753 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:31:15.768734 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0eaa3254a71a277fbe54fed0569cedb3fad2625aa2787f07cf0a327479c9690\": container with ID starting with b0eaa3254a71a277fbe54fed0569cedb3fad2625aa2787f07cf0a327479c9690 not found: ID does not exist" containerID="b0eaa3254a71a277fbe54fed0569cedb3fad2625aa2787f07cf0a327479c9690" Jul 10 19:31:15.768808 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.768761 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0eaa3254a71a277fbe54fed0569cedb3fad2625aa2787f07cf0a327479c9690"} err="failed to get container status \"b0eaa3254a71a277fbe54fed0569cedb3fad2625aa2787f07cf0a327479c9690\": rpc error: code = NotFound desc = could not find container \"b0eaa3254a71a277fbe54fed0569cedb3fad2625aa2787f07cf0a327479c9690\": container with ID starting with b0eaa3254a71a277fbe54fed0569cedb3fad2625aa2787f07cf0a327479c9690 not found: ID does not exist" Jul 10 19:31:15.768808 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.768778 2560 scope.go:117] "RemoveContainer" containerID="27863b890199e86b74cc594231d6bbc850ae1c8548d358bc9126b859790445dc" Jul 10 19:31:15.769015 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:31:15.769002 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27863b890199e86b74cc594231d6bbc850ae1c8548d358bc9126b859790445dc\": container with ID starting with 27863b890199e86b74cc594231d6bbc850ae1c8548d358bc9126b859790445dc not found: ID does not exist" containerID="27863b890199e86b74cc594231d6bbc850ae1c8548d358bc9126b859790445dc" Jul 10 19:31:15.769054 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.769019 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27863b890199e86b74cc594231d6bbc850ae1c8548d358bc9126b859790445dc"} err="failed to get container status \"27863b890199e86b74cc594231d6bbc850ae1c8548d358bc9126b859790445dc\": rpc error: code = NotFound desc = could not find container \"27863b890199e86b74cc594231d6bbc850ae1c8548d358bc9126b859790445dc\": container with ID starting with 27863b890199e86b74cc594231d6bbc850ae1c8548d358bc9126b859790445dc not found: ID does not exist" Jul 10 19:31:15.777825 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.777805 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg"] Jul 10 19:31:15.780999 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:15.780981 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7d2414db-openshift-default-69d5588d7f-2vlfg"] Jul 10 19:31:16.746985 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:16.746953 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:16.805923 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:16.805893 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:16.892695 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:16.892660 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:16.990214 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:16.990183 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:17.123162 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:17.123075 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:17.252668 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:17.252636 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32c278d3-c0d2-487f-8f5a-79ee0e11fa58" path="/var/lib/kubelet/pods/32c278d3-c0d2-487f-8f5a-79ee0e11fa58/volumes" Jul 10 19:31:23.093712 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:23.093629 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:23.193598 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:23.193565 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:29.987801 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:29.987764 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:30.301546 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.301463 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv"] Jul 10 19:31:30.301919 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.301898 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32c278d3-c0d2-487f-8f5a-79ee0e11fa58" containerName="istio-proxy" Jul 10 19:31:30.301919 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.301914 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="32c278d3-c0d2-487f-8f5a-79ee0e11fa58" containerName="istio-proxy" Jul 10 19:31:30.302067 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.301932 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e5148c4-78c5-4157-a063-1f61ff3aa2ad" containerName="maas-api" Jul 10 19:31:30.302067 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.301937 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e5148c4-78c5-4157-a063-1f61ff3aa2ad" containerName="maas-api" Jul 10 19:31:30.302067 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.302003 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e5148c4-78c5-4157-a063-1f61ff3aa2ad" containerName="maas-api" Jul 10 19:31:30.302067 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.302011 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="32c278d3-c0d2-487f-8f5a-79ee0e11fa58" containerName="istio-proxy" Jul 10 19:31:30.306453 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.306425 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:30.309592 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.309556 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-cacc33b0-openshift-default-dockercfg-bstkl\"" Jul 10 19:31:30.320427 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.320408 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv"] Jul 10 19:31:30.325538 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.325516 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:30.413292 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.413258 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-istiod-ca-cert\") pod \"e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:30.413493 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.413329 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-workload-socket\") pod \"e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:30.413493 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.413368 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-workload-certs\") pod \"e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:30.413493 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.413392 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-istio-podinfo\") pod \"e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:30.413493 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.413418 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-istio-token\") pod \"e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:30.413680 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.413490 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r227v\" (UniqueName: \"kubernetes.io/projected/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-kube-api-access-r227v\") pod \"e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:30.413680 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.413543 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-credential-socket\") pod \"e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:30.413680 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.413585 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-istio-envoy\") pod \"e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:30.413680 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.413610 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-istio-data\") pod \"e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:30.448115 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.448081 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:30.514306 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.514276 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-workload-socket\") pod \"e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:30.514306 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.514310 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-workload-certs\") pod \"e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:30.514551 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.514327 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-istio-podinfo\") pod \"e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:30.514551 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.514349 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-istio-token\") pod \"e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:30.514551 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.514370 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r227v\" (UniqueName: \"kubernetes.io/projected/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-kube-api-access-r227v\") pod \"e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:30.514551 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.514388 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-credential-socket\") pod \"e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:30.514551 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.514414 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-istio-envoy\") pod \"e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:30.514551 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.514466 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-istio-data\") pod \"e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:30.514826 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.514569 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-istiod-ca-cert\") pod \"e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:30.514826 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.514790 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-workload-certs\") pod \"e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:30.514956 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.514920 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-istio-data\") pod \"e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:30.515011 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.514936 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-workload-socket\") pod \"e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:30.515011 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.514965 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-credential-socket\") pod \"e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:30.515263 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.515246 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-istiod-ca-cert\") pod \"e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:30.516596 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.516576 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-istio-envoy\") pod \"e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:30.516947 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.516925 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-istio-podinfo\") pod \"e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:30.526097 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.526068 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r227v\" (UniqueName: \"kubernetes.io/projected/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-kube-api-access-r227v\") pod \"e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:30.526185 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.526159 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-istio-token\") pod \"e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:30.616631 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.616539 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:30.750808 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.750731 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv"] Jul 10 19:31:30.753333 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:31:30.753310 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6eb7104_6679_4ecf_8f08_b61a3ec8233a.slice/crio-89bdad7bc9f4d755aba9e76797aa008981a58b55efcf47ba5839e74ef934dbd5 WatchSource:0}: Error finding container 89bdad7bc9f4d755aba9e76797aa008981a58b55efcf47ba5839e74ef934dbd5: Status 404 returned error can't find the container with id 89bdad7bc9f4d755aba9e76797aa008981a58b55efcf47ba5839e74ef934dbd5 Jul 10 19:31:30.755346 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.755314 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:31:30.755464 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.755387 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:31:30.755464 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.755429 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:31:30.830663 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.830189 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" event={"ID":"a6eb7104-6679-4ecf-8f08-b61a3ec8233a","Type":"ContainerStarted","Data":"f4fccafbc7db9027de949e9f87ee4afa0d45dadff780ffb37dd6f3adf8d94494"} Jul 10 19:31:30.830663 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.830281 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" event={"ID":"a6eb7104-6679-4ecf-8f08-b61a3ec8233a","Type":"ContainerStarted","Data":"89bdad7bc9f4d755aba9e76797aa008981a58b55efcf47ba5839e74ef934dbd5"} Jul 10 19:31:30.856508 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:30.856433 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" podStartSLOduration=0.856413906 podStartE2EDuration="856.413906ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:31:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:31:30.854047997 +0000 UTC m=+1982.184188807" watchObservedRunningTime="2026-07-10 19:31:30.856413906 +0000 UTC m=+1982.186554714" Jul 10 19:31:31.617533 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:31.617501 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:31.621460 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:31.621428 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:31.835404 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:31.835376 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:31.836241 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:31.836223 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:32.797148 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:32.797115 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:32.866395 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:32.866362 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cacc33b0-89456f87-dbtnh"] Jul 10 19:31:32.866821 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:32.866805 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32c278d3-c0d2-487f-8f5a-79ee0e11fa58" containerName="istio-proxy" Jul 10 19:31:32.866821 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:32.866821 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="32c278d3-c0d2-487f-8f5a-79ee0e11fa58" containerName="istio-proxy" Jul 10 19:31:32.866971 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:32.866887 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="32c278d3-c0d2-487f-8f5a-79ee0e11fa58" containerName="istio-proxy" Jul 10 19:31:32.869323 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:32.869288 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:32.869905 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:32.869890 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cacc33b0-89456f87-dbtnh" Jul 10 19:31:32.872780 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:32.872761 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 19:31:32.874135 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:32.874117 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 19:31:32.874241 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:32.874149 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-cacc33b0-serving-cert\"" Jul 10 19:31:32.874241 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:32.874151 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-44lh8\"" Jul 10 19:31:32.880693 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:32.880673 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cacc33b0-89456f87-dbtnh"] Jul 10 19:31:32.932308 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:32.932278 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:33.043212 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:33.043175 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbr96\" (UniqueName: \"kubernetes.io/projected/1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2-kube-api-access-qbr96\") pod \"maas-api-e2e-mt-cacc33b0-89456f87-dbtnh\" (UID: \"1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cacc33b0-89456f87-dbtnh" Jul 10 19:31:33.043384 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:33.043237 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2-maas-api-tls\") pod \"maas-api-e2e-mt-cacc33b0-89456f87-dbtnh\" (UID: \"1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cacc33b0-89456f87-dbtnh" Jul 10 19:31:33.043384 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:33.043336 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2-openshift-service-ca\") pod \"maas-api-e2e-mt-cacc33b0-89456f87-dbtnh\" (UID: \"1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cacc33b0-89456f87-dbtnh" Jul 10 19:31:33.137752 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:33.137666 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:33.144837 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:33.144812 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbr96\" (UniqueName: \"kubernetes.io/projected/1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2-kube-api-access-qbr96\") pod \"maas-api-e2e-mt-cacc33b0-89456f87-dbtnh\" (UID: \"1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cacc33b0-89456f87-dbtnh" Jul 10 19:31:33.144948 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:33.144865 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2-maas-api-tls\") pod \"maas-api-e2e-mt-cacc33b0-89456f87-dbtnh\" (UID: \"1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cacc33b0-89456f87-dbtnh" Jul 10 19:31:33.144948 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:33.144919 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2-openshift-service-ca\") pod \"maas-api-e2e-mt-cacc33b0-89456f87-dbtnh\" (UID: \"1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cacc33b0-89456f87-dbtnh" Jul 10 19:31:33.145541 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:33.145524 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2-openshift-service-ca\") pod \"maas-api-e2e-mt-cacc33b0-89456f87-dbtnh\" (UID: \"1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cacc33b0-89456f87-dbtnh" Jul 10 19:31:33.147330 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:33.147308 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2-maas-api-tls\") pod \"maas-api-e2e-mt-cacc33b0-89456f87-dbtnh\" (UID: \"1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cacc33b0-89456f87-dbtnh" Jul 10 19:31:33.160349 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:33.160329 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbr96\" (UniqueName: \"kubernetes.io/projected/1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2-kube-api-access-qbr96\") pod \"maas-api-e2e-mt-cacc33b0-89456f87-dbtnh\" (UID: \"1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cacc33b0-89456f87-dbtnh" Jul 10 19:31:33.182266 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:33.182247 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cacc33b0-89456f87-dbtnh" Jul 10 19:31:33.314919 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:33.314893 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cacc33b0-89456f87-dbtnh"] Jul 10 19:31:33.316614 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:31:33.316579 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd6b2ca_958d_4f59_85f8_8bab0b6cbab2.slice/crio-0b398e2200b4d18d2c2f19253848e45048d73a07338b61cd5e9837879bffcfc4 WatchSource:0}: Error finding container 0b398e2200b4d18d2c2f19253848e45048d73a07338b61cd5e9837879bffcfc4: Status 404 returned error can't find the container with id 0b398e2200b4d18d2c2f19253848e45048d73a07338b61cd5e9837879bffcfc4 Jul 10 19:31:33.426972 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:33.426940 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:33.529277 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:33.529252 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:33.846635 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:33.846606 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cacc33b0-89456f87-dbtnh" event={"ID":"1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2","Type":"ContainerStarted","Data":"0b398e2200b4d18d2c2f19253848e45048d73a07338b61cd5e9837879bffcfc4"} Jul 10 19:31:34.853076 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:34.853040 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cacc33b0-89456f87-dbtnh" event={"ID":"1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2","Type":"ContainerStarted","Data":"1b2cf2c7c530914fa92e21f782f0ae25a584c8e86584ca952d716a28c358da9f"} Jul 10 19:31:34.853480 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:34.853097 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cacc33b0-89456f87-dbtnh" Jul 10 19:31:34.872215 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:34.872162 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cacc33b0-89456f87-dbtnh" podStartSLOduration=2.397507715 podStartE2EDuration="2.872147475s" totalImagesPullingTime="474.63976ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:31:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:31:34.871592931 +0000 UTC m=+1986.201733741" watchObservedRunningTime="2026-07-10 19:31:34.872147475 +0000 UTC m=+1986.202288283" Jul 10 19:31:35.552492 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:35.552429 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:37.312983 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:37.312954 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:40.863253 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:40.863224 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cacc33b0-89456f87-dbtnh" Jul 10 19:31:46.836738 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:46.836706 2560 patch_prober.go:28] interesting pod/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 10 19:31:46.837140 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:46.836769 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" podUID="a6eb7104-6679-4ecf-8f08-b61a3ec8233a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 10 19:31:46.919485 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:46.919434 2560 generic.go:356] "Generic (PLEG): container finished" podID="a6eb7104-6679-4ecf-8f08-b61a3ec8233a" containerID="f4fccafbc7db9027de949e9f87ee4afa0d45dadff780ffb37dd6f3adf8d94494" exitCode=0 Jul 10 19:31:46.919641 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:46.919509 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" event={"ID":"a6eb7104-6679-4ecf-8f08-b61a3ec8233a","Type":"ContainerDied","Data":"f4fccafbc7db9027de949e9f87ee4afa0d45dadff780ffb37dd6f3adf8d94494"} Jul 10 19:31:46.919927 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:46.919911 2560 scope.go:117] "RemoveContainer" containerID="f4fccafbc7db9027de949e9f87ee4afa0d45dadff780ffb37dd6f3adf8d94494" Jul 10 19:31:46.920916 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:46.920888 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:31:46.921002 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:46.920956 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:31:46.921002 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:46.920992 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:31:47.929948 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:47.929907 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" event={"ID":"a6eb7104-6679-4ecf-8f08-b61a3ec8233a","Type":"ContainerStarted","Data":"febd35582a167f164b8ec37fe1913cd983460578727a03724d947d2abb5ca81a"} Jul 10 19:31:48.617666 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:48.617633 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:48.617898 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:48.617876 2560 patch_prober.go:28] interesting pod/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 10 19:31:48.617985 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:48.617930 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" podUID="a6eb7104-6679-4ecf-8f08-b61a3ec8233a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 10 19:31:49.617235 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:49.617203 2560 patch_prober.go:28] interesting pod/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 10 19:31:49.617646 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:49.617257 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" podUID="a6eb7104-6679-4ecf-8f08-b61a3ec8233a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 10 19:31:50.617679 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:50.617642 2560 patch_prober.go:28] interesting pod/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 10 19:31:50.618058 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:50.617708 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" podUID="a6eb7104-6679-4ecf-8f08-b61a3ec8233a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 10 19:31:51.616964 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:51.616931 2560 patch_prober.go:28] interesting pod/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 10 19:31:51.617137 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:51.616999 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" podUID="a6eb7104-6679-4ecf-8f08-b61a3ec8233a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 10 19:31:52.621002 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:52.620973 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:52.621345 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:52.621233 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:31:52.621802 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:31:52.621783 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:32:00.578634 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:32:00.578586 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:32:36.242663 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:32:36.242626 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cacc33b0-89456f87-dbtnh"] Jul 10 19:32:36.243246 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:32:36.242852 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cacc33b0-89456f87-dbtnh" podUID="1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2" containerName="maas-api" containerID="cri-o://1b2cf2c7c530914fa92e21f782f0ae25a584c8e86584ca952d716a28c358da9f" gracePeriod=30 Jul 10 19:32:36.251908 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:32:36.251873 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:32:36.495156 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:32:36.495094 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cacc33b0-89456f87-dbtnh" Jul 10 19:32:36.624812 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:32:36.624776 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2-openshift-service-ca\") pod \"1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2\" (UID: \"1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2\") " Jul 10 19:32:36.624941 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:32:36.624855 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbr96\" (UniqueName: \"kubernetes.io/projected/1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2-kube-api-access-qbr96\") pod \"1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2\" (UID: \"1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2\") " Jul 10 19:32:36.624991 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:32:36.624954 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2-maas-api-tls\") pod \"1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2\" (UID: \"1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2\") " Jul 10 19:32:36.625141 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:32:36.625120 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2" (UID: "1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:32:36.625322 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:32:36.625306 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2-openshift-service-ca\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:32:36.626781 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:32:36.626759 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2" (UID: "1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:32:36.626894 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:32:36.626871 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2-kube-api-access-qbr96" (OuterVolumeSpecName: "kube-api-access-qbr96") pod "1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2" (UID: "1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2"). InnerVolumeSpecName "kube-api-access-qbr96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:32:36.726190 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:32:36.726159 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbr96\" (UniqueName: \"kubernetes.io/projected/1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2-kube-api-access-qbr96\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:32:36.726190 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:32:36.726184 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2-maas-api-tls\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:32:37.174855 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:32:37.174822 2560 generic.go:356] "Generic (PLEG): container finished" podID="1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2" containerID="1b2cf2c7c530914fa92e21f782f0ae25a584c8e86584ca952d716a28c358da9f" exitCode=0 Jul 10 19:32:37.175014 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:32:37.174860 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cacc33b0-89456f87-dbtnh" Jul 10 19:32:37.175014 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:32:37.174906 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cacc33b0-89456f87-dbtnh" event={"ID":"1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2","Type":"ContainerDied","Data":"1b2cf2c7c530914fa92e21f782f0ae25a584c8e86584ca952d716a28c358da9f"} Jul 10 19:32:37.175014 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:32:37.174998 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cacc33b0-89456f87-dbtnh" event={"ID":"1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2","Type":"ContainerDied","Data":"0b398e2200b4d18d2c2f19253848e45048d73a07338b61cd5e9837879bffcfc4"} Jul 10 19:32:37.175133 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:32:37.175029 2560 scope.go:117] "RemoveContainer" containerID="1b2cf2c7c530914fa92e21f782f0ae25a584c8e86584ca952d716a28c358da9f" Jul 10 19:32:37.184095 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:32:37.184081 2560 scope.go:117] "RemoveContainer" containerID="1b2cf2c7c530914fa92e21f782f0ae25a584c8e86584ca952d716a28c358da9f" Jul 10 19:32:37.184334 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:32:37.184315 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b2cf2c7c530914fa92e21f782f0ae25a584c8e86584ca952d716a28c358da9f\": container with ID starting with 1b2cf2c7c530914fa92e21f782f0ae25a584c8e86584ca952d716a28c358da9f not found: ID does not exist" containerID="1b2cf2c7c530914fa92e21f782f0ae25a584c8e86584ca952d716a28c358da9f" Jul 10 19:32:37.184372 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:32:37.184343 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b2cf2c7c530914fa92e21f782f0ae25a584c8e86584ca952d716a28c358da9f"} err="failed to get container status \"1b2cf2c7c530914fa92e21f782f0ae25a584c8e86584ca952d716a28c358da9f\": rpc error: code = NotFound desc = could not find container \"1b2cf2c7c530914fa92e21f782f0ae25a584c8e86584ca952d716a28c358da9f\": container with ID starting with 1b2cf2c7c530914fa92e21f782f0ae25a584c8e86584ca952d716a28c358da9f not found: ID does not exist" Jul 10 19:32:37.196885 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:32:37.196864 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cacc33b0-89456f87-dbtnh"] Jul 10 19:32:37.202528 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:32:37.202506 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cacc33b0-89456f87-dbtnh"] Jul 10 19:32:37.251897 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:32:37.251867 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2" path="/var/lib/kubelet/pods/1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2/volumes" Jul 10 19:32:37.331898 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:32:37.331878 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:32:40.826085 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:32:40.826053 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:32:45.859326 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:32:45.859288 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:32:49.903348 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:32:49.903306 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:32:50.004221 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:32:50.004193 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:32:55.886868 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:32:55.886786 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:22.287899 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:22.287856 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:23.205805 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:23.205770 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:28.970789 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:28.970749 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:29.071253 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:29.071217 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:29.303501 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:29.303412 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l52t_2a22956a-4e96-41de-945c-a851b587a442/ovn-acl-logging/0.log" Jul 10 19:33:29.304508 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:29.304487 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l52t_2a22956a-4e96-41de-945c-a851b587a442/ovn-acl-logging/0.log" Jul 10 19:33:32.965460 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:32.965407 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv"] Jul 10 19:33:32.965905 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:32.965762 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" podUID="a6eb7104-6679-4ecf-8f08-b61a3ec8233a" containerName="istio-proxy" containerID="cri-o://febd35582a167f164b8ec37fe1913cd983460578727a03724d947d2abb5ca81a" gracePeriod=30 Jul 10 19:33:32.987920 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:32.987885 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:33.213457 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:33.213405 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:34.206536 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:34.206499 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:34.315461 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:34.315403 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:34.406776 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:34.406747 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:35.034942 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:35.034914 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:35.133917 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:35.133881 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:35.234391 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:35.234366 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:36.517220 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:36.517178 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:37.623083 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:37.623052 2560 patch_prober.go:28] interesting pod/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 19:33:37.623472 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:37.623113 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" podUID="a6eb7104-6679-4ecf-8f08-b61a3ec8233a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 19:33:38.203790 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.203770 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:33:38.331988 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.331887 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-istiod-ca-cert\") pod \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " Jul 10 19:33:38.331988 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.331939 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-credential-socket\") pod \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " Jul 10 19:33:38.331988 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.331985 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-istio-data\") pod \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " Jul 10 19:33:38.332277 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.332063 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-istio-podinfo\") pod \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " Jul 10 19:33:38.332277 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.332142 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r227v\" (UniqueName: \"kubernetes.io/projected/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-kube-api-access-r227v\") pod \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " Jul 10 19:33:38.332277 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.332174 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-istio-token\") pod \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " Jul 10 19:33:38.332277 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.332197 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-istio-envoy\") pod \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " Jul 10 19:33:38.332277 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.332228 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-workload-socket\") pod \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " Jul 10 19:33:38.332277 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.332230 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a6eb7104-6679-4ecf-8f08-b61a3ec8233a" (UID: "a6eb7104-6679-4ecf-8f08-b61a3ec8233a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:33:38.332277 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.332272 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-workload-certs\") pod \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\" (UID: \"a6eb7104-6679-4ecf-8f08-b61a3ec8233a\") " Jul 10 19:33:38.332677 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.332286 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a6eb7104-6679-4ecf-8f08-b61a3ec8233a" (UID: "a6eb7104-6679-4ecf-8f08-b61a3ec8233a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:33:38.332737 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.332672 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-istiod-ca-cert\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:33:38.332737 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.332692 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-credential-socket\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:33:38.332737 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.332721 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a6eb7104-6679-4ecf-8f08-b61a3ec8233a" (UID: "a6eb7104-6679-4ecf-8f08-b61a3ec8233a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:33:38.333634 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.333107 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-istio-data" (OuterVolumeSpecName: "istio-data") pod "a6eb7104-6679-4ecf-8f08-b61a3ec8233a" (UID: "a6eb7104-6679-4ecf-8f08-b61a3ec8233a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:33:38.333634 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.333620 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a6eb7104-6679-4ecf-8f08-b61a3ec8233a" (UID: "a6eb7104-6679-4ecf-8f08-b61a3ec8233a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:33:38.335239 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.335214 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-kube-api-access-r227v" (OuterVolumeSpecName: "kube-api-access-r227v") pod "a6eb7104-6679-4ecf-8f08-b61a3ec8233a" (UID: "a6eb7104-6679-4ecf-8f08-b61a3ec8233a"). InnerVolumeSpecName "kube-api-access-r227v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:33:38.336677 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.336618 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a6eb7104-6679-4ecf-8f08-b61a3ec8233a" (UID: "a6eb7104-6679-4ecf-8f08-b61a3ec8233a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:33:38.336677 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.336636 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-istio-token" (OuterVolumeSpecName: "istio-token") pod "a6eb7104-6679-4ecf-8f08-b61a3ec8233a" (UID: "a6eb7104-6679-4ecf-8f08-b61a3ec8233a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:33:38.336677 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.336618 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a6eb7104-6679-4ecf-8f08-b61a3ec8233a" (UID: "a6eb7104-6679-4ecf-8f08-b61a3ec8233a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 19:33:38.433790 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.433766 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-workload-certs\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:33:38.433790 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.433786 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-istio-data\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:33:38.433911 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.433794 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-istio-podinfo\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:33:38.433911 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.433802 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r227v\" (UniqueName: \"kubernetes.io/projected/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-kube-api-access-r227v\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:33:38.433911 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.433810 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-istio-token\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:33:38.433911 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.433818 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-istio-envoy\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:33:38.433911 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.433826 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eb7104-6679-4ecf-8f08-b61a3ec8233a-workload-socket\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:33:38.467643 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.467609 2560 generic.go:356] "Generic (PLEG): container finished" podID="a6eb7104-6679-4ecf-8f08-b61a3ec8233a" containerID="febd35582a167f164b8ec37fe1913cd983460578727a03724d947d2abb5ca81a" exitCode=0 Jul 10 19:33:38.467735 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.467662 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" Jul 10 19:33:38.467735 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.467678 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" event={"ID":"a6eb7104-6679-4ecf-8f08-b61a3ec8233a","Type":"ContainerDied","Data":"febd35582a167f164b8ec37fe1913cd983460578727a03724d947d2abb5ca81a"} Jul 10 19:33:38.467735 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.467730 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv" event={"ID":"a6eb7104-6679-4ecf-8f08-b61a3ec8233a","Type":"ContainerDied","Data":"89bdad7bc9f4d755aba9e76797aa008981a58b55efcf47ba5839e74ef934dbd5"} Jul 10 19:33:38.467889 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.467771 2560 scope.go:117] "RemoveContainer" containerID="febd35582a167f164b8ec37fe1913cd983460578727a03724d947d2abb5ca81a" Jul 10 19:33:38.476351 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.476331 2560 scope.go:117] "RemoveContainer" containerID="f4fccafbc7db9027de949e9f87ee4afa0d45dadff780ffb37dd6f3adf8d94494" Jul 10 19:33:38.483811 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.483794 2560 scope.go:117] "RemoveContainer" containerID="febd35582a167f164b8ec37fe1913cd983460578727a03724d947d2abb5ca81a" Jul 10 19:33:38.484101 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:33:38.484084 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"febd35582a167f164b8ec37fe1913cd983460578727a03724d947d2abb5ca81a\": container with ID starting with febd35582a167f164b8ec37fe1913cd983460578727a03724d947d2abb5ca81a not found: ID does not exist" containerID="febd35582a167f164b8ec37fe1913cd983460578727a03724d947d2abb5ca81a" Jul 10 19:33:38.484193 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.484111 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"febd35582a167f164b8ec37fe1913cd983460578727a03724d947d2abb5ca81a"} err="failed to get container status \"febd35582a167f164b8ec37fe1913cd983460578727a03724d947d2abb5ca81a\": rpc error: code = NotFound desc = could not find container \"febd35582a167f164b8ec37fe1913cd983460578727a03724d947d2abb5ca81a\": container with ID starting with febd35582a167f164b8ec37fe1913cd983460578727a03724d947d2abb5ca81a not found: ID does not exist" Jul 10 19:33:38.484193 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.484133 2560 scope.go:117] "RemoveContainer" containerID="f4fccafbc7db9027de949e9f87ee4afa0d45dadff780ffb37dd6f3adf8d94494" Jul 10 19:33:38.484386 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:33:38.484363 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4fccafbc7db9027de949e9f87ee4afa0d45dadff780ffb37dd6f3adf8d94494\": container with ID starting with f4fccafbc7db9027de949e9f87ee4afa0d45dadff780ffb37dd6f3adf8d94494 not found: ID does not exist" containerID="f4fccafbc7db9027de949e9f87ee4afa0d45dadff780ffb37dd6f3adf8d94494" Jul 10 19:33:38.484458 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.484394 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4fccafbc7db9027de949e9f87ee4afa0d45dadff780ffb37dd6f3adf8d94494"} err="failed to get container status \"f4fccafbc7db9027de949e9f87ee4afa0d45dadff780ffb37dd6f3adf8d94494\": rpc error: code = NotFound desc = could not find container \"f4fccafbc7db9027de949e9f87ee4afa0d45dadff780ffb37dd6f3adf8d94494\": container with ID starting with f4fccafbc7db9027de949e9f87ee4afa0d45dadff780ffb37dd6f3adf8d94494 not found: ID does not exist" Jul 10 19:33:38.490277 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.490255 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv"] Jul 10 19:33:38.494917 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:38.494898 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-cacc33b0-openshift-default-77fc76d695-22wrv"] Jul 10 19:33:39.252174 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:39.252144 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6eb7104-6679-4ecf-8f08-b61a3ec8233a" path="/var/lib/kubelet/pods/a6eb7104-6679-4ecf-8f08-b61a3ec8233a/volumes" Jul 10 19:33:42.872597 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:42.872559 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:42.973199 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:42.973147 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:43.519963 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:43.519934 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:50.590601 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:50.590567 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:50.652692 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:50.652667 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:50.759576 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:50.759546 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:50.859573 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:33:50.859493 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:01.257380 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:01.257334 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:02.899846 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:02.899815 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:22.527211 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:22.527136 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:28.716775 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:28.716736 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:28.790017 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:28.789986 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:29.058335 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.058242 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:29.363874 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.363799 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j"] Jul 10 19:34:29.364165 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.364153 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6eb7104-6679-4ecf-8f08-b61a3ec8233a" containerName="istio-proxy" Jul 10 19:34:29.364214 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.364166 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6eb7104-6679-4ecf-8f08-b61a3ec8233a" containerName="istio-proxy" Jul 10 19:34:29.364214 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.364175 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6eb7104-6679-4ecf-8f08-b61a3ec8233a" containerName="istio-proxy" Jul 10 19:34:29.364214 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.364180 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6eb7104-6679-4ecf-8f08-b61a3ec8233a" containerName="istio-proxy" Jul 10 19:34:29.364214 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.364196 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2" containerName="maas-api" Jul 10 19:34:29.364214 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.364203 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2" containerName="maas-api" Jul 10 19:34:29.364380 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.364268 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cd6b2ca-958d-4f59-85f8-8bab0b6cbab2" containerName="maas-api" Jul 10 19:34:29.364380 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.364275 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6eb7104-6679-4ecf-8f08-b61a3ec8233a" containerName="istio-proxy" Jul 10 19:34:29.367274 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.367257 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:29.370054 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.370034 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-fd22f1-openshift-default-dockercfg-kcrjk\"" Jul 10 19:34:29.382392 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.382360 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j"] Jul 10 19:34:29.395548 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.395518 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:29.473680 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.473642 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scvb8\" (UniqueName: \"kubernetes.io/projected/fa8962f2-d40b-4c87-b616-f954c2af8e6b-kube-api-access-scvb8\") pod \"e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:29.473864 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.473689 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa8962f2-d40b-4c87-b616-f954c2af8e6b-workload-certs\") pod \"e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:29.473864 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.473774 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8962f2-d40b-4c87-b616-f954c2af8e6b-credential-socket\") pod \"e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:29.473864 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.473832 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa8962f2-d40b-4c87-b616-f954c2af8e6b-istio-token\") pod \"e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:29.473984 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.473919 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa8962f2-d40b-4c87-b616-f954c2af8e6b-istio-podinfo\") pod \"e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:29.473984 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.473948 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa8962f2-d40b-4c87-b616-f954c2af8e6b-istiod-ca-cert\") pod \"e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:29.473984 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.473973 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa8962f2-d40b-4c87-b616-f954c2af8e6b-istio-envoy\") pod \"e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:29.474092 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.474004 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8962f2-d40b-4c87-b616-f954c2af8e6b-workload-socket\") pod \"e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:29.474092 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.474038 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa8962f2-d40b-4c87-b616-f954c2af8e6b-istio-data\") pod \"e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:29.575091 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.575059 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa8962f2-d40b-4c87-b616-f954c2af8e6b-istio-podinfo\") pod \"e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:29.575091 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.575095 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa8962f2-d40b-4c87-b616-f954c2af8e6b-istiod-ca-cert\") pod \"e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:29.575321 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.575117 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa8962f2-d40b-4c87-b616-f954c2af8e6b-istio-envoy\") pod \"e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:29.575321 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.575135 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8962f2-d40b-4c87-b616-f954c2af8e6b-workload-socket\") pod \"e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:29.575321 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.575166 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa8962f2-d40b-4c87-b616-f954c2af8e6b-istio-data\") pod \"e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:29.575321 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.575184 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scvb8\" (UniqueName: \"kubernetes.io/projected/fa8962f2-d40b-4c87-b616-f954c2af8e6b-kube-api-access-scvb8\") pod \"e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:29.575321 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.575206 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa8962f2-d40b-4c87-b616-f954c2af8e6b-workload-certs\") pod \"e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:29.575321 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.575242 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8962f2-d40b-4c87-b616-f954c2af8e6b-credential-socket\") pod \"e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:29.575321 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.575287 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa8962f2-d40b-4c87-b616-f954c2af8e6b-istio-token\") pod \"e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:29.575766 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.575567 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8962f2-d40b-4c87-b616-f954c2af8e6b-workload-socket\") pod \"e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:29.575766 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.575658 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa8962f2-d40b-4c87-b616-f954c2af8e6b-workload-certs\") pod \"e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:29.575766 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.575755 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8962f2-d40b-4c87-b616-f954c2af8e6b-credential-socket\") pod \"e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:29.575904 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.575818 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa8962f2-d40b-4c87-b616-f954c2af8e6b-istiod-ca-cert\") pod \"e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:29.575904 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.575834 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa8962f2-d40b-4c87-b616-f954c2af8e6b-istio-data\") pod \"e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:29.577551 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.577522 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa8962f2-d40b-4c87-b616-f954c2af8e6b-istio-envoy\") pod \"e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:29.577754 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.577736 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa8962f2-d40b-4c87-b616-f954c2af8e6b-istio-podinfo\") pod \"e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:29.583734 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.583708 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa8962f2-d40b-4c87-b616-f954c2af8e6b-istio-token\") pod \"e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:29.583861 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.583840 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scvb8\" (UniqueName: \"kubernetes.io/projected/fa8962f2-d40b-4c87-b616-f954c2af8e6b-kube-api-access-scvb8\") pod \"e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:29.670894 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.670868 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:29.678916 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.678900 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:29.836820 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.836799 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j"] Jul 10 19:34:29.839110 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:34:29.839083 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa8962f2_d40b_4c87_b616_f954c2af8e6b.slice/crio-d0f702dc616edc8fd8be211a4322a0a654bcdbe93a066df48c6e884b9b161ec1 WatchSource:0}: Error finding container d0f702dc616edc8fd8be211a4322a0a654bcdbe93a066df48c6e884b9b161ec1: Status 404 returned error can't find the container with id d0f702dc616edc8fd8be211a4322a0a654bcdbe93a066df48c6e884b9b161ec1 Jul 10 19:34:29.841392 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.841358 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:34:29.841497 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.841422 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:34:29.841497 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:29.841477 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:34:30.722641 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:30.722609 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" event={"ID":"fa8962f2-d40b-4c87-b616-f954c2af8e6b","Type":"ContainerStarted","Data":"67ac60a1399924a94305dca236bf8c352ef9bd3dc6f4e0d4d4bfcb22096be78b"} Jul 10 19:34:30.722833 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:30.722665 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" event={"ID":"fa8962f2-d40b-4c87-b616-f954c2af8e6b","Type":"ContainerStarted","Data":"d0f702dc616edc8fd8be211a4322a0a654bcdbe93a066df48c6e884b9b161ec1"} Jul 10 19:34:30.745563 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:30.745515 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" podStartSLOduration=1.7455016159999999 podStartE2EDuration="1.745501616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:34:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:34:30.743823302 +0000 UTC m=+2162.073964137" watchObservedRunningTime="2026-07-10 19:34:30.745501616 +0000 UTC m=+2162.075642436" Jul 10 19:34:31.680082 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:31.680050 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:31.684404 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:31.684382 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:31.727834 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:31.727805 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:31.728552 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:31.728536 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:32.622344 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:32.622306 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:32.701982 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:32.701946 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:32.709676 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:32.709655 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb"] Jul 10 19:34:32.710126 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:32.710114 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6eb7104-6679-4ecf-8f08-b61a3ec8233a" containerName="istio-proxy" Jul 10 19:34:32.713274 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:32.713260 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb" Jul 10 19:34:32.716161 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:32.716138 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-fd22f1-serving-cert\"" Jul 10 19:34:32.716269 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:32.716209 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 19:34:32.717395 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:32.717377 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 19:34:32.717487 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:32.717416 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-44lh8\"" Jul 10 19:34:32.724299 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:32.724279 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb"] Jul 10 19:34:32.764406 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:32.764380 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:32.806657 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:32.806633 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f876\" (UniqueName: \"kubernetes.io/projected/a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2-kube-api-access-5f876\") pod \"maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb\" (UID: \"a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb" Jul 10 19:34:32.806780 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:32.806683 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2-maas-api-tls\") pod \"maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb\" (UID: \"a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb" Jul 10 19:34:32.806780 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:32.806725 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb\" (UID: \"a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb" Jul 10 19:34:32.893210 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:32.893138 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:32.908363 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:32.908331 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f876\" (UniqueName: \"kubernetes.io/projected/a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2-kube-api-access-5f876\") pod \"maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb\" (UID: \"a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb" Jul 10 19:34:32.908521 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:32.908419 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2-maas-api-tls\") pod \"maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb\" (UID: \"a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb" Jul 10 19:34:32.908521 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:32.908490 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb\" (UID: \"a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb" Jul 10 19:34:32.909194 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:32.909168 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb\" (UID: \"a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb" Jul 10 19:34:32.911217 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:32.911188 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2-maas-api-tls\") pod \"maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb\" (UID: \"a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb" Jul 10 19:34:32.916696 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:32.916672 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f876\" (UniqueName: \"kubernetes.io/projected/a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2-kube-api-access-5f876\") pod \"maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb\" (UID: \"a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb" Jul 10 19:34:33.025511 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:33.025483 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb" Jul 10 19:34:33.164133 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:33.164105 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb"] Jul 10 19:34:33.165909 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:34:33.165878 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0bb5fdc_7f44_4413_9bb0_aaeaa3f8a8b2.slice/crio-2f8e9fae5431e8c79283d3981ebfad410f0c0f7123a1af53a95ad49c4dbb6f3c WatchSource:0}: Error finding container 2f8e9fae5431e8c79283d3981ebfad410f0c0f7123a1af53a95ad49c4dbb6f3c: Status 404 returned error can't find the container with id 2f8e9fae5431e8c79283d3981ebfad410f0c0f7123a1af53a95ad49c4dbb6f3c Jul 10 19:34:33.294599 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:33.294567 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:33.394905 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:33.394879 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:33.739623 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:33.739584 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb" event={"ID":"a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2","Type":"ContainerStarted","Data":"f697ba19f1c7697b5b92e9adf64b6f7c9f877153f4f3f1757efea44cda5d8ee1"} Jul 10 19:34:33.739940 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:33.739665 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb" Jul 10 19:34:33.739940 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:33.739694 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb" event={"ID":"a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2","Type":"ContainerStarted","Data":"2f8e9fae5431e8c79283d3981ebfad410f0c0f7123a1af53a95ad49c4dbb6f3c"} Jul 10 19:34:33.758459 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:33.758408 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb" podStartSLOduration=1.259966468 podStartE2EDuration="1.758394273s" totalImagesPullingTime="498.427805ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:34:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:34:33.756354418 +0000 UTC m=+2165.086495228" watchObservedRunningTime="2026-07-10 19:34:33.758394273 +0000 UTC m=+2165.088535111" Jul 10 19:34:37.115462 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:37.115407 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:37.628321 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:37.628244 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:37.778946 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:37.778913 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx"] Jul 10 19:34:37.782333 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:37.782315 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:34:37.785168 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:37.785146 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8c42c2-openshift-default-dockercfg-nkbh4\"" Jul 10 19:34:37.794653 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:37.794636 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx"] Jul 10 19:34:37.808432 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:37.808403 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:37.954786 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:37.954753 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65sq7\" (UniqueName: \"kubernetes.io/projected/5c428018-2c69-4300-affa-5ee255940f86-kube-api-access-65sq7\") pod \"e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:34:37.954971 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:37.954807 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c428018-2c69-4300-affa-5ee255940f86-credential-socket\") pod \"e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:34:37.954971 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:37.954846 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c428018-2c69-4300-affa-5ee255940f86-istio-data\") pod \"e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:34:37.954971 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:37.954879 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c428018-2c69-4300-affa-5ee255940f86-workload-certs\") pod \"e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:34:37.954971 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:37.954900 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c428018-2c69-4300-affa-5ee255940f86-istio-podinfo\") pod \"e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:34:37.954971 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:37.954953 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c428018-2c69-4300-affa-5ee255940f86-workload-socket\") pod \"e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:34:37.955177 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:37.954989 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c428018-2c69-4300-affa-5ee255940f86-istiod-ca-cert\") pod \"e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:34:37.955177 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:37.955017 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c428018-2c69-4300-affa-5ee255940f86-istio-token\") pod \"e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:34:37.955177 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:37.955046 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c428018-2c69-4300-affa-5ee255940f86-istio-envoy\") pod \"e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:34:38.008722 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:38.008690 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:38.055648 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:38.055623 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c428018-2c69-4300-affa-5ee255940f86-istio-data\") pod \"e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:34:38.055779 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:38.055661 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c428018-2c69-4300-affa-5ee255940f86-workload-certs\") pod \"e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:34:38.055779 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:38.055677 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c428018-2c69-4300-affa-5ee255940f86-istio-podinfo\") pod \"e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:34:38.055779 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:38.055710 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c428018-2c69-4300-affa-5ee255940f86-workload-socket\") pod \"e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:34:38.055926 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:38.055829 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c428018-2c69-4300-affa-5ee255940f86-istiod-ca-cert\") pod \"e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:34:38.055926 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:38.055873 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c428018-2c69-4300-affa-5ee255940f86-istio-token\") pod \"e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:34:38.056035 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:38.055909 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c428018-2c69-4300-affa-5ee255940f86-istio-envoy\") pod \"e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:34:38.056092 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:38.056036 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c428018-2c69-4300-affa-5ee255940f86-workload-certs\") pod \"e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:34:38.056150 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:38.056098 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c428018-2c69-4300-affa-5ee255940f86-workload-socket\") pod \"e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:34:38.056150 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:38.056140 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c428018-2c69-4300-affa-5ee255940f86-istio-data\") pod \"e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:34:38.056369 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:38.056344 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65sq7\" (UniqueName: \"kubernetes.io/projected/5c428018-2c69-4300-affa-5ee255940f86-kube-api-access-65sq7\") pod \"e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:34:38.056472 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:38.056399 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c428018-2c69-4300-affa-5ee255940f86-credential-socket\") pod \"e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:34:38.056577 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:38.056557 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c428018-2c69-4300-affa-5ee255940f86-istiod-ca-cert\") pod \"e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:34:38.056739 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:38.056718 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c428018-2c69-4300-affa-5ee255940f86-credential-socket\") pod \"e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:34:38.058092 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:38.058072 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c428018-2c69-4300-affa-5ee255940f86-istio-envoy\") pod \"e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:34:38.058254 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:38.058237 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c428018-2c69-4300-affa-5ee255940f86-istio-podinfo\") pod \"e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:34:38.064105 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:38.064086 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c428018-2c69-4300-affa-5ee255940f86-istio-token\") pod \"e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:34:38.064409 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:38.064391 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65sq7\" (UniqueName: \"kubernetes.io/projected/5c428018-2c69-4300-affa-5ee255940f86-kube-api-access-65sq7\") pod \"e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:34:38.094086 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:38.094057 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:34:38.229074 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:38.229047 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx"] Jul 10 19:34:38.230430 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:34:38.230399 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c428018_2c69_4300_affa_5ee255940f86.slice/crio-fe3191e32ac9bc8749b9db66630460753de11a7304bee4ca59289a9248a70fb0 WatchSource:0}: Error finding container fe3191e32ac9bc8749b9db66630460753de11a7304bee4ca59289a9248a70fb0: Status 404 returned error can't find the container with id fe3191e32ac9bc8749b9db66630460753de11a7304bee4ca59289a9248a70fb0 Jul 10 19:34:38.232766 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:38.232725 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:34:38.232847 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:38.232817 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:34:38.232891 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:38.232871 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:34:38.765898 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:38.765868 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" event={"ID":"5c428018-2c69-4300-affa-5ee255940f86","Type":"ContainerStarted","Data":"fb6610215f9498ad3f4e8e46b41ef0e49d06f545f7708f6b8e9281d6fdceeb74"} Jul 10 19:34:38.766092 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:38.765915 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" event={"ID":"5c428018-2c69-4300-affa-5ee255940f86","Type":"ContainerStarted","Data":"fe3191e32ac9bc8749b9db66630460753de11a7304bee4ca59289a9248a70fb0"} Jul 10 19:34:38.789612 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:38.789566 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" podStartSLOduration=1.789551538 podStartE2EDuration="1.789551538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:34:38.787236652 +0000 UTC m=+2170.117377478" watchObservedRunningTime="2026-07-10 19:34:38.789551538 +0000 UTC m=+2170.119692346" Jul 10 19:34:39.095101 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:39.095025 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:34:39.099632 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:39.099599 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:34:39.749730 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:39.749701 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb" Jul 10 19:34:39.771889 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:39.771850 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:34:39.773026 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:39.773002 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:34:40.850258 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:40.850224 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:40.882647 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:40.882622 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz"] Jul 10 19:34:40.894847 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:40.894822 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz" Jul 10 19:34:40.898527 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:40.898258 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-8c42c2-serving-cert\"" Jul 10 19:34:40.898527 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:40.898298 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz"] Jul 10 19:34:40.952359 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:40.952332 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:40.981835 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:40.981807 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dx46\" (UniqueName: \"kubernetes.io/projected/6179d2be-3381-480b-a07f-3b5fdf2ff66a-kube-api-access-9dx46\") pod \"maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz\" (UID: \"6179d2be-3381-480b-a07f-3b5fdf2ff66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz" Jul 10 19:34:40.981920 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:40.981889 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6179d2be-3381-480b-a07f-3b5fdf2ff66a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz\" (UID: \"6179d2be-3381-480b-a07f-3b5fdf2ff66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz" Jul 10 19:34:40.981963 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:40.981953 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6179d2be-3381-480b-a07f-3b5fdf2ff66a-maas-api-tls\") pod \"maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz\" (UID: \"6179d2be-3381-480b-a07f-3b5fdf2ff66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz" Jul 10 19:34:41.085462 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:41.085393 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6179d2be-3381-480b-a07f-3b5fdf2ff66a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz\" (UID: \"6179d2be-3381-480b-a07f-3b5fdf2ff66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz" Jul 10 19:34:41.085648 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:41.085567 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6179d2be-3381-480b-a07f-3b5fdf2ff66a-maas-api-tls\") pod \"maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz\" (UID: \"6179d2be-3381-480b-a07f-3b5fdf2ff66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz" Jul 10 19:34:41.085648 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:41.085632 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dx46\" (UniqueName: \"kubernetes.io/projected/6179d2be-3381-480b-a07f-3b5fdf2ff66a-kube-api-access-9dx46\") pod \"maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz\" (UID: \"6179d2be-3381-480b-a07f-3b5fdf2ff66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz" Jul 10 19:34:41.086181 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:34:41.086154 2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-8c42c2-serving-cert: secret "maas-api-e2e-inf-b-8c42c2-serving-cert" not found Jul 10 19:34:41.086305 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:34:41.086238 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6179d2be-3381-480b-a07f-3b5fdf2ff66a-maas-api-tls podName:6179d2be-3381-480b-a07f-3b5fdf2ff66a nodeName:}" failed. No retries permitted until 2026-07-10 19:34:41.586216959 +0000 UTC m=+2172.916357766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6179d2be-3381-480b-a07f-3b5fdf2ff66a-maas-api-tls") pod "maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz" (UID: "6179d2be-3381-480b-a07f-3b5fdf2ff66a") : secret "maas-api-e2e-inf-b-8c42c2-serving-cert" not found Jul 10 19:34:41.086897 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:41.086873 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6179d2be-3381-480b-a07f-3b5fdf2ff66a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz\" (UID: \"6179d2be-3381-480b-a07f-3b5fdf2ff66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz" Jul 10 19:34:41.095781 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:41.095745 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dx46\" (UniqueName: \"kubernetes.io/projected/6179d2be-3381-480b-a07f-3b5fdf2ff66a-kube-api-access-9dx46\") pod \"maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz\" (UID: \"6179d2be-3381-480b-a07f-3b5fdf2ff66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz" Jul 10 19:34:41.356538 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:41.356493 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:41.479529 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:41.479500 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:41.584496 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:41.584438 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:41.591613 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:41.591585 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6179d2be-3381-480b-a07f-3b5fdf2ff66a-maas-api-tls\") pod \"maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz\" (UID: \"6179d2be-3381-480b-a07f-3b5fdf2ff66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz" Jul 10 19:34:41.593880 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:41.593859 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6179d2be-3381-480b-a07f-3b5fdf2ff66a-maas-api-tls\") pod \"maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz\" (UID: \"6179d2be-3381-480b-a07f-3b5fdf2ff66a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz" Jul 10 19:34:41.808120 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:41.808091 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz" Jul 10 19:34:41.988637 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:41.988615 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz"] Jul 10 19:34:41.989691 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:34:41.989669 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6179d2be_3381_480b_a07f_3b5fdf2ff66a.slice/crio-554f1beb66f45a32ed7f2419677e3482041f9ed35812307b596dd92b6303e3a7 WatchSource:0}: Error finding container 554f1beb66f45a32ed7f2419677e3482041f9ed35812307b596dd92b6303e3a7: Status 404 returned error can't find the container with id 554f1beb66f45a32ed7f2419677e3482041f9ed35812307b596dd92b6303e3a7 Jul 10 19:34:42.791600 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:42.791568 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz" event={"ID":"6179d2be-3381-480b-a07f-3b5fdf2ff66a","Type":"ContainerStarted","Data":"4fe9e1fabdf85eb2a9b27e802241dfc6baa15e77639a2bb2c8b9df0c38c135b9"} Jul 10 19:34:42.791789 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:42.791629 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz" event={"ID":"6179d2be-3381-480b-a07f-3b5fdf2ff66a","Type":"ContainerStarted","Data":"554f1beb66f45a32ed7f2419677e3482041f9ed35812307b596dd92b6303e3a7"} Jul 10 19:34:42.791789 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:42.791683 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz" Jul 10 19:34:42.810982 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:42.810939 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz" podStartSLOduration=2.209160811 podStartE2EDuration="2.810924806s" totalImagesPullingTime="601.763995ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:34:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:34:42.808487418 +0000 UTC m=+2174.138628250" watchObservedRunningTime="2026-07-10 19:34:42.810924806 +0000 UTC m=+2174.141065615" Jul 10 19:34:45.200264 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:45.200226 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:47.769583 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:47.769545 2560 patch_prober.go:28] interesting pod/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 19:34:47.770018 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:47.769610 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" podUID="fa8962f2-d40b-4c87-b616-f954c2af8e6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 19:34:48.802291 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:48.802216 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz" Jul 10 19:34:48.828979 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:48.827840 2560 generic.go:356] "Generic (PLEG): container finished" podID="fa8962f2-d40b-4c87-b616-f954c2af8e6b" containerID="67ac60a1399924a94305dca236bf8c352ef9bd3dc6f4e0d4d4bfcb22096be78b" exitCode=0 Jul 10 19:34:48.828979 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:48.827908 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" event={"ID":"fa8962f2-d40b-4c87-b616-f954c2af8e6b","Type":"ContainerDied","Data":"67ac60a1399924a94305dca236bf8c352ef9bd3dc6f4e0d4d4bfcb22096be78b"} Jul 10 19:34:48.828979 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:48.828361 2560 scope.go:117] "RemoveContainer" containerID="67ac60a1399924a94305dca236bf8c352ef9bd3dc6f4e0d4d4bfcb22096be78b" Jul 10 19:34:48.831048 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:48.830573 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:34:48.831048 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:48.830656 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:34:48.831048 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:48.830705 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:34:49.837824 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:49.837782 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" event={"ID":"fa8962f2-d40b-4c87-b616-f954c2af8e6b","Type":"ContainerStarted","Data":"f33975e2075b5755c9c45f966b954fe7b30420f2aac56f9888cbd82eb3c5bf47"} Jul 10 19:34:50.679541 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:50.679505 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:50.679731 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:50.679550 2560 patch_prober.go:28] interesting pod/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 10 19:34:50.679731 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:50.679594 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" podUID="fa8962f2-d40b-4c87-b616-f954c2af8e6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 10 19:34:51.166419 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:51.166377 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:51.271328 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:51.271294 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:51.679548 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:51.679511 2560 patch_prober.go:28] interesting pod/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 10 19:34:51.679724 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:51.679572 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" podUID="fa8962f2-d40b-4c87-b616-f954c2af8e6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 10 19:34:52.679563 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:52.679523 2560 patch_prober.go:28] interesting pod/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 10 19:34:52.680002 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:52.679584 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" podUID="fa8962f2-d40b-4c87-b616-f954c2af8e6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 10 19:34:53.683467 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:53.683419 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:53.683915 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:53.683716 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:53.684562 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:53.684538 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:34:54.225373 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:54.225337 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:54.323954 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:54.323918 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:55.358867 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:55.358626 2560 patch_prober.go:28] interesting pod/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": read tcp 10.132.0.2:43870->10.132.0.49:15021: read: connection reset by peer" start-of-body= Jul 10 19:34:55.358867 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:55.358704 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" podUID="5c428018-2c69-4300-affa-5ee255940f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": read tcp 10.132.0.2:43870->10.132.0.49:15021: read: connection reset by peer" Jul 10 19:34:55.875549 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:55.875517 2560 generic.go:356] "Generic (PLEG): container finished" podID="5c428018-2c69-4300-affa-5ee255940f86" containerID="fb6610215f9498ad3f4e8e46b41ef0e49d06f545f7708f6b8e9281d6fdceeb74" exitCode=0 Jul 10 19:34:55.875731 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:55.875587 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" event={"ID":"5c428018-2c69-4300-affa-5ee255940f86","Type":"ContainerDied","Data":"fb6610215f9498ad3f4e8e46b41ef0e49d06f545f7708f6b8e9281d6fdceeb74"} Jul 10 19:34:55.875973 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:55.875959 2560 scope.go:117] "RemoveContainer" containerID="fb6610215f9498ad3f4e8e46b41ef0e49d06f545f7708f6b8e9281d6fdceeb74" Jul 10 19:34:55.876895 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:55.876871 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:34:55.876946 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:55.876934 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:34:55.876985 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:55.876965 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:34:56.882339 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:56.882299 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" event={"ID":"5c428018-2c69-4300-affa-5ee255940f86","Type":"ContainerStarted","Data":"646ea2f2fefd43daa18c5378ed93f81d214c592e7176ce8f0829ce65117eaa95"} Jul 10 19:34:57.094692 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:57.094658 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:34:57.094860 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:57.094829 2560 patch_prober.go:28] interesting pod/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 10 19:34:57.094908 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:57.094878 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" podUID="5c428018-2c69-4300-affa-5ee255940f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 10 19:34:58.094590 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:58.094553 2560 patch_prober.go:28] interesting pod/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 10 19:34:58.094958 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:58.094607 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" podUID="5c428018-2c69-4300-affa-5ee255940f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 10 19:34:59.095374 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:59.095339 2560 patch_prober.go:28] interesting pod/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 10 19:34:59.095794 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:34:59.095407 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" podUID="5c428018-2c69-4300-affa-5ee255940f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 10 19:35:00.094857 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:35:00.094823 2560 patch_prober.go:28] interesting pod/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 10 19:35:00.095021 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:35:00.094887 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" podUID="5c428018-2c69-4300-affa-5ee255940f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 10 19:35:01.098905 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:35:01.098873 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:35:01.099281 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:35:01.099126 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:35:01.099783 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:35:01.099767 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:35:03.347299 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:35:03.347266 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:03.450248 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:35:03.450206 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:09.652371 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:35:09.652336 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:13.843582 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:35:13.843547 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:44.109336 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:35:44.109289 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:19.358297 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:19.358258 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:20.395915 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:20.395881 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:20.997461 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:20.997404 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:21.098117 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:21.098073 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:30.933470 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:30.933418 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb"] Jul 10 19:36:30.933909 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:30.933704 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb" podUID="a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2" containerName="maas-api" containerID="cri-o://f697ba19f1c7697b5b92e9adf64b6f7c9f877153f4f3f1757efea44cda5d8ee1" gracePeriod=30 Jul 10 19:36:30.959973 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:30.959943 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:31.190202 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:31.190138 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb" Jul 10 19:36:31.305695 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:31.305666 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5f876\" (UniqueName: \"kubernetes.io/projected/a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2-kube-api-access-5f876\") pod \"a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2\" (UID: \"a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2\") " Jul 10 19:36:31.305896 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:31.305713 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2-maas-api-tls\") pod \"a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2\" (UID: \"a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2\") " Jul 10 19:36:31.305896 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:31.305743 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2-openshift-service-ca\") pod \"a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2\" (UID: \"a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2\") " Jul 10 19:36:31.306185 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:31.306159 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2" (UID: "a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:36:31.307855 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:31.307830 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2" (UID: "a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:36:31.307933 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:31.307878 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2-kube-api-access-5f876" (OuterVolumeSpecName: "kube-api-access-5f876") pod "a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2" (UID: "a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2"). InnerVolumeSpecName "kube-api-access-5f876". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:36:31.374668 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:31.374635 2560 generic.go:356] "Generic (PLEG): container finished" podID="a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2" containerID="f697ba19f1c7697b5b92e9adf64b6f7c9f877153f4f3f1757efea44cda5d8ee1" exitCode=0 Jul 10 19:36:31.374835 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:31.374685 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb" Jul 10 19:36:31.374835 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:31.374726 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb" event={"ID":"a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2","Type":"ContainerDied","Data":"f697ba19f1c7697b5b92e9adf64b6f7c9f877153f4f3f1757efea44cda5d8ee1"} Jul 10 19:36:31.374835 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:31.374785 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb" event={"ID":"a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2","Type":"ContainerDied","Data":"2f8e9fae5431e8c79283d3981ebfad410f0c0f7123a1af53a95ad49c4dbb6f3c"} Jul 10 19:36:31.374835 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:31.374813 2560 scope.go:117] "RemoveContainer" containerID="f697ba19f1c7697b5b92e9adf64b6f7c9f877153f4f3f1757efea44cda5d8ee1" Jul 10 19:36:31.384204 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:31.384187 2560 scope.go:117] "RemoveContainer" containerID="f697ba19f1c7697b5b92e9adf64b6f7c9f877153f4f3f1757efea44cda5d8ee1" Jul 10 19:36:31.384522 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:36:31.384498 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f697ba19f1c7697b5b92e9adf64b6f7c9f877153f4f3f1757efea44cda5d8ee1\": container with ID starting with f697ba19f1c7697b5b92e9adf64b6f7c9f877153f4f3f1757efea44cda5d8ee1 not found: ID does not exist" containerID="f697ba19f1c7697b5b92e9adf64b6f7c9f877153f4f3f1757efea44cda5d8ee1" Jul 10 19:36:31.384630 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:31.384527 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f697ba19f1c7697b5b92e9adf64b6f7c9f877153f4f3f1757efea44cda5d8ee1"} err="failed to get container status \"f697ba19f1c7697b5b92e9adf64b6f7c9f877153f4f3f1757efea44cda5d8ee1\": rpc error: code = NotFound desc = could not find container \"f697ba19f1c7697b5b92e9adf64b6f7c9f877153f4f3f1757efea44cda5d8ee1\": container with ID starting with f697ba19f1c7697b5b92e9adf64b6f7c9f877153f4f3f1757efea44cda5d8ee1 not found: ID does not exist" Jul 10 19:36:31.396894 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:31.396868 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb"] Jul 10 19:36:31.400168 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:31.400146 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fd22f1-7ff8c7b68b-gwshb"] Jul 10 19:36:31.407494 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:31.407472 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f876\" (UniqueName: \"kubernetes.io/projected/a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2-kube-api-access-5f876\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:36:31.407571 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:31.407503 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2-maas-api-tls\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:36:31.407571 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:31.407518 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2-openshift-service-ca\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:36:33.254864 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:33.254832 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2" path="/var/lib/kubelet/pods/a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2/volumes" Jul 10 19:36:35.649149 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:35.649111 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:41.533254 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:41.533216 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:41.639433 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:41.639398 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:46.408000 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:46.407966 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j"] Jul 10 19:36:46.408490 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:46.408186 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" podUID="fa8962f2-d40b-4c87-b616-f954c2af8e6b" containerName="istio-proxy" containerID="cri-o://f33975e2075b5755c9c45f966b954fe7b30420f2aac56f9888cbd82eb3c5bf47" gracePeriod=30 Jul 10 19:36:46.423340 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:46.423312 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:46.721758 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:46.721670 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:47.379738 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:47.379706 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:47.474245 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:47.474199 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:51.663772 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:51.663745 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:36:51.793149 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:51.793077 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa8962f2-d40b-4c87-b616-f954c2af8e6b-istio-token\") pod \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " Jul 10 19:36:51.793149 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:51.793108 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa8962f2-d40b-4c87-b616-f954c2af8e6b-istiod-ca-cert\") pod \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " Jul 10 19:36:51.793329 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:51.793174 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scvb8\" (UniqueName: \"kubernetes.io/projected/fa8962f2-d40b-4c87-b616-f954c2af8e6b-kube-api-access-scvb8\") pod \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " Jul 10 19:36:51.793329 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:51.793199 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa8962f2-d40b-4c87-b616-f954c2af8e6b-workload-certs\") pod \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " Jul 10 19:36:51.793329 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:51.793225 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8962f2-d40b-4c87-b616-f954c2af8e6b-workload-socket\") pod \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " Jul 10 19:36:51.793329 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:51.793242 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa8962f2-d40b-4c87-b616-f954c2af8e6b-istio-data\") pod \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " Jul 10 19:36:51.793574 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:51.793367 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa8962f2-d40b-4c87-b616-f954c2af8e6b-istio-podinfo\") pod \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " Jul 10 19:36:51.793574 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:51.793482 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8962f2-d40b-4c87-b616-f954c2af8e6b-credential-socket\") pod \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " Jul 10 19:36:51.793574 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:51.793530 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa8962f2-d40b-4c87-b616-f954c2af8e6b-istio-envoy\") pod \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\" (UID: \"fa8962f2-d40b-4c87-b616-f954c2af8e6b\") " Jul 10 19:36:51.793737 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:51.793592 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8962f2-d40b-4c87-b616-f954c2af8e6b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fa8962f2-d40b-4c87-b616-f954c2af8e6b" (UID: "fa8962f2-d40b-4c87-b616-f954c2af8e6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:36:51.793737 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:51.793651 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa8962f2-d40b-4c87-b616-f954c2af8e6b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fa8962f2-d40b-4c87-b616-f954c2af8e6b" (UID: "fa8962f2-d40b-4c87-b616-f954c2af8e6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:36:51.793843 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:51.793798 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8962f2-d40b-4c87-b616-f954c2af8e6b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fa8962f2-d40b-4c87-b616-f954c2af8e6b" (UID: "fa8962f2-d40b-4c87-b616-f954c2af8e6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:36:51.793843 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:51.793819 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8962f2-d40b-4c87-b616-f954c2af8e6b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fa8962f2-d40b-4c87-b616-f954c2af8e6b" (UID: "fa8962f2-d40b-4c87-b616-f954c2af8e6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:36:51.794001 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:51.793984 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8962f2-d40b-4c87-b616-f954c2af8e6b-workload-socket\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:36:51.794064 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:51.794007 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8962f2-d40b-4c87-b616-f954c2af8e6b-credential-socket\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:36:51.794064 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:51.794036 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa8962f2-d40b-4c87-b616-f954c2af8e6b-istiod-ca-cert\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:36:51.794064 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:51.794050 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa8962f2-d40b-4c87-b616-f954c2af8e6b-workload-certs\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:36:51.794399 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:51.794378 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8962f2-d40b-4c87-b616-f954c2af8e6b-istio-data" (OuterVolumeSpecName: "istio-data") pod "fa8962f2-d40b-4c87-b616-f954c2af8e6b" (UID: "fa8962f2-d40b-4c87-b616-f954c2af8e6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:36:51.795484 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:51.795433 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa8962f2-d40b-4c87-b616-f954c2af8e6b-istio-token" (OuterVolumeSpecName: "istio-token") pod "fa8962f2-d40b-4c87-b616-f954c2af8e6b" (UID: "fa8962f2-d40b-4c87-b616-f954c2af8e6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:36:51.795674 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:51.795651 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa8962f2-d40b-4c87-b616-f954c2af8e6b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fa8962f2-d40b-4c87-b616-f954c2af8e6b" (UID: "fa8962f2-d40b-4c87-b616-f954c2af8e6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 19:36:51.795842 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:51.795822 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa8962f2-d40b-4c87-b616-f954c2af8e6b-kube-api-access-scvb8" (OuterVolumeSpecName: "kube-api-access-scvb8") pod "fa8962f2-d40b-4c87-b616-f954c2af8e6b" (UID: "fa8962f2-d40b-4c87-b616-f954c2af8e6b"). InnerVolumeSpecName "kube-api-access-scvb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:36:51.795934 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:51.795919 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8962f2-d40b-4c87-b616-f954c2af8e6b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fa8962f2-d40b-4c87-b616-f954c2af8e6b" (UID: "fa8962f2-d40b-4c87-b616-f954c2af8e6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:36:51.894675 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:51.894644 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa8962f2-d40b-4c87-b616-f954c2af8e6b-istio-podinfo\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:36:51.894675 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:51.894675 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa8962f2-d40b-4c87-b616-f954c2af8e6b-istio-envoy\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:36:51.894886 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:51.894688 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa8962f2-d40b-4c87-b616-f954c2af8e6b-istio-token\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:36:51.894886 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:51.894700 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scvb8\" (UniqueName: \"kubernetes.io/projected/fa8962f2-d40b-4c87-b616-f954c2af8e6b-kube-api-access-scvb8\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:36:51.894886 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:51.894713 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa8962f2-d40b-4c87-b616-f954c2af8e6b-istio-data\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:36:52.486249 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:52.486208 2560 generic.go:356] "Generic (PLEG): container finished" podID="fa8962f2-d40b-4c87-b616-f954c2af8e6b" containerID="f33975e2075b5755c9c45f966b954fe7b30420f2aac56f9888cbd82eb3c5bf47" exitCode=0 Jul 10 19:36:52.486470 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:52.486261 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" Jul 10 19:36:52.486470 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:52.486303 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" event={"ID":"fa8962f2-d40b-4c87-b616-f954c2af8e6b","Type":"ContainerDied","Data":"f33975e2075b5755c9c45f966b954fe7b30420f2aac56f9888cbd82eb3c5bf47"} Jul 10 19:36:52.486470 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:52.486369 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j" event={"ID":"fa8962f2-d40b-4c87-b616-f954c2af8e6b","Type":"ContainerDied","Data":"d0f702dc616edc8fd8be211a4322a0a654bcdbe93a066df48c6e884b9b161ec1"} Jul 10 19:36:52.486470 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:52.486401 2560 scope.go:117] "RemoveContainer" containerID="f33975e2075b5755c9c45f966b954fe7b30420f2aac56f9888cbd82eb3c5bf47" Jul 10 19:36:52.495810 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:52.495794 2560 scope.go:117] "RemoveContainer" containerID="67ac60a1399924a94305dca236bf8c352ef9bd3dc6f4e0d4d4bfcb22096be78b" Jul 10 19:36:52.504265 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:52.504251 2560 scope.go:117] "RemoveContainer" containerID="f33975e2075b5755c9c45f966b954fe7b30420f2aac56f9888cbd82eb3c5bf47" Jul 10 19:36:52.504515 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:36:52.504496 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f33975e2075b5755c9c45f966b954fe7b30420f2aac56f9888cbd82eb3c5bf47\": container with ID starting with f33975e2075b5755c9c45f966b954fe7b30420f2aac56f9888cbd82eb3c5bf47 not found: ID does not exist" containerID="f33975e2075b5755c9c45f966b954fe7b30420f2aac56f9888cbd82eb3c5bf47" Jul 10 19:36:52.504574 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:52.504522 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f33975e2075b5755c9c45f966b954fe7b30420f2aac56f9888cbd82eb3c5bf47"} err="failed to get container status \"f33975e2075b5755c9c45f966b954fe7b30420f2aac56f9888cbd82eb3c5bf47\": rpc error: code = NotFound desc = could not find container \"f33975e2075b5755c9c45f966b954fe7b30420f2aac56f9888cbd82eb3c5bf47\": container with ID starting with f33975e2075b5755c9c45f966b954fe7b30420f2aac56f9888cbd82eb3c5bf47 not found: ID does not exist" Jul 10 19:36:52.504574 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:52.504538 2560 scope.go:117] "RemoveContainer" containerID="67ac60a1399924a94305dca236bf8c352ef9bd3dc6f4e0d4d4bfcb22096be78b" Jul 10 19:36:52.504776 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:36:52.504758 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67ac60a1399924a94305dca236bf8c352ef9bd3dc6f4e0d4d4bfcb22096be78b\": container with ID starting with 67ac60a1399924a94305dca236bf8c352ef9bd3dc6f4e0d4d4bfcb22096be78b not found: ID does not exist" containerID="67ac60a1399924a94305dca236bf8c352ef9bd3dc6f4e0d4d4bfcb22096be78b" Jul 10 19:36:52.504831 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:52.504785 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67ac60a1399924a94305dca236bf8c352ef9bd3dc6f4e0d4d4bfcb22096be78b"} err="failed to get container status \"67ac60a1399924a94305dca236bf8c352ef9bd3dc6f4e0d4d4bfcb22096be78b\": rpc error: code = NotFound desc = could not find container \"67ac60a1399924a94305dca236bf8c352ef9bd3dc6f4e0d4d4bfcb22096be78b\": container with ID starting with 67ac60a1399924a94305dca236bf8c352ef9bd3dc6f4e0d4d4bfcb22096be78b not found: ID does not exist" Jul 10 19:36:52.508522 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:52.508501 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j"] Jul 10 19:36:52.512589 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:52.512569 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-fd22f1-openshift-default-64d65c4df9-kj85j"] Jul 10 19:36:53.252854 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:53.252821 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa8962f2-d40b-4c87-b616-f954c2af8e6b" path="/var/lib/kubelet/pods/fa8962f2-d40b-4c87-b616-f954c2af8e6b/volumes" Jul 10 19:36:57.266931 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:57.266896 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz"] Jul 10 19:36:57.267407 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:57.267078 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz" podUID="6179d2be-3381-480b-a07f-3b5fdf2ff66a" containerName="maas-api" containerID="cri-o://4fe9e1fabdf85eb2a9b27e802241dfc6baa15e77639a2bb2c8b9df0c38c135b9" gracePeriod=30 Jul 10 19:36:57.287004 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:57.286971 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:57.513882 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:57.513849 2560 generic.go:356] "Generic (PLEG): container finished" podID="6179d2be-3381-480b-a07f-3b5fdf2ff66a" containerID="4fe9e1fabdf85eb2a9b27e802241dfc6baa15e77639a2bb2c8b9df0c38c135b9" exitCode=0 Jul 10 19:36:57.514025 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:57.513924 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz" event={"ID":"6179d2be-3381-480b-a07f-3b5fdf2ff66a","Type":"ContainerDied","Data":"4fe9e1fabdf85eb2a9b27e802241dfc6baa15e77639a2bb2c8b9df0c38c135b9"} Jul 10 19:36:57.514025 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:57.513990 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz" event={"ID":"6179d2be-3381-480b-a07f-3b5fdf2ff66a","Type":"ContainerDied","Data":"554f1beb66f45a32ed7f2419677e3482041f9ed35812307b596dd92b6303e3a7"} Jul 10 19:36:57.514025 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:57.514012 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="554f1beb66f45a32ed7f2419677e3482041f9ed35812307b596dd92b6303e3a7" Jul 10 19:36:57.521753 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:57.521698 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz" Jul 10 19:36:57.647278 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:57.647246 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dx46\" (UniqueName: \"kubernetes.io/projected/6179d2be-3381-480b-a07f-3b5fdf2ff66a-kube-api-access-9dx46\") pod \"6179d2be-3381-480b-a07f-3b5fdf2ff66a\" (UID: \"6179d2be-3381-480b-a07f-3b5fdf2ff66a\") " Jul 10 19:36:57.647504 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:57.647333 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6179d2be-3381-480b-a07f-3b5fdf2ff66a-maas-api-tls\") pod \"6179d2be-3381-480b-a07f-3b5fdf2ff66a\" (UID: \"6179d2be-3381-480b-a07f-3b5fdf2ff66a\") " Jul 10 19:36:57.647504 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:57.647378 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6179d2be-3381-480b-a07f-3b5fdf2ff66a-openshift-service-ca\") pod \"6179d2be-3381-480b-a07f-3b5fdf2ff66a\" (UID: \"6179d2be-3381-480b-a07f-3b5fdf2ff66a\") " Jul 10 19:36:57.647798 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:57.647767 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6179d2be-3381-480b-a07f-3b5fdf2ff66a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6179d2be-3381-480b-a07f-3b5fdf2ff66a" (UID: "6179d2be-3381-480b-a07f-3b5fdf2ff66a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:36:57.649221 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:57.649203 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6179d2be-3381-480b-a07f-3b5fdf2ff66a-kube-api-access-9dx46" (OuterVolumeSpecName: "kube-api-access-9dx46") pod "6179d2be-3381-480b-a07f-3b5fdf2ff66a" (UID: "6179d2be-3381-480b-a07f-3b5fdf2ff66a"). InnerVolumeSpecName "kube-api-access-9dx46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:36:57.649423 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:57.649403 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6179d2be-3381-480b-a07f-3b5fdf2ff66a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6179d2be-3381-480b-a07f-3b5fdf2ff66a" (UID: "6179d2be-3381-480b-a07f-3b5fdf2ff66a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:36:57.748824 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:57.748795 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dx46\" (UniqueName: \"kubernetes.io/projected/6179d2be-3381-480b-a07f-3b5fdf2ff66a-kube-api-access-9dx46\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:36:57.748824 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:57.748823 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6179d2be-3381-480b-a07f-3b5fdf2ff66a-maas-api-tls\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:36:57.749016 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:57.748835 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6179d2be-3381-480b-a07f-3b5fdf2ff66a-openshift-service-ca\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:36:58.519583 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:58.519555 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz" Jul 10 19:36:58.542194 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:58.542164 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz"] Jul 10 19:36:58.545691 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:58.545664 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8c42c2-5656d85b9b-58wmz"] Jul 10 19:36:59.252307 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:36:59.252278 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6179d2be-3381-480b-a07f-3b5fdf2ff66a" path="/var/lib/kubelet/pods/6179d2be-3381-480b-a07f-3b5fdf2ff66a/volumes" Jul 10 19:37:01.940573 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:01.940538 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:08.010804 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:08.010764 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:08.073254 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:08.073224 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:12.634037 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:12.634003 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx"] Jul 10 19:37:12.634504 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:12.634296 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" podUID="5c428018-2c69-4300-affa-5ee255940f86" containerName="istio-proxy" containerID="cri-o://646ea2f2fefd43daa18c5378ed93f81d214c592e7176ce8f0829ce65117eaa95" gracePeriod=30 Jul 10 19:37:12.650231 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:12.650203 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:12.934301 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:12.934264 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:13.355817 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.355732 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:13.669881 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.669848 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9"] Jul 10 19:37:13.670419 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.670404 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa8962f2-d40b-4c87-b616-f954c2af8e6b" containerName="istio-proxy" Jul 10 19:37:13.670494 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.670422 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa8962f2-d40b-4c87-b616-f954c2af8e6b" containerName="istio-proxy" Jul 10 19:37:13.670494 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.670469 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa8962f2-d40b-4c87-b616-f954c2af8e6b" containerName="istio-proxy" Jul 10 19:37:13.670494 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.670477 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa8962f2-d40b-4c87-b616-f954c2af8e6b" containerName="istio-proxy" Jul 10 19:37:13.670494 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.670488 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6179d2be-3381-480b-a07f-3b5fdf2ff66a" containerName="maas-api" Jul 10 19:37:13.670634 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.670498 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6179d2be-3381-480b-a07f-3b5fdf2ff66a" containerName="maas-api" Jul 10 19:37:13.670634 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.670528 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2" containerName="maas-api" Jul 10 19:37:13.670634 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.670537 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2" containerName="maas-api" Jul 10 19:37:13.670634 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.670626 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa8962f2-d40b-4c87-b616-f954c2af8e6b" containerName="istio-proxy" Jul 10 19:37:13.670767 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.670640 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0bb5fdc-7f44-4413-9bb0-aaeaa3f8a8b2" containerName="maas-api" Jul 10 19:37:13.670767 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.670652 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6179d2be-3381-480b-a07f-3b5fdf2ff66a" containerName="maas-api" Jul 10 19:37:13.673593 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.673575 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:13.676346 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.676329 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-9d1921-openshift-default-dockercfg-twq4p\"" Jul 10 19:37:13.684859 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.684829 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9"] Jul 10 19:37:13.686654 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.686631 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs86r\" (UniqueName: \"kubernetes.io/projected/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-kube-api-access-hs86r\") pod \"e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:13.686903 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.686883 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-istio-envoy\") pod \"e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:13.687029 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.686990 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-istiod-ca-cert\") pod \"e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:13.687149 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.687078 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-workload-socket\") pod \"e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:13.687149 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.687133 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-istio-token\") pod \"e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:13.687247 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.687154 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-workload-certs\") pod \"e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:13.687247 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.687192 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-istio-podinfo\") pod \"e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:13.687247 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.687222 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-credential-socket\") pod \"e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:13.687247 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.687236 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-istio-data\") pod \"e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:13.694545 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.694513 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:13.788215 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.788176 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs86r\" (UniqueName: \"kubernetes.io/projected/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-kube-api-access-hs86r\") pod \"e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:13.788430 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.788251 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-istio-envoy\") pod \"e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:13.788430 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.788278 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-istiod-ca-cert\") pod \"e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:13.788430 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.788316 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-workload-socket\") pod \"e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:13.788430 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.788358 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-istio-token\") pod \"e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:13.788430 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.788383 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-workload-certs\") pod \"e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:13.788430 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.788408 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-istio-podinfo\") pod \"e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:13.788851 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.788462 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-credential-socket\") pod \"e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:13.788851 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.788484 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-istio-data\") pod \"e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:13.788957 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.788893 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-workload-socket\") pod \"e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:13.789017 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.788954 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-workload-certs\") pod \"e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:13.789184 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.789142 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-istiod-ca-cert\") pod \"e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:13.789305 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.789247 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-credential-socket\") pod \"e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:13.789305 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.789274 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-istio-data\") pod \"e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:13.791249 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.791229 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-istio-envoy\") pod \"e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:13.791582 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.791561 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-istio-podinfo\") pod \"e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:13.799510 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.799487 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-istio-token\") pod \"e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:13.799809 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.799787 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs86r\" (UniqueName: \"kubernetes.io/projected/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-kube-api-access-hs86r\") pod \"e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:13.987518 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:13.987437 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:14.124129 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:14.124100 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9"] Jul 10 19:37:14.127134 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:37:14.127108 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd98c4594_3a8e_423f_9c8d_62fc6fca2a8c.slice/crio-93ba00acf33c1fdcba9b75ba8b1c2a1606607d9595f7534eb39fde0e0a718dc5 WatchSource:0}: Error finding container 93ba00acf33c1fdcba9b75ba8b1c2a1606607d9595f7534eb39fde0e0a718dc5: Status 404 returned error can't find the container with id 93ba00acf33c1fdcba9b75ba8b1c2a1606607d9595f7534eb39fde0e0a718dc5 Jul 10 19:37:14.129728 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:14.129691 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:37:14.129798 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:14.129772 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:37:14.129836 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:14.129811 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:37:14.191909 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:14.191883 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:14.604024 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:14.603989 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" event={"ID":"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c","Type":"ContainerStarted","Data":"662b365518b6a7688044e56c3bd7310b2a787fdc0fa114e1d723a85b6ea5b469"} Jul 10 19:37:14.604198 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:14.604081 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" event={"ID":"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c","Type":"ContainerStarted","Data":"93ba00acf33c1fdcba9b75ba8b1c2a1606607d9595f7534eb39fde0e0a718dc5"} Jul 10 19:37:14.626516 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:14.626462 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" podStartSLOduration=1.626428135 podStartE2EDuration="1.626428135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:37:14.625789232 +0000 UTC m=+2325.955930090" watchObservedRunningTime="2026-07-10 19:37:14.626428135 +0000 UTC m=+2325.956568945" Jul 10 19:37:14.988195 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:14.988168 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:15.992706 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:15.992671 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:16.100880 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:16.100850 2560 patch_prober.go:28] interesting pod/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 19:37:16.101028 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:16.100901 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" podUID="5c428018-2c69-4300-affa-5ee255940f86" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 19:37:16.614930 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:16.614892 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:16.615903 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:16.615858 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:16.651394 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:16.651364 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:16.725944 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:16.725907 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:16.730463 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:16.730419 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9d1921-565d546d46-87vsv"] Jul 10 19:37:16.731134 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:16.731118 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa8962f2-d40b-4c87-b616-f954c2af8e6b" containerName="istio-proxy" Jul 10 19:37:16.734565 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:16.734547 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d1921-565d546d46-87vsv" Jul 10 19:37:16.737452 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:16.737300 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-9d1921-serving-cert\"" Jul 10 19:37:16.737452 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:16.737316 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 19:37:16.737452 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:16.737404 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-44lh8\"" Jul 10 19:37:16.737866 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:16.737847 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 19:37:16.744300 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:16.744273 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9d1921-565d546d46-87vsv"] Jul 10 19:37:16.795381 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:16.795355 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:16.817502 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:16.817481 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx2vr\" (UniqueName: \"kubernetes.io/projected/22df21e7-d0a6-4e9f-97bf-132de73f5670-kube-api-access-gx2vr\") pod \"maas-api-e2e-api-a-9d1921-565d546d46-87vsv\" (UID: \"22df21e7-d0a6-4e9f-97bf-132de73f5670\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d1921-565d546d46-87vsv" Jul 10 19:37:16.817594 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:16.817578 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22df21e7-d0a6-4e9f-97bf-132de73f5670-maas-api-tls\") pod \"maas-api-e2e-api-a-9d1921-565d546d46-87vsv\" (UID: \"22df21e7-d0a6-4e9f-97bf-132de73f5670\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d1921-565d546d46-87vsv" Jul 10 19:37:16.817655 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:16.817620 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22df21e7-d0a6-4e9f-97bf-132de73f5670-openshift-service-ca\") pod \"maas-api-e2e-api-a-9d1921-565d546d46-87vsv\" (UID: \"22df21e7-d0a6-4e9f-97bf-132de73f5670\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d1921-565d546d46-87vsv" Jul 10 19:37:16.918629 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:16.918600 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22df21e7-d0a6-4e9f-97bf-132de73f5670-maas-api-tls\") pod \"maas-api-e2e-api-a-9d1921-565d546d46-87vsv\" (UID: \"22df21e7-d0a6-4e9f-97bf-132de73f5670\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d1921-565d546d46-87vsv" Jul 10 19:37:16.918821 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:16.918665 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22df21e7-d0a6-4e9f-97bf-132de73f5670-openshift-service-ca\") pod \"maas-api-e2e-api-a-9d1921-565d546d46-87vsv\" (UID: \"22df21e7-d0a6-4e9f-97bf-132de73f5670\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d1921-565d546d46-87vsv" Jul 10 19:37:16.918821 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:16.918703 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx2vr\" (UniqueName: \"kubernetes.io/projected/22df21e7-d0a6-4e9f-97bf-132de73f5670-kube-api-access-gx2vr\") pod \"maas-api-e2e-api-a-9d1921-565d546d46-87vsv\" (UID: \"22df21e7-d0a6-4e9f-97bf-132de73f5670\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d1921-565d546d46-87vsv" Jul 10 19:37:16.919292 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:16.919272 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22df21e7-d0a6-4e9f-97bf-132de73f5670-openshift-service-ca\") pod \"maas-api-e2e-api-a-9d1921-565d546d46-87vsv\" (UID: \"22df21e7-d0a6-4e9f-97bf-132de73f5670\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d1921-565d546d46-87vsv" Jul 10 19:37:16.921000 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:16.920972 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22df21e7-d0a6-4e9f-97bf-132de73f5670-maas-api-tls\") pod \"maas-api-e2e-api-a-9d1921-565d546d46-87vsv\" (UID: \"22df21e7-d0a6-4e9f-97bf-132de73f5670\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d1921-565d546d46-87vsv" Jul 10 19:37:16.929202 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:16.929181 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx2vr\" (UniqueName: \"kubernetes.io/projected/22df21e7-d0a6-4e9f-97bf-132de73f5670-kube-api-access-gx2vr\") pod \"maas-api-e2e-api-a-9d1921-565d546d46-87vsv\" (UID: \"22df21e7-d0a6-4e9f-97bf-132de73f5670\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d1921-565d546d46-87vsv" Jul 10 19:37:16.960729 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:16.960699 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:17.047506 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:17.047475 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d1921-565d546d46-87vsv" Jul 10 19:37:17.188894 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:17.188866 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9d1921-565d546d46-87vsv"] Jul 10 19:37:17.190355 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:37:17.190333 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22df21e7_d0a6_4e9f_97bf_132de73f5670.slice/crio-0ffe8b967124193aa26e6f5e53eb6af581e4476ebc664707f6394eedabdf97b2 WatchSource:0}: Error finding container 0ffe8b967124193aa26e6f5e53eb6af581e4476ebc664707f6394eedabdf97b2: Status 404 returned error can't find the container with id 0ffe8b967124193aa26e6f5e53eb6af581e4476ebc664707f6394eedabdf97b2 Jul 10 19:37:17.191845 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:17.191826 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 19:37:17.391307 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:17.391262 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:17.490291 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:17.490217 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:17.621831 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:17.621788 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d1921-565d546d46-87vsv" event={"ID":"22df21e7-d0a6-4e9f-97bf-132de73f5670","Type":"ContainerStarted","Data":"0ffe8b967124193aa26e6f5e53eb6af581e4476ebc664707f6394eedabdf97b2"} Jul 10 19:37:17.871490 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:17.871465 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:37:17.930529 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:17.930507 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c428018-2c69-4300-affa-5ee255940f86-istio-podinfo\") pod \"5c428018-2c69-4300-affa-5ee255940f86\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " Jul 10 19:37:17.930671 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:17.930553 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c428018-2c69-4300-affa-5ee255940f86-workload-certs\") pod \"5c428018-2c69-4300-affa-5ee255940f86\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " Jul 10 19:37:17.930671 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:17.930608 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c428018-2c69-4300-affa-5ee255940f86-istiod-ca-cert\") pod \"5c428018-2c69-4300-affa-5ee255940f86\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " Jul 10 19:37:17.930671 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:17.930636 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c428018-2c69-4300-affa-5ee255940f86-istio-token\") pod \"5c428018-2c69-4300-affa-5ee255940f86\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " Jul 10 19:37:17.930830 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:17.930672 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65sq7\" (UniqueName: \"kubernetes.io/projected/5c428018-2c69-4300-affa-5ee255940f86-kube-api-access-65sq7\") pod \"5c428018-2c69-4300-affa-5ee255940f86\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " Jul 10 19:37:17.930830 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:17.930707 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c428018-2c69-4300-affa-5ee255940f86-istio-envoy\") pod \"5c428018-2c69-4300-affa-5ee255940f86\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " Jul 10 19:37:17.930830 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:17.930754 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c428018-2c69-4300-affa-5ee255940f86-credential-socket\") pod \"5c428018-2c69-4300-affa-5ee255940f86\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " Jul 10 19:37:17.930830 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:17.930792 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c428018-2c69-4300-affa-5ee255940f86-workload-socket\") pod \"5c428018-2c69-4300-affa-5ee255940f86\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " Jul 10 19:37:17.931017 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:17.930851 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c428018-2c69-4300-affa-5ee255940f86-istio-data\") pod \"5c428018-2c69-4300-affa-5ee255940f86\" (UID: \"5c428018-2c69-4300-affa-5ee255940f86\") " Jul 10 19:37:17.931017 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:17.930879 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c428018-2c69-4300-affa-5ee255940f86-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5c428018-2c69-4300-affa-5ee255940f86" (UID: "5c428018-2c69-4300-affa-5ee255940f86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:37:17.931112 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:17.931077 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c428018-2c69-4300-affa-5ee255940f86-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5c428018-2c69-4300-affa-5ee255940f86" (UID: "5c428018-2c69-4300-affa-5ee255940f86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:37:17.931168 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:17.931122 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c428018-2c69-4300-affa-5ee255940f86-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5c428018-2c69-4300-affa-5ee255940f86" (UID: "5c428018-2c69-4300-affa-5ee255940f86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:37:17.931350 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:17.931283 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c428018-2c69-4300-affa-5ee255940f86-workload-certs\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:37:17.931350 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:17.931308 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c428018-2c69-4300-affa-5ee255940f86-credential-socket\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:37:17.931350 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:17.931322 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c428018-2c69-4300-affa-5ee255940f86-workload-socket\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:37:17.931811 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:17.931775 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c428018-2c69-4300-affa-5ee255940f86-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5c428018-2c69-4300-affa-5ee255940f86" (UID: "5c428018-2c69-4300-affa-5ee255940f86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:37:17.932163 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:17.932116 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c428018-2c69-4300-affa-5ee255940f86-istio-data" (OuterVolumeSpecName: "istio-data") pod "5c428018-2c69-4300-affa-5ee255940f86" (UID: "5c428018-2c69-4300-affa-5ee255940f86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:37:17.933214 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:17.933189 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c428018-2c69-4300-affa-5ee255940f86-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5c428018-2c69-4300-affa-5ee255940f86" (UID: "5c428018-2c69-4300-affa-5ee255940f86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 19:37:17.933299 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:17.933284 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c428018-2c69-4300-affa-5ee255940f86-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5c428018-2c69-4300-affa-5ee255940f86" (UID: "5c428018-2c69-4300-affa-5ee255940f86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:37:17.933584 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:17.933560 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c428018-2c69-4300-affa-5ee255940f86-kube-api-access-65sq7" (OuterVolumeSpecName: "kube-api-access-65sq7") pod "5c428018-2c69-4300-affa-5ee255940f86" (UID: "5c428018-2c69-4300-affa-5ee255940f86"). InnerVolumeSpecName "kube-api-access-65sq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:37:17.933833 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:17.933814 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c428018-2c69-4300-affa-5ee255940f86-istio-token" (OuterVolumeSpecName: "istio-token") pod "5c428018-2c69-4300-affa-5ee255940f86" (UID: "5c428018-2c69-4300-affa-5ee255940f86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:37:18.032335 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:18.032283 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c428018-2c69-4300-affa-5ee255940f86-istio-data\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:37:18.032335 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:18.032303 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c428018-2c69-4300-affa-5ee255940f86-istio-podinfo\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:37:18.032335 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:18.032312 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c428018-2c69-4300-affa-5ee255940f86-istiod-ca-cert\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:37:18.032335 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:18.032320 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c428018-2c69-4300-affa-5ee255940f86-istio-token\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:37:18.032335 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:18.032328 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65sq7\" (UniqueName: \"kubernetes.io/projected/5c428018-2c69-4300-affa-5ee255940f86-kube-api-access-65sq7\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:37:18.032335 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:18.032337 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c428018-2c69-4300-affa-5ee255940f86-istio-envoy\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:37:18.628699 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:18.628662 2560 generic.go:356] "Generic (PLEG): container finished" podID="5c428018-2c69-4300-affa-5ee255940f86" containerID="646ea2f2fefd43daa18c5378ed93f81d214c592e7176ce8f0829ce65117eaa95" exitCode=0 Jul 10 19:37:18.629168 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:18.628716 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" Jul 10 19:37:18.629168 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:18.628745 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" event={"ID":"5c428018-2c69-4300-affa-5ee255940f86","Type":"ContainerDied","Data":"646ea2f2fefd43daa18c5378ed93f81d214c592e7176ce8f0829ce65117eaa95"} Jul 10 19:37:18.629168 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:18.628794 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx" event={"ID":"5c428018-2c69-4300-affa-5ee255940f86","Type":"ContainerDied","Data":"fe3191e32ac9bc8749b9db66630460753de11a7304bee4ca59289a9248a70fb0"} Jul 10 19:37:18.629168 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:18.628816 2560 scope.go:117] "RemoveContainer" containerID="646ea2f2fefd43daa18c5378ed93f81d214c592e7176ce8f0829ce65117eaa95" Jul 10 19:37:18.630327 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:18.630294 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d1921-565d546d46-87vsv" event={"ID":"22df21e7-d0a6-4e9f-97bf-132de73f5670","Type":"ContainerStarted","Data":"94c35f8016c2fd73e9492e919f4ca97796ab1285ce90ea06b65d7fc196d6690b"} Jul 10 19:37:18.637712 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:18.637695 2560 scope.go:117] "RemoveContainer" containerID="fb6610215f9498ad3f4e8e46b41ef0e49d06f545f7708f6b8e9281d6fdceeb74" Jul 10 19:37:18.645590 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:18.645574 2560 scope.go:117] "RemoveContainer" containerID="646ea2f2fefd43daa18c5378ed93f81d214c592e7176ce8f0829ce65117eaa95" Jul 10 19:37:18.645844 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:37:18.645827 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"646ea2f2fefd43daa18c5378ed93f81d214c592e7176ce8f0829ce65117eaa95\": container with ID starting with 646ea2f2fefd43daa18c5378ed93f81d214c592e7176ce8f0829ce65117eaa95 not found: ID does not exist" containerID="646ea2f2fefd43daa18c5378ed93f81d214c592e7176ce8f0829ce65117eaa95" Jul 10 19:37:18.645883 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:18.645853 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"646ea2f2fefd43daa18c5378ed93f81d214c592e7176ce8f0829ce65117eaa95"} err="failed to get container status \"646ea2f2fefd43daa18c5378ed93f81d214c592e7176ce8f0829ce65117eaa95\": rpc error: code = NotFound desc = could not find container \"646ea2f2fefd43daa18c5378ed93f81d214c592e7176ce8f0829ce65117eaa95\": container with ID starting with 646ea2f2fefd43daa18c5378ed93f81d214c592e7176ce8f0829ce65117eaa95 not found: ID does not exist" Jul 10 19:37:18.645883 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:18.645870 2560 scope.go:117] "RemoveContainer" containerID="fb6610215f9498ad3f4e8e46b41ef0e49d06f545f7708f6b8e9281d6fdceeb74" Jul 10 19:37:18.646074 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:37:18.646059 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb6610215f9498ad3f4e8e46b41ef0e49d06f545f7708f6b8e9281d6fdceeb74\": container with ID starting with fb6610215f9498ad3f4e8e46b41ef0e49d06f545f7708f6b8e9281d6fdceeb74 not found: ID does not exist" containerID="fb6610215f9498ad3f4e8e46b41ef0e49d06f545f7708f6b8e9281d6fdceeb74" Jul 10 19:37:18.646110 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:18.646080 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb6610215f9498ad3f4e8e46b41ef0e49d06f545f7708f6b8e9281d6fdceeb74"} err="failed to get container status \"fb6610215f9498ad3f4e8e46b41ef0e49d06f545f7708f6b8e9281d6fdceeb74\": rpc error: code = NotFound desc = could not find container \"fb6610215f9498ad3f4e8e46b41ef0e49d06f545f7708f6b8e9281d6fdceeb74\": container with ID starting with fb6610215f9498ad3f4e8e46b41ef0e49d06f545f7708f6b8e9281d6fdceeb74 not found: ID does not exist" Jul 10 19:37:18.654227 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:18.654175 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d1921-565d546d46-87vsv" podStartSLOduration=2.149495314 podStartE2EDuration="2.654159606s" totalImagesPullingTime="504.664292ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:37:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:37:18.653427258 +0000 UTC m=+2329.983568102" watchObservedRunningTime="2026-07-10 19:37:18.654159606 +0000 UTC m=+2329.984300466" Jul 10 19:37:18.668380 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:18.668355 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx"] Jul 10 19:37:18.672045 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:18.672027 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8c42c2-openshift-default-7c6c74f5f5-mzgqx"] Jul 10 19:37:19.253899 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:19.253867 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c428018-2c69-4300-affa-5ee255940f86" path="/var/lib/kubelet/pods/5c428018-2c69-4300-affa-5ee255940f86/volumes" Jul 10 19:37:19.636483 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:19.636392 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d1921-565d546d46-87vsv" Jul 10 19:37:21.233916 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:21.233827 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:21.764492 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:21.764435 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:22.056955 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.056884 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj"] Jul 10 19:37:22.057285 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.057273 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c428018-2c69-4300-affa-5ee255940f86" containerName="istio-proxy" Jul 10 19:37:22.057323 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.057286 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c428018-2c69-4300-affa-5ee255940f86" containerName="istio-proxy" Jul 10 19:37:22.057375 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.057367 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c428018-2c69-4300-affa-5ee255940f86" containerName="istio-proxy" Jul 10 19:37:22.057415 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.057377 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c428018-2c69-4300-affa-5ee255940f86" containerName="istio-proxy" Jul 10 19:37:22.062429 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.062408 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:22.067710 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.067681 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e78db6-openshift-default-dockercfg-rc797\"" Jul 10 19:37:22.077133 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.077108 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj"] Jul 10 19:37:22.097695 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.097669 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:22.170490 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.170454 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc4bb\" (UniqueName: \"kubernetes.io/projected/5054fc8a-c200-4de3-8a55-8f1f4b968044-kube-api-access-pc4bb\") pod \"e2e-api-b-e78db6-openshift-default-564c5784fc-755dj\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:22.170826 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.170605 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5054fc8a-c200-4de3-8a55-8f1f4b968044-istio-podinfo\") pod \"e2e-api-b-e78db6-openshift-default-564c5784fc-755dj\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:22.170826 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.170652 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5054fc8a-c200-4de3-8a55-8f1f4b968044-workload-socket\") pod \"e2e-api-b-e78db6-openshift-default-564c5784fc-755dj\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:22.170826 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.170682 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5054fc8a-c200-4de3-8a55-8f1f4b968044-credential-socket\") pod \"e2e-api-b-e78db6-openshift-default-564c5784fc-755dj\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:22.170826 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.170745 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5054fc8a-c200-4de3-8a55-8f1f4b968044-workload-certs\") pod \"e2e-api-b-e78db6-openshift-default-564c5784fc-755dj\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:22.170826 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.170793 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5054fc8a-c200-4de3-8a55-8f1f4b968044-istio-data\") pod \"e2e-api-b-e78db6-openshift-default-564c5784fc-755dj\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:22.170826 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.170830 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5054fc8a-c200-4de3-8a55-8f1f4b968044-istiod-ca-cert\") pod \"e2e-api-b-e78db6-openshift-default-564c5784fc-755dj\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:22.171193 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.170874 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5054fc8a-c200-4de3-8a55-8f1f4b968044-istio-token\") pod \"e2e-api-b-e78db6-openshift-default-564c5784fc-755dj\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:22.171193 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.170937 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5054fc8a-c200-4de3-8a55-8f1f4b968044-istio-envoy\") pod \"e2e-api-b-e78db6-openshift-default-564c5784fc-755dj\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:22.251858 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.251824 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:22.271759 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.271736 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5054fc8a-c200-4de3-8a55-8f1f4b968044-istio-podinfo\") pod \"e2e-api-b-e78db6-openshift-default-564c5784fc-755dj\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:22.271882 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.271766 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5054fc8a-c200-4de3-8a55-8f1f4b968044-workload-socket\") pod \"e2e-api-b-e78db6-openshift-default-564c5784fc-755dj\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:22.271882 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.271787 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5054fc8a-c200-4de3-8a55-8f1f4b968044-credential-socket\") pod \"e2e-api-b-e78db6-openshift-default-564c5784fc-755dj\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:22.271882 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.271836 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5054fc8a-c200-4de3-8a55-8f1f4b968044-workload-certs\") pod \"e2e-api-b-e78db6-openshift-default-564c5784fc-755dj\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:22.271882 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.271866 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5054fc8a-c200-4de3-8a55-8f1f4b968044-istio-data\") pod \"e2e-api-b-e78db6-openshift-default-564c5784fc-755dj\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:22.272055 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.271901 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5054fc8a-c200-4de3-8a55-8f1f4b968044-istiod-ca-cert\") pod \"e2e-api-b-e78db6-openshift-default-564c5784fc-755dj\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:22.272055 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.272029 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5054fc8a-c200-4de3-8a55-8f1f4b968044-istio-token\") pod \"e2e-api-b-e78db6-openshift-default-564c5784fc-755dj\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:22.272151 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.272098 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5054fc8a-c200-4de3-8a55-8f1f4b968044-istio-envoy\") pod \"e2e-api-b-e78db6-openshift-default-564c5784fc-755dj\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:22.272208 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.272166 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc4bb\" (UniqueName: \"kubernetes.io/projected/5054fc8a-c200-4de3-8a55-8f1f4b968044-kube-api-access-pc4bb\") pod \"e2e-api-b-e78db6-openshift-default-564c5784fc-755dj\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:22.272268 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.272227 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5054fc8a-c200-4de3-8a55-8f1f4b968044-credential-socket\") pod \"e2e-api-b-e78db6-openshift-default-564c5784fc-755dj\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:22.272325 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.272296 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5054fc8a-c200-4de3-8a55-8f1f4b968044-workload-socket\") pod \"e2e-api-b-e78db6-openshift-default-564c5784fc-755dj\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:22.272514 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.272483 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5054fc8a-c200-4de3-8a55-8f1f4b968044-istio-data\") pod \"e2e-api-b-e78db6-openshift-default-564c5784fc-755dj\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:22.272640 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.272516 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5054fc8a-c200-4de3-8a55-8f1f4b968044-workload-certs\") pod \"e2e-api-b-e78db6-openshift-default-564c5784fc-755dj\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:22.272700 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.272634 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5054fc8a-c200-4de3-8a55-8f1f4b968044-istiod-ca-cert\") pod \"e2e-api-b-e78db6-openshift-default-564c5784fc-755dj\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:22.274370 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.274346 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5054fc8a-c200-4de3-8a55-8f1f4b968044-istio-envoy\") pod \"e2e-api-b-e78db6-openshift-default-564c5784fc-755dj\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:22.274582 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.274566 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5054fc8a-c200-4de3-8a55-8f1f4b968044-istio-podinfo\") pod \"e2e-api-b-e78db6-openshift-default-564c5784fc-755dj\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:22.280549 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.280529 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc4bb\" (UniqueName: \"kubernetes.io/projected/5054fc8a-c200-4de3-8a55-8f1f4b968044-kube-api-access-pc4bb\") pod \"e2e-api-b-e78db6-openshift-default-564c5784fc-755dj\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:22.280698 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.280681 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5054fc8a-c200-4de3-8a55-8f1f4b968044-istio-token\") pod \"e2e-api-b-e78db6-openshift-default-564c5784fc-755dj\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:22.373221 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.373144 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:22.512796 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.512770 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj"] Jul 10 19:37:22.514277 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:37:22.514251 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5054fc8a_c200_4de3_8a55_8f1f4b968044.slice/crio-27b847d38241defa68f86da6788ba732a199d4b8d99982cee6dac90fa8f701d5 WatchSource:0}: Error finding container 27b847d38241defa68f86da6788ba732a199d4b8d99982cee6dac90fa8f701d5: Status 404 returned error can't find the container with id 27b847d38241defa68f86da6788ba732a199d4b8d99982cee6dac90fa8f701d5 Jul 10 19:37:22.516452 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.516406 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:37:22.516520 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.516490 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:37:22.516558 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.516519 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:37:22.655619 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.655528 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" event={"ID":"5054fc8a-c200-4de3-8a55-8f1f4b968044","Type":"ContainerStarted","Data":"7a2cace6de5c3159bdd49dbc35551c8842712730951efe9b2ab73e95a6f16204"} Jul 10 19:37:22.655619 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.655590 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" event={"ID":"5054fc8a-c200-4de3-8a55-8f1f4b968044","Type":"ContainerStarted","Data":"27b847d38241defa68f86da6788ba732a199d4b8d99982cee6dac90fa8f701d5"} Jul 10 19:37:22.678063 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:22.678010 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" podStartSLOduration=0.677994814 podStartE2EDuration="677.994814ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:37:22.675832831 +0000 UTC m=+2334.005973678" watchObservedRunningTime="2026-07-10 19:37:22.677994814 +0000 UTC m=+2334.008135624" Jul 10 19:37:23.373270 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:23.373238 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:23.378081 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:23.378056 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:23.661066 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:23.661036 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:23.661990 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:23.661972 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:24.675595 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:24.675557 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:24.776746 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:24.776713 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp"] Jul 10 19:37:24.777141 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:24.777127 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c428018-2c69-4300-affa-5ee255940f86" containerName="istio-proxy" Jul 10 19:37:24.777191 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:24.777145 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c428018-2c69-4300-affa-5ee255940f86" containerName="istio-proxy" Jul 10 19:37:24.780368 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:24.780319 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp" Jul 10 19:37:24.781868 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:24.781843 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:24.783560 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:24.783541 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-e78db6-serving-cert\"" Jul 10 19:37:24.792028 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:24.792007 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp"] Jul 10 19:37:24.848466 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:24.845528 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:24.897794 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:24.897738 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4d68be2-cd64-4813-be44-131f011bfb03-openshift-service-ca\") pod \"maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp\" (UID: \"c4d68be2-cd64-4813-be44-131f011bfb03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp" Jul 10 19:37:24.897988 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:24.897822 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j97qb\" (UniqueName: \"kubernetes.io/projected/c4d68be2-cd64-4813-be44-131f011bfb03-kube-api-access-j97qb\") pod \"maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp\" (UID: \"c4d68be2-cd64-4813-be44-131f011bfb03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp" Jul 10 19:37:24.897988 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:24.897905 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4d68be2-cd64-4813-be44-131f011bfb03-maas-api-tls\") pod \"maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp\" (UID: \"c4d68be2-cd64-4813-be44-131f011bfb03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp" Jul 10 19:37:24.999222 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:24.999135 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4d68be2-cd64-4813-be44-131f011bfb03-openshift-service-ca\") pod \"maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp\" (UID: \"c4d68be2-cd64-4813-be44-131f011bfb03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp" Jul 10 19:37:24.999222 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:24.999183 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j97qb\" (UniqueName: \"kubernetes.io/projected/c4d68be2-cd64-4813-be44-131f011bfb03-kube-api-access-j97qb\") pod \"maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp\" (UID: \"c4d68be2-cd64-4813-be44-131f011bfb03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp" Jul 10 19:37:24.999222 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:24.999223 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4d68be2-cd64-4813-be44-131f011bfb03-maas-api-tls\") pod \"maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp\" (UID: \"c4d68be2-cd64-4813-be44-131f011bfb03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp" Jul 10 19:37:24.999542 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:37:24.999359 2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-e78db6-serving-cert: secret "maas-api-e2e-api-b-e78db6-serving-cert" not found Jul 10 19:37:24.999542 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:37:24.999412 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4d68be2-cd64-4813-be44-131f011bfb03-maas-api-tls podName:c4d68be2-cd64-4813-be44-131f011bfb03 nodeName:}" failed. No retries permitted until 2026-07-10 19:37:25.499396037 +0000 UTC m=+2336.829536825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c4d68be2-cd64-4813-be44-131f011bfb03-maas-api-tls") pod "maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp" (UID: "c4d68be2-cd64-4813-be44-131f011bfb03") : secret "maas-api-e2e-api-b-e78db6-serving-cert" not found Jul 10 19:37:24.999855 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:24.999833 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4d68be2-cd64-4813-be44-131f011bfb03-openshift-service-ca\") pod \"maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp\" (UID: \"c4d68be2-cd64-4813-be44-131f011bfb03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp" Jul 10 19:37:25.008168 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:25.008144 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j97qb\" (UniqueName: \"kubernetes.io/projected/c4d68be2-cd64-4813-be44-131f011bfb03-kube-api-access-j97qb\") pod \"maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp\" (UID: \"c4d68be2-cd64-4813-be44-131f011bfb03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp" Jul 10 19:37:25.385243 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:25.385158 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:25.481014 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:25.480975 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:25.505341 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:25.505318 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4d68be2-cd64-4813-be44-131f011bfb03-maas-api-tls\") pod \"maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp\" (UID: \"c4d68be2-cd64-4813-be44-131f011bfb03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp" Jul 10 19:37:25.508055 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:25.508038 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4d68be2-cd64-4813-be44-131f011bfb03-maas-api-tls\") pod \"maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp\" (UID: \"c4d68be2-cd64-4813-be44-131f011bfb03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp" Jul 10 19:37:25.581739 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:25.581702 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:25.648252 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:25.648082 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d1921-565d546d46-87vsv" Jul 10 19:37:25.695071 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:25.695038 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp" Jul 10 19:37:25.885718 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:25.885695 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp"] Jul 10 19:37:25.887468 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:37:25.887426 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4d68be2_cd64_4813_be44_131f011bfb03.slice/crio-a689f3e87e105e44ef37f863d6bbd3552c16b8b15029cd2f635f910cd760c1b6 WatchSource:0}: Error finding container a689f3e87e105e44ef37f863d6bbd3552c16b8b15029cd2f635f910cd760c1b6: Status 404 returned error can't find the container with id a689f3e87e105e44ef37f863d6bbd3552c16b8b15029cd2f635f910cd760c1b6 Jul 10 19:37:26.679185 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:26.679148 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp" event={"ID":"c4d68be2-cd64-4813-be44-131f011bfb03","Type":"ContainerStarted","Data":"3f78025ec2fca07aa165e478e3f4ba728c4ab13c75abec4149f898000a7b48a9"} Jul 10 19:37:26.679370 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:26.679199 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp" event={"ID":"c4d68be2-cd64-4813-be44-131f011bfb03","Type":"ContainerStarted","Data":"a689f3e87e105e44ef37f863d6bbd3552c16b8b15029cd2f635f910cd760c1b6"} Jul 10 19:37:26.679370 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:26.679321 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp" Jul 10 19:37:26.699846 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:26.699802 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp" podStartSLOduration=2.246383827 podStartE2EDuration="2.699785953s" totalImagesPullingTime="453.402126ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:37:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:37:26.69723714 +0000 UTC m=+2338.027377987" watchObservedRunningTime="2026-07-10 19:37:26.699785953 +0000 UTC m=+2338.029926762" Jul 10 19:37:29.260101 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:29.260066 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:30.703093 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:30.703053 2560 generic.go:356] "Generic (PLEG): container finished" podID="d98c4594-3a8e-423f-9c8d-62fc6fca2a8c" containerID="662b365518b6a7688044e56c3bd7310b2a787fdc0fa114e1d723a85b6ea5b469" exitCode=0 Jul 10 19:37:30.703576 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:30.703127 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" event={"ID":"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c","Type":"ContainerDied","Data":"662b365518b6a7688044e56c3bd7310b2a787fdc0fa114e1d723a85b6ea5b469"} Jul 10 19:37:30.703576 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:30.703525 2560 scope.go:117] "RemoveContainer" containerID="662b365518b6a7688044e56c3bd7310b2a787fdc0fa114e1d723a85b6ea5b469" Jul 10 19:37:30.704493 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:30.704465 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:37:30.704572 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:30.704532 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:37:30.704572 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:30.704566 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:37:31.615416 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:31.615384 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:31.710388 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:31.710351 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" event={"ID":"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c","Type":"ContainerStarted","Data":"e00a2b185b7bdeaf981d2ced5954cad2740e490e79d29f9e0892d28a4d09ebfd"} Jul 10 19:37:31.987600 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:31.987559 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:31.988671 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:31.988647 2560 patch_prober.go:28] interesting pod/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 10 19:37:31.988795 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:31.988695 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" podUID="d98c4594-3a8e-423f-9c8d-62fc6fca2a8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 10 19:37:32.690402 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:32.690372 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp" Jul 10 19:37:32.988399 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:32.988311 2560 patch_prober.go:28] interesting pod/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 10 19:37:32.988399 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:32.988378 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" podUID="d98c4594-3a8e-423f-9c8d-62fc6fca2a8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 10 19:37:33.987833 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:33.987794 2560 patch_prober.go:28] interesting pod/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 10 19:37:33.988013 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:33.987861 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" podUID="d98c4594-3a8e-423f-9c8d-62fc6fca2a8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 10 19:37:34.988644 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:34.988607 2560 patch_prober.go:28] interesting pod/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 10 19:37:34.989025 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:34.988701 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" podUID="d98c4594-3a8e-423f-9c8d-62fc6fca2a8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 10 19:37:35.988173 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:35.988137 2560 patch_prober.go:28] interesting pod/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 10 19:37:35.988361 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:35.988230 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" podUID="d98c4594-3a8e-423f-9c8d-62fc6fca2a8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 10 19:37:36.991727 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:36.991693 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:36.992172 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:36.991986 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:36.992773 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:36.992758 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:37:38.324774 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:38.324734 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:38.424520 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:38.424487 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:39.702615 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:39.702568 2560 patch_prober.go:28] interesting pod/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 19:37:39.703063 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:39.702636 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" podUID="5054fc8a-c200-4de3-8a55-8f1f4b968044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 19:37:41.774201 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:41.774166 2560 generic.go:356] "Generic (PLEG): container finished" podID="5054fc8a-c200-4de3-8a55-8f1f4b968044" containerID="7a2cace6de5c3159bdd49dbc35551c8842712730951efe9b2ab73e95a6f16204" exitCode=0 Jul 10 19:37:41.774622 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:41.774238 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" event={"ID":"5054fc8a-c200-4de3-8a55-8f1f4b968044","Type":"ContainerDied","Data":"7a2cace6de5c3159bdd49dbc35551c8842712730951efe9b2ab73e95a6f16204"} Jul 10 19:37:41.774673 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:41.774654 2560 scope.go:117] "RemoveContainer" containerID="7a2cace6de5c3159bdd49dbc35551c8842712730951efe9b2ab73e95a6f16204" Jul 10 19:37:41.775573 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:41.775547 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:37:41.775638 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:41.775618 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:37:41.775678 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:41.775650 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:37:42.781582 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:42.781546 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" event={"ID":"5054fc8a-c200-4de3-8a55-8f1f4b968044","Type":"ContainerStarted","Data":"142f0d6cd291826c277f60c40a3abf55a8b2164cbcfd40e439b25e2dd1ab1e1e"} Jul 10 19:37:43.373789 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:43.373756 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:43.374017 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:43.373994 2560 patch_prober.go:28] interesting pod/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 10 19:37:43.374068 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:43.374053 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" podUID="5054fc8a-c200-4de3-8a55-8f1f4b968044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 10 19:37:44.374235 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:44.374196 2560 patch_prober.go:28] interesting pod/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 10 19:37:44.374709 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:44.374254 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" podUID="5054fc8a-c200-4de3-8a55-8f1f4b968044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 10 19:37:45.374006 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:45.373974 2560 patch_prober.go:28] interesting pod/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 10 19:37:45.374176 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:45.374034 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" podUID="5054fc8a-c200-4de3-8a55-8f1f4b968044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 10 19:37:46.174351 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:46.174312 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:46.274123 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:46.274091 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:46.373851 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:46.373815 2560 patch_prober.go:28] interesting pod/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 10 19:37:46.374040 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:46.373876 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" podUID="5054fc8a-c200-4de3-8a55-8f1f4b968044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 10 19:37:47.376940 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:47.376910 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:47.377404 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:47.377288 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:47.377926 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:47.377906 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:37:48.315228 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:48.315195 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:48.416225 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:48.416196 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:48.549210 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:48.549180 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9d1921-565d546d46-87vsv"] Jul 10 19:37:48.549491 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:48.549425 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d1921-565d546d46-87vsv" podUID="22df21e7-d0a6-4e9f-97bf-132de73f5670" containerName="maas-api" containerID="cri-o://94c35f8016c2fd73e9492e919f4ca97796ab1285ce90ea06b65d7fc196d6690b" gracePeriod=30 Jul 10 19:37:48.571965 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:48.571941 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:48.801051 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:48.801027 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d1921-565d546d46-87vsv" Jul 10 19:37:48.812025 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:48.811997 2560 generic.go:356] "Generic (PLEG): container finished" podID="22df21e7-d0a6-4e9f-97bf-132de73f5670" containerID="94c35f8016c2fd73e9492e919f4ca97796ab1285ce90ea06b65d7fc196d6690b" exitCode=0 Jul 10 19:37:48.812154 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:48.812029 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d1921-565d546d46-87vsv" Jul 10 19:37:48.812154 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:48.812083 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d1921-565d546d46-87vsv" event={"ID":"22df21e7-d0a6-4e9f-97bf-132de73f5670","Type":"ContainerDied","Data":"94c35f8016c2fd73e9492e919f4ca97796ab1285ce90ea06b65d7fc196d6690b"} Jul 10 19:37:48.812154 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:48.812129 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d1921-565d546d46-87vsv" event={"ID":"22df21e7-d0a6-4e9f-97bf-132de73f5670","Type":"ContainerDied","Data":"0ffe8b967124193aa26e6f5e53eb6af581e4476ebc664707f6394eedabdf97b2"} Jul 10 19:37:48.812154 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:48.812152 2560 scope.go:117] "RemoveContainer" containerID="94c35f8016c2fd73e9492e919f4ca97796ab1285ce90ea06b65d7fc196d6690b" Jul 10 19:37:48.821742 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:48.821722 2560 scope.go:117] "RemoveContainer" containerID="94c35f8016c2fd73e9492e919f4ca97796ab1285ce90ea06b65d7fc196d6690b" Jul 10 19:37:48.822006 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:37:48.821984 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94c35f8016c2fd73e9492e919f4ca97796ab1285ce90ea06b65d7fc196d6690b\": container with ID starting with 94c35f8016c2fd73e9492e919f4ca97796ab1285ce90ea06b65d7fc196d6690b not found: ID does not exist" containerID="94c35f8016c2fd73e9492e919f4ca97796ab1285ce90ea06b65d7fc196d6690b" Jul 10 19:37:48.822095 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:48.822016 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94c35f8016c2fd73e9492e919f4ca97796ab1285ce90ea06b65d7fc196d6690b"} err="failed to get container status \"94c35f8016c2fd73e9492e919f4ca97796ab1285ce90ea06b65d7fc196d6690b\": rpc error: code = NotFound desc = could not find container \"94c35f8016c2fd73e9492e919f4ca97796ab1285ce90ea06b65d7fc196d6690b\": container with ID starting with 94c35f8016c2fd73e9492e919f4ca97796ab1285ce90ea06b65d7fc196d6690b not found: ID does not exist" Jul 10 19:37:48.852338 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:48.852314 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22df21e7-d0a6-4e9f-97bf-132de73f5670-maas-api-tls\") pod \"22df21e7-d0a6-4e9f-97bf-132de73f5670\" (UID: \"22df21e7-d0a6-4e9f-97bf-132de73f5670\") " Jul 10 19:37:48.852438 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:48.852343 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gx2vr\" (UniqueName: \"kubernetes.io/projected/22df21e7-d0a6-4e9f-97bf-132de73f5670-kube-api-access-gx2vr\") pod \"22df21e7-d0a6-4e9f-97bf-132de73f5670\" (UID: \"22df21e7-d0a6-4e9f-97bf-132de73f5670\") " Jul 10 19:37:48.852438 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:48.852424 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22df21e7-d0a6-4e9f-97bf-132de73f5670-openshift-service-ca\") pod \"22df21e7-d0a6-4e9f-97bf-132de73f5670\" (UID: \"22df21e7-d0a6-4e9f-97bf-132de73f5670\") " Jul 10 19:37:48.852805 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:48.852778 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22df21e7-d0a6-4e9f-97bf-132de73f5670-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "22df21e7-d0a6-4e9f-97bf-132de73f5670" (UID: "22df21e7-d0a6-4e9f-97bf-132de73f5670"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:37:48.854402 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:48.854378 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22df21e7-d0a6-4e9f-97bf-132de73f5670-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "22df21e7-d0a6-4e9f-97bf-132de73f5670" (UID: "22df21e7-d0a6-4e9f-97bf-132de73f5670"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:37:48.854402 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:48.854386 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22df21e7-d0a6-4e9f-97bf-132de73f5670-kube-api-access-gx2vr" (OuterVolumeSpecName: "kube-api-access-gx2vr") pod "22df21e7-d0a6-4e9f-97bf-132de73f5670" (UID: "22df21e7-d0a6-4e9f-97bf-132de73f5670"). InnerVolumeSpecName "kube-api-access-gx2vr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:37:48.953510 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:48.953490 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22df21e7-d0a6-4e9f-97bf-132de73f5670-maas-api-tls\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:37:48.953510 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:48.953510 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx2vr\" (UniqueName: \"kubernetes.io/projected/22df21e7-d0a6-4e9f-97bf-132de73f5670-kube-api-access-gx2vr\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:37:48.953637 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:48.953520 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22df21e7-d0a6-4e9f-97bf-132de73f5670-openshift-service-ca\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:37:49.138294 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:49.138267 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9d1921-565d546d46-87vsv"] Jul 10 19:37:49.142569 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:49.142547 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9d1921-565d546d46-87vsv"] Jul 10 19:37:49.253084 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:49.253055 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22df21e7-d0a6-4e9f-97bf-132de73f5670" path="/var/lib/kubelet/pods/22df21e7-d0a6-4e9f-97bf-132de73f5670/volumes" Jul 10 19:37:50.707791 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:50.707760 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:50.809899 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:50.809872 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:52.868198 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:52.868162 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:58.871433 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:58.871400 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:58.977074 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:37:58.977035 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:03.677027 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:03.676995 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9"] Jul 10 19:38:03.677564 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:03.677186 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" podUID="d98c4594-3a8e-423f-9c8d-62fc6fca2a8c" containerName="istio-proxy" containerID="cri-o://e00a2b185b7bdeaf981d2ced5954cad2740e490e79d29f9e0892d28a4d09ebfd" gracePeriod=30 Jul 10 19:38:03.702102 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:03.702072 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:03.994725 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:03.994634 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:04.808771 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:04.808738 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:04.906177 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:04.906139 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:06.994569 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:06.994533 2560 patch_prober.go:28] interesting pod/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 19:38:06.994941 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:06.994611 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" podUID="d98c4594-3a8e-423f-9c8d-62fc6fca2a8c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 19:38:08.913753 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:08.913725 2560 generic.go:356] "Generic (PLEG): container finished" podID="d98c4594-3a8e-423f-9c8d-62fc6fca2a8c" containerID="e00a2b185b7bdeaf981d2ced5954cad2740e490e79d29f9e0892d28a4d09ebfd" exitCode=0 Jul 10 19:38:08.914062 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:08.913783 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" event={"ID":"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c","Type":"ContainerDied","Data":"e00a2b185b7bdeaf981d2ced5954cad2740e490e79d29f9e0892d28a4d09ebfd"} Jul 10 19:38:08.914062 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:08.913841 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" event={"ID":"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c","Type":"ContainerDied","Data":"93ba00acf33c1fdcba9b75ba8b1c2a1606607d9595f7534eb39fde0e0a718dc5"} Jul 10 19:38:08.914062 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:08.913860 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93ba00acf33c1fdcba9b75ba8b1c2a1606607d9595f7534eb39fde0e0a718dc5" Jul 10 19:38:08.914062 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:08.913874 2560 scope.go:117] "RemoveContainer" containerID="662b365518b6a7688044e56c3bd7310b2a787fdc0fa114e1d723a85b6ea5b469" Jul 10 19:38:08.920032 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:08.920015 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:38:09.033988 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:09.033914 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-istiod-ca-cert\") pod \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " Jul 10 19:38:09.033988 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:09.033948 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-credential-socket\") pod \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " Jul 10 19:38:09.033988 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:09.033986 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-workload-certs\") pod \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " Jul 10 19:38:09.034238 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:09.034036 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-istio-token\") pod \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " Jul 10 19:38:09.034238 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:09.034115 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-istio-podinfo\") pod \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " Jul 10 19:38:09.034238 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:09.034157 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d98c4594-3a8e-423f-9c8d-62fc6fca2a8c" (UID: "d98c4594-3a8e-423f-9c8d-62fc6fca2a8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:38:09.034238 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:09.034206 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-workload-socket\") pod \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " Jul 10 19:38:09.034476 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:09.034244 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hs86r\" (UniqueName: \"kubernetes.io/projected/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-kube-api-access-hs86r\") pod \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " Jul 10 19:38:09.034476 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:09.034261 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-istio-envoy\") pod \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " Jul 10 19:38:09.034476 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:09.034301 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d98c4594-3a8e-423f-9c8d-62fc6fca2a8c" (UID: "d98c4594-3a8e-423f-9c8d-62fc6fca2a8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:38:09.034476 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:09.034326 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-istio-data\") pod \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\" (UID: \"d98c4594-3a8e-423f-9c8d-62fc6fca2a8c\") " Jul 10 19:38:09.034476 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:09.034336 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d98c4594-3a8e-423f-9c8d-62fc6fca2a8c" (UID: "d98c4594-3a8e-423f-9c8d-62fc6fca2a8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:38:09.034476 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:09.034468 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d98c4594-3a8e-423f-9c8d-62fc6fca2a8c" (UID: "d98c4594-3a8e-423f-9c8d-62fc6fca2a8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:38:09.034805 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:09.034738 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-workload-certs\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:38:09.034805 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:09.034757 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-workload-socket\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:38:09.034805 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:09.034771 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-istiod-ca-cert\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:38:09.034805 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:09.034784 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-credential-socket\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:38:09.035152 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:09.035131 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-istio-data" (OuterVolumeSpecName: "istio-data") pod "d98c4594-3a8e-423f-9c8d-62fc6fca2a8c" (UID: "d98c4594-3a8e-423f-9c8d-62fc6fca2a8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:38:09.036403 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:09.036378 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-istio-token" (OuterVolumeSpecName: "istio-token") pod "d98c4594-3a8e-423f-9c8d-62fc6fca2a8c" (UID: "d98c4594-3a8e-423f-9c8d-62fc6fca2a8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:38:09.036403 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:09.036378 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d98c4594-3a8e-423f-9c8d-62fc6fca2a8c" (UID: "d98c4594-3a8e-423f-9c8d-62fc6fca2a8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:38:09.036622 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:09.036603 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d98c4594-3a8e-423f-9c8d-62fc6fca2a8c" (UID: "d98c4594-3a8e-423f-9c8d-62fc6fca2a8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 19:38:09.036680 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:09.036656 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-kube-api-access-hs86r" (OuterVolumeSpecName: "kube-api-access-hs86r") pod "d98c4594-3a8e-423f-9c8d-62fc6fca2a8c" (UID: "d98c4594-3a8e-423f-9c8d-62fc6fca2a8c"). InnerVolumeSpecName "kube-api-access-hs86r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:38:09.135862 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:09.135832 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-istio-podinfo\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:38:09.135862 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:09.135860 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs86r\" (UniqueName: \"kubernetes.io/projected/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-kube-api-access-hs86r\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:38:09.136044 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:09.135869 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-istio-envoy\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:38:09.136044 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:09.135877 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-istio-data\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:38:09.136044 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:09.135885 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c-istio-token\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:38:09.920015 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:09.919987 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9" Jul 10 19:38:09.956039 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:09.956009 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9"] Jul 10 19:38:09.959516 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:09.959492 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-9d1921-openshift-default-f948d4979-qp6w9"] Jul 10 19:38:11.253018 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:11.252981 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d98c4594-3a8e-423f-9c8d-62fc6fca2a8c" path="/var/lib/kubelet/pods/d98c4594-3a8e-423f-9c8d-62fc6fca2a8c/volumes" Jul 10 19:38:14.577190 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:14.577158 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp"] Jul 10 19:38:14.577694 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:14.577401 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp" podUID="c4d68be2-cd64-4813-be44-131f011bfb03" containerName="maas-api" containerID="cri-o://3f78025ec2fca07aa165e478e3f4ba728c4ab13c75abec4149f898000a7b48a9" gracePeriod=30 Jul 10 19:38:14.612406 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:14.612372 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:14.826353 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:14.826331 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp" Jul 10 19:38:14.946674 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:14.946647 2560 generic.go:356] "Generic (PLEG): container finished" podID="c4d68be2-cd64-4813-be44-131f011bfb03" containerID="3f78025ec2fca07aa165e478e3f4ba728c4ab13c75abec4149f898000a7b48a9" exitCode=0 Jul 10 19:38:14.946800 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:14.946687 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp" Jul 10 19:38:14.946800 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:14.946691 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp" event={"ID":"c4d68be2-cd64-4813-be44-131f011bfb03","Type":"ContainerDied","Data":"3f78025ec2fca07aa165e478e3f4ba728c4ab13c75abec4149f898000a7b48a9"} Jul 10 19:38:14.946874 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:14.946833 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp" event={"ID":"c4d68be2-cd64-4813-be44-131f011bfb03","Type":"ContainerDied","Data":"a689f3e87e105e44ef37f863d6bbd3552c16b8b15029cd2f635f910cd760c1b6"} Jul 10 19:38:14.946874 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:14.946861 2560 scope.go:117] "RemoveContainer" containerID="3f78025ec2fca07aa165e478e3f4ba728c4ab13c75abec4149f898000a7b48a9" Jul 10 19:38:14.954903 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:14.954888 2560 scope.go:117] "RemoveContainer" containerID="3f78025ec2fca07aa165e478e3f4ba728c4ab13c75abec4149f898000a7b48a9" Jul 10 19:38:14.955164 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:38:14.955148 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f78025ec2fca07aa165e478e3f4ba728c4ab13c75abec4149f898000a7b48a9\": container with ID starting with 3f78025ec2fca07aa165e478e3f4ba728c4ab13c75abec4149f898000a7b48a9 not found: ID does not exist" containerID="3f78025ec2fca07aa165e478e3f4ba728c4ab13c75abec4149f898000a7b48a9" Jul 10 19:38:14.955206 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:14.955173 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f78025ec2fca07aa165e478e3f4ba728c4ab13c75abec4149f898000a7b48a9"} err="failed to get container status \"3f78025ec2fca07aa165e478e3f4ba728c4ab13c75abec4149f898000a7b48a9\": rpc error: code = NotFound desc = could not find container \"3f78025ec2fca07aa165e478e3f4ba728c4ab13c75abec4149f898000a7b48a9\": container with ID starting with 3f78025ec2fca07aa165e478e3f4ba728c4ab13c75abec4149f898000a7b48a9 not found: ID does not exist" Jul 10 19:38:14.988506 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:14.988485 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4d68be2-cd64-4813-be44-131f011bfb03-maas-api-tls\") pod \"c4d68be2-cd64-4813-be44-131f011bfb03\" (UID: \"c4d68be2-cd64-4813-be44-131f011bfb03\") " Jul 10 19:38:14.988618 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:14.988519 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j97qb\" (UniqueName: \"kubernetes.io/projected/c4d68be2-cd64-4813-be44-131f011bfb03-kube-api-access-j97qb\") pod \"c4d68be2-cd64-4813-be44-131f011bfb03\" (UID: \"c4d68be2-cd64-4813-be44-131f011bfb03\") " Jul 10 19:38:14.988618 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:14.988572 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4d68be2-cd64-4813-be44-131f011bfb03-openshift-service-ca\") pod \"c4d68be2-cd64-4813-be44-131f011bfb03\" (UID: \"c4d68be2-cd64-4813-be44-131f011bfb03\") " Jul 10 19:38:14.988967 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:14.988939 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4d68be2-cd64-4813-be44-131f011bfb03-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c4d68be2-cd64-4813-be44-131f011bfb03" (UID: "c4d68be2-cd64-4813-be44-131f011bfb03"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:38:14.990385 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:14.990356 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4d68be2-cd64-4813-be44-131f011bfb03-kube-api-access-j97qb" (OuterVolumeSpecName: "kube-api-access-j97qb") pod "c4d68be2-cd64-4813-be44-131f011bfb03" (UID: "c4d68be2-cd64-4813-be44-131f011bfb03"). InnerVolumeSpecName "kube-api-access-j97qb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:38:14.990385 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:14.990377 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4d68be2-cd64-4813-be44-131f011bfb03-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c4d68be2-cd64-4813-be44-131f011bfb03" (UID: "c4d68be2-cd64-4813-be44-131f011bfb03"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:38:15.089882 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:15.089863 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4d68be2-cd64-4813-be44-131f011bfb03-maas-api-tls\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:38:15.089882 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:15.089883 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j97qb\" (UniqueName: \"kubernetes.io/projected/c4d68be2-cd64-4813-be44-131f011bfb03-kube-api-access-j97qb\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:38:15.090007 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:15.089893 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4d68be2-cd64-4813-be44-131f011bfb03-openshift-service-ca\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:38:15.271476 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:15.271436 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp"] Jul 10 19:38:15.275148 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:15.275126 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e78db6-85f688d5bf-ghsjp"] Jul 10 19:38:17.252398 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:17.252363 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4d68be2-cd64-4813-be44-131f011bfb03" path="/var/lib/kubelet/pods/c4d68be2-cd64-4813-be44-131f011bfb03/volumes" Jul 10 19:38:19.259413 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:19.259382 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:25.145072 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:25.145034 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:25.226318 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:25.226285 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:29.338670 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:29.338641 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l52t_2a22956a-4e96-41de-945c-a851b587a442/ovn-acl-logging/0.log" Jul 10 19:38:29.339565 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:29.339546 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l52t_2a22956a-4e96-41de-945c-a851b587a442/ovn-acl-logging/0.log" Jul 10 19:38:29.988715 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:29.988680 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj"] Jul 10 19:38:29.988966 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:29.988936 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" podUID="5054fc8a-c200-4de3-8a55-8f1f4b968044" containerName="istio-proxy" containerID="cri-o://142f0d6cd291826c277f60c40a3abf55a8b2164cbcfd40e439b25e2dd1ab1e1e" gracePeriod=30 Jul 10 19:38:30.012950 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:30.012913 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:30.339063 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:30.338978 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:30.698610 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:30.698577 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:31.097783 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.097693 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts"] Jul 10 19:38:31.098272 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.098252 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22df21e7-d0a6-4e9f-97bf-132de73f5670" containerName="maas-api" Jul 10 19:38:31.098318 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.098277 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="22df21e7-d0a6-4e9f-97bf-132de73f5670" containerName="maas-api" Jul 10 19:38:31.098356 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.098322 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d98c4594-3a8e-423f-9c8d-62fc6fca2a8c" containerName="istio-proxy" Jul 10 19:38:31.098356 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.098331 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d98c4594-3a8e-423f-9c8d-62fc6fca2a8c" containerName="istio-proxy" Jul 10 19:38:31.098356 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.098354 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d98c4594-3a8e-423f-9c8d-62fc6fca2a8c" containerName="istio-proxy" Jul 10 19:38:31.098463 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.098362 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d98c4594-3a8e-423f-9c8d-62fc6fca2a8c" containerName="istio-proxy" Jul 10 19:38:31.098463 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.098373 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4d68be2-cd64-4813-be44-131f011bfb03" containerName="maas-api" Jul 10 19:38:31.098463 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.098381 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4d68be2-cd64-4813-be44-131f011bfb03" containerName="maas-api" Jul 10 19:38:31.098553 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.098499 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="22df21e7-d0a6-4e9f-97bf-132de73f5670" containerName="maas-api" Jul 10 19:38:31.098553 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.098509 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d98c4594-3a8e-423f-9c8d-62fc6fca2a8c" containerName="istio-proxy" Jul 10 19:38:31.098553 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.098518 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4d68be2-cd64-4813-be44-131f011bfb03" containerName="maas-api" Jul 10 19:38:31.101983 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.101961 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:31.104817 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.104787 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-7207cb-openshift-default-dockercfg-jhsxr\"" Jul 10 19:38:31.111522 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.111500 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b23f0def-02ec-422f-bc7e-b54897787bc1-istio-podinfo\") pod \"e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:31.111840 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.111816 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b23f0def-02ec-422f-bc7e-b54897787bc1-istiod-ca-cert\") pod \"e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:31.112107 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.112086 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b23f0def-02ec-422f-bc7e-b54897787bc1-istio-token\") pod \"e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:31.112259 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.112234 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sx78\" (UniqueName: \"kubernetes.io/projected/b23f0def-02ec-422f-bc7e-b54897787bc1-kube-api-access-8sx78\") pod \"e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:31.112368 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.112337 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b23f0def-02ec-422f-bc7e-b54897787bc1-credential-socket\") pod \"e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:31.112461 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.112368 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b23f0def-02ec-422f-bc7e-b54897787bc1-istio-envoy\") pod \"e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:31.112461 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.112427 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b23f0def-02ec-422f-bc7e-b54897787bc1-workload-socket\") pod \"e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:31.112944 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.112907 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b23f0def-02ec-422f-bc7e-b54897787bc1-workload-certs\") pod \"e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:31.113041 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.113030 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b23f0def-02ec-422f-bc7e-b54897787bc1-istio-data\") pod \"e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:31.113882 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.113705 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:31.114965 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.114944 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts"] Jul 10 19:38:31.215028 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.214995 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b23f0def-02ec-422f-bc7e-b54897787bc1-istio-podinfo\") pod \"e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:31.215213 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.215054 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b23f0def-02ec-422f-bc7e-b54897787bc1-istiod-ca-cert\") pod \"e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:31.215213 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.215073 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b23f0def-02ec-422f-bc7e-b54897787bc1-istio-token\") pod \"e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:31.215213 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.215090 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sx78\" (UniqueName: \"kubernetes.io/projected/b23f0def-02ec-422f-bc7e-b54897787bc1-kube-api-access-8sx78\") pod \"e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:31.215213 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.215113 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b23f0def-02ec-422f-bc7e-b54897787bc1-credential-socket\") pod \"e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:31.215213 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.215139 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b23f0def-02ec-422f-bc7e-b54897787bc1-istio-envoy\") pod \"e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:31.215213 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.215178 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b23f0def-02ec-422f-bc7e-b54897787bc1-workload-socket\") pod \"e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:31.215213 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.215203 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b23f0def-02ec-422f-bc7e-b54897787bc1-workload-certs\") pod \"e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:31.215617 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.215225 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b23f0def-02ec-422f-bc7e-b54897787bc1-istio-data\") pod \"e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:31.215617 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.215544 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b23f0def-02ec-422f-bc7e-b54897787bc1-credential-socket\") pod \"e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:31.215721 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.215634 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b23f0def-02ec-422f-bc7e-b54897787bc1-workload-socket\") pod \"e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:31.215721 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.215664 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b23f0def-02ec-422f-bc7e-b54897787bc1-istio-data\") pod \"e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:31.215828 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.215796 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b23f0def-02ec-422f-bc7e-b54897787bc1-istiod-ca-cert\") pod \"e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:31.215987 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.215955 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b23f0def-02ec-422f-bc7e-b54897787bc1-workload-certs\") pod \"e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:31.217545 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.217528 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b23f0def-02ec-422f-bc7e-b54897787bc1-istio-podinfo\") pod \"e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:31.217784 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.217763 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b23f0def-02ec-422f-bc7e-b54897787bc1-istio-envoy\") pod \"e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:31.223819 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.223797 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b23f0def-02ec-422f-bc7e-b54897787bc1-istio-token\") pod \"e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:31.223946 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.223926 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sx78\" (UniqueName: \"kubernetes.io/projected/b23f0def-02ec-422f-bc7e-b54897787bc1-kube-api-access-8sx78\") pod \"e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:31.293709 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.293684 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:31.415542 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.415511 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:31.549088 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.549037 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts"] Jul 10 19:38:31.551426 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:38:31.551398 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb23f0def_02ec_422f_bc7e_b54897787bc1.slice/crio-ebb90a4f2478f1c934a3851b2dc2df8e88d68a1a03fe1ecb9a65c2b6c77c1fbd WatchSource:0}: Error finding container ebb90a4f2478f1c934a3851b2dc2df8e88d68a1a03fe1ecb9a65c2b6c77c1fbd: Status 404 returned error can't find the container with id ebb90a4f2478f1c934a3851b2dc2df8e88d68a1a03fe1ecb9a65c2b6c77c1fbd Jul 10 19:38:31.554050 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.554018 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:38:31.554144 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.554094 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:38:31.554144 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:31.554136 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:38:32.032524 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:32.032483 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" event={"ID":"b23f0def-02ec-422f-bc7e-b54897787bc1","Type":"ContainerStarted","Data":"e38a36d5f92db7c4e86709fda50620d5a1bf3fa11a9a61e8e3bf4307de97bf93"} Jul 10 19:38:32.032707 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:32.032580 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" event={"ID":"b23f0def-02ec-422f-bc7e-b54897787bc1","Type":"ContainerStarted","Data":"ebb90a4f2478f1c934a3851b2dc2df8e88d68a1a03fe1ecb9a65c2b6c77c1fbd"} Jul 10 19:38:32.057016 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:32.056965 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" podStartSLOduration=1.05694638 podStartE2EDuration="1.05694638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:38:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:38:32.055876294 +0000 UTC m=+2403.386017139" watchObservedRunningTime="2026-07-10 19:38:32.05694638 +0000 UTC m=+2403.387087188" Jul 10 19:38:32.379654 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:32.379566 2560 patch_prober.go:28] interesting pod/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 19:38:32.379654 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:32.379634 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" podUID="5054fc8a-c200-4de3-8a55-8f1f4b968044" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 19:38:32.416433 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:32.416408 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:32.420723 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:32.420702 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:33.038333 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:33.038293 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:33.039074 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:33.039050 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:34.070427 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:34.070387 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:34.141625 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:34.141574 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:34.143420 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:34.143394 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99"] Jul 10 19:38:34.143971 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:34.143953 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d98c4594-3a8e-423f-9c8d-62fc6fca2a8c" containerName="istio-proxy" Jul 10 19:38:34.146287 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:34.146267 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99" Jul 10 19:38:34.149277 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:34.149256 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-44lh8\"" Jul 10 19:38:34.149385 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:34.149288 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 19:38:34.149385 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:34.149323 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-7207cb-serving-cert\"" Jul 10 19:38:34.151143 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:34.151126 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 19:38:34.157061 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:34.157038 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99"] Jul 10 19:38:34.204395 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:34.204342 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:34.246214 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:34.246178 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99e7d171-8e0f-4f45-bc67-952aac3b32f7-maas-api-tls\") pod \"maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99\" (UID: \"99e7d171-8e0f-4f45-bc67-952aac3b32f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99" Jul 10 19:38:34.246371 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:34.246250 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hxsq\" (UniqueName: \"kubernetes.io/projected/99e7d171-8e0f-4f45-bc67-952aac3b32f7-kube-api-access-5hxsq\") pod \"maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99\" (UID: \"99e7d171-8e0f-4f45-bc67-952aac3b32f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99" Jul 10 19:38:34.246371 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:34.246320 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99e7d171-8e0f-4f45-bc67-952aac3b32f7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99\" (UID: \"99e7d171-8e0f-4f45-bc67-952aac3b32f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99" Jul 10 19:38:34.347161 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:34.347073 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99e7d171-8e0f-4f45-bc67-952aac3b32f7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99\" (UID: \"99e7d171-8e0f-4f45-bc67-952aac3b32f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99" Jul 10 19:38:34.347332 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:34.347201 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99e7d171-8e0f-4f45-bc67-952aac3b32f7-maas-api-tls\") pod \"maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99\" (UID: \"99e7d171-8e0f-4f45-bc67-952aac3b32f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99" Jul 10 19:38:34.347332 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:34.347252 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hxsq\" (UniqueName: \"kubernetes.io/projected/99e7d171-8e0f-4f45-bc67-952aac3b32f7-kube-api-access-5hxsq\") pod \"maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99\" (UID: \"99e7d171-8e0f-4f45-bc67-952aac3b32f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99" Jul 10 19:38:34.347483 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:38:34.347340 2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-7207cb-serving-cert: secret "maas-api-e2e-shared-a-7207cb-serving-cert" not found Jul 10 19:38:34.347483 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:38:34.347414 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e7d171-8e0f-4f45-bc67-952aac3b32f7-maas-api-tls podName:99e7d171-8e0f-4f45-bc67-952aac3b32f7 nodeName:}" failed. No retries permitted until 2026-07-10 19:38:34.84739722 +0000 UTC m=+2406.177538011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/99e7d171-8e0f-4f45-bc67-952aac3b32f7-maas-api-tls") pod "maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99" (UID: "99e7d171-8e0f-4f45-bc67-952aac3b32f7") : secret "maas-api-e2e-shared-a-7207cb-serving-cert" not found Jul 10 19:38:34.347753 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:34.347735 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99e7d171-8e0f-4f45-bc67-952aac3b32f7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99\" (UID: \"99e7d171-8e0f-4f45-bc67-952aac3b32f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99" Jul 10 19:38:34.356925 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:34.356904 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hxsq\" (UniqueName: \"kubernetes.io/projected/99e7d171-8e0f-4f45-bc67-952aac3b32f7-kube-api-access-5hxsq\") pod \"maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99\" (UID: \"99e7d171-8e0f-4f45-bc67-952aac3b32f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99" Jul 10 19:38:34.521574 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:34.521545 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:34.745027 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:34.744994 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:34.843654 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:34.843617 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:34.853192 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:34.853169 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99e7d171-8e0f-4f45-bc67-952aac3b32f7-maas-api-tls\") pod \"maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99\" (UID: \"99e7d171-8e0f-4f45-bc67-952aac3b32f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99" Jul 10 19:38:34.855476 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:34.855458 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99e7d171-8e0f-4f45-bc67-952aac3b32f7-maas-api-tls\") pod \"maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99\" (UID: \"99e7d171-8e0f-4f45-bc67-952aac3b32f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99" Jul 10 19:38:35.059932 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:35.059906 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99" Jul 10 19:38:35.238610 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:35.238587 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:38:35.358381 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:35.358306 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5054fc8a-c200-4de3-8a55-8f1f4b968044-istiod-ca-cert\") pod \"5054fc8a-c200-4de3-8a55-8f1f4b968044\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " Jul 10 19:38:35.358381 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:35.358360 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5054fc8a-c200-4de3-8a55-8f1f4b968044-istio-podinfo\") pod \"5054fc8a-c200-4de3-8a55-8f1f4b968044\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " Jul 10 19:38:35.358624 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:35.358398 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5054fc8a-c200-4de3-8a55-8f1f4b968044-workload-certs\") pod \"5054fc8a-c200-4de3-8a55-8f1f4b968044\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " Jul 10 19:38:35.358624 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:35.358428 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5054fc8a-c200-4de3-8a55-8f1f4b968044-istio-data\") pod \"5054fc8a-c200-4de3-8a55-8f1f4b968044\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " Jul 10 19:38:35.358624 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:35.358589 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pc4bb\" (UniqueName: \"kubernetes.io/projected/5054fc8a-c200-4de3-8a55-8f1f4b968044-kube-api-access-pc4bb\") pod \"5054fc8a-c200-4de3-8a55-8f1f4b968044\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " Jul 10 19:38:35.358785 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:35.358661 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5054fc8a-c200-4de3-8a55-8f1f4b968044-credential-socket\") pod \"5054fc8a-c200-4de3-8a55-8f1f4b968044\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " Jul 10 19:38:35.358785 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:35.358709 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5054fc8a-c200-4de3-8a55-8f1f4b968044-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5054fc8a-c200-4de3-8a55-8f1f4b968044" (UID: "5054fc8a-c200-4de3-8a55-8f1f4b968044"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:38:35.358785 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:35.358736 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5054fc8a-c200-4de3-8a55-8f1f4b968044-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5054fc8a-c200-4de3-8a55-8f1f4b968044" (UID: "5054fc8a-c200-4de3-8a55-8f1f4b968044"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:38:35.358785 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:35.358745 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5054fc8a-c200-4de3-8a55-8f1f4b968044-workload-socket\") pod \"5054fc8a-c200-4de3-8a55-8f1f4b968044\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " Jul 10 19:38:35.358981 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:35.358807 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5054fc8a-c200-4de3-8a55-8f1f4b968044-istio-envoy\") pod \"5054fc8a-c200-4de3-8a55-8f1f4b968044\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " Jul 10 19:38:35.358981 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:35.358846 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5054fc8a-c200-4de3-8a55-8f1f4b968044-istio-token\") pod \"5054fc8a-c200-4de3-8a55-8f1f4b968044\" (UID: \"5054fc8a-c200-4de3-8a55-8f1f4b968044\") " Jul 10 19:38:35.358981 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:35.358919 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5054fc8a-c200-4de3-8a55-8f1f4b968044-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5054fc8a-c200-4de3-8a55-8f1f4b968044" (UID: "5054fc8a-c200-4de3-8a55-8f1f4b968044"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:38:35.359119 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:35.359010 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5054fc8a-c200-4de3-8a55-8f1f4b968044-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5054fc8a-c200-4de3-8a55-8f1f4b968044" (UID: "5054fc8a-c200-4de3-8a55-8f1f4b968044"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:38:35.359375 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:35.359350 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5054fc8a-c200-4de3-8a55-8f1f4b968044-workload-certs\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:38:35.359485 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:35.359379 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5054fc8a-c200-4de3-8a55-8f1f4b968044-credential-socket\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:38:35.359485 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:35.359383 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5054fc8a-c200-4de3-8a55-8f1f4b968044-istio-data" (OuterVolumeSpecName: "istio-data") pod "5054fc8a-c200-4de3-8a55-8f1f4b968044" (UID: "5054fc8a-c200-4de3-8a55-8f1f4b968044"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:38:35.359485 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:35.359393 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5054fc8a-c200-4de3-8a55-8f1f4b968044-workload-socket\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:38:35.359485 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:35.359406 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5054fc8a-c200-4de3-8a55-8f1f4b968044-istiod-ca-cert\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:38:35.360838 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:35.360815 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5054fc8a-c200-4de3-8a55-8f1f4b968044-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5054fc8a-c200-4de3-8a55-8f1f4b968044" (UID: "5054fc8a-c200-4de3-8a55-8f1f4b968044"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 19:38:35.360934 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:35.360867 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5054fc8a-c200-4de3-8a55-8f1f4b968044-kube-api-access-pc4bb" (OuterVolumeSpecName: "kube-api-access-pc4bb") pod "5054fc8a-c200-4de3-8a55-8f1f4b968044" (UID: "5054fc8a-c200-4de3-8a55-8f1f4b968044"). InnerVolumeSpecName "kube-api-access-pc4bb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:38:35.361021 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:35.361007 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5054fc8a-c200-4de3-8a55-8f1f4b968044-istio-token" (OuterVolumeSpecName: "istio-token") pod "5054fc8a-c200-4de3-8a55-8f1f4b968044" (UID: "5054fc8a-c200-4de3-8a55-8f1f4b968044"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:38:35.361189 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:35.361174 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5054fc8a-c200-4de3-8a55-8f1f4b968044-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5054fc8a-c200-4de3-8a55-8f1f4b968044" (UID: "5054fc8a-c200-4de3-8a55-8f1f4b968044"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:38:35.406639 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:35.406615 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99"] Jul 10 19:38:35.408203 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:38:35.408179 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e7d171_8e0f_4f45_bc67_952aac3b32f7.slice/crio-702782781dee3f66fb60bd4ff6c45272377d6f5a703d4661a60f8c015e92b857 WatchSource:0}: Error finding container 702782781dee3f66fb60bd4ff6c45272377d6f5a703d4661a60f8c015e92b857: Status 404 returned error can't find the container with id 702782781dee3f66fb60bd4ff6c45272377d6f5a703d4661a60f8c015e92b857 Jul 10 19:38:35.460915 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:35.460896 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5054fc8a-c200-4de3-8a55-8f1f4b968044-istio-envoy\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:38:35.460915 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:35.460916 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5054fc8a-c200-4de3-8a55-8f1f4b968044-istio-token\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:38:35.461051 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:35.460928 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5054fc8a-c200-4de3-8a55-8f1f4b968044-istio-podinfo\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:38:35.461051 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:35.460939 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5054fc8a-c200-4de3-8a55-8f1f4b968044-istio-data\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:38:35.461051 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:35.460953 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc4bb\" (UniqueName: \"kubernetes.io/projected/5054fc8a-c200-4de3-8a55-8f1f4b968044-kube-api-access-pc4bb\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:38:36.054223 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:36.054189 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99" event={"ID":"99e7d171-8e0f-4f45-bc67-952aac3b32f7","Type":"ContainerStarted","Data":"60e24656a801aa4cf09b0acd6d9139d1004547bcd29f8f77a216c1b55d106f66"} Jul 10 19:38:36.054393 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:36.054242 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99" event={"ID":"99e7d171-8e0f-4f45-bc67-952aac3b32f7","Type":"ContainerStarted","Data":"702782781dee3f66fb60bd4ff6c45272377d6f5a703d4661a60f8c015e92b857"} Jul 10 19:38:36.054393 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:36.054270 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99" Jul 10 19:38:36.055765 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:36.055738 2560 generic.go:356] "Generic (PLEG): container finished" podID="5054fc8a-c200-4de3-8a55-8f1f4b968044" containerID="142f0d6cd291826c277f60c40a3abf55a8b2164cbcfd40e439b25e2dd1ab1e1e" exitCode=0 Jul 10 19:38:36.055888 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:36.055780 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" Jul 10 19:38:36.055888 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:36.055809 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" event={"ID":"5054fc8a-c200-4de3-8a55-8f1f4b968044","Type":"ContainerDied","Data":"142f0d6cd291826c277f60c40a3abf55a8b2164cbcfd40e439b25e2dd1ab1e1e"} Jul 10 19:38:36.055888 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:36.055851 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj" event={"ID":"5054fc8a-c200-4de3-8a55-8f1f4b968044","Type":"ContainerDied","Data":"27b847d38241defa68f86da6788ba732a199d4b8d99982cee6dac90fa8f701d5"} Jul 10 19:38:36.055888 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:36.055873 2560 scope.go:117] "RemoveContainer" containerID="142f0d6cd291826c277f60c40a3abf55a8b2164cbcfd40e439b25e2dd1ab1e1e" Jul 10 19:38:36.064530 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:36.064366 2560 scope.go:117] "RemoveContainer" containerID="7a2cace6de5c3159bdd49dbc35551c8842712730951efe9b2ab73e95a6f16204" Jul 10 19:38:36.072250 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:36.072233 2560 scope.go:117] "RemoveContainer" containerID="142f0d6cd291826c277f60c40a3abf55a8b2164cbcfd40e439b25e2dd1ab1e1e" Jul 10 19:38:36.072809 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:38:36.072786 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"142f0d6cd291826c277f60c40a3abf55a8b2164cbcfd40e439b25e2dd1ab1e1e\": container with ID starting with 142f0d6cd291826c277f60c40a3abf55a8b2164cbcfd40e439b25e2dd1ab1e1e not found: ID does not exist" containerID="142f0d6cd291826c277f60c40a3abf55a8b2164cbcfd40e439b25e2dd1ab1e1e" Jul 10 19:38:36.072915 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:36.072817 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"142f0d6cd291826c277f60c40a3abf55a8b2164cbcfd40e439b25e2dd1ab1e1e"} err="failed to get container status \"142f0d6cd291826c277f60c40a3abf55a8b2164cbcfd40e439b25e2dd1ab1e1e\": rpc error: code = NotFound desc = could not find container \"142f0d6cd291826c277f60c40a3abf55a8b2164cbcfd40e439b25e2dd1ab1e1e\": container with ID starting with 142f0d6cd291826c277f60c40a3abf55a8b2164cbcfd40e439b25e2dd1ab1e1e not found: ID does not exist" Jul 10 19:38:36.072915 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:36.072837 2560 scope.go:117] "RemoveContainer" containerID="7a2cace6de5c3159bdd49dbc35551c8842712730951efe9b2ab73e95a6f16204" Jul 10 19:38:36.073146 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:38:36.073116 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a2cace6de5c3159bdd49dbc35551c8842712730951efe9b2ab73e95a6f16204\": container with ID starting with 7a2cace6de5c3159bdd49dbc35551c8842712730951efe9b2ab73e95a6f16204 not found: ID does not exist" containerID="7a2cace6de5c3159bdd49dbc35551c8842712730951efe9b2ab73e95a6f16204" Jul 10 19:38:36.073197 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:36.073148 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a2cace6de5c3159bdd49dbc35551c8842712730951efe9b2ab73e95a6f16204"} err="failed to get container status \"7a2cace6de5c3159bdd49dbc35551c8842712730951efe9b2ab73e95a6f16204\": rpc error: code = NotFound desc = could not find container \"7a2cace6de5c3159bdd49dbc35551c8842712730951efe9b2ab73e95a6f16204\": container with ID starting with 7a2cace6de5c3159bdd49dbc35551c8842712730951efe9b2ab73e95a6f16204 not found: ID does not exist" Jul 10 19:38:36.073234 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:36.073197 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99" podStartSLOduration=1.623035027 podStartE2EDuration="2.073178395s" totalImagesPullingTime="450.143368ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:38:36.072229943 +0000 UTC m=+2407.402370779" watchObservedRunningTime="2026-07-10 19:38:36.073178395 +0000 UTC m=+2407.403319203" Jul 10 19:38:36.089250 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:36.089231 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj"] Jul 10 19:38:36.094938 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:36.094918 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e78db6-openshift-default-564c5784fc-755dj"] Jul 10 19:38:37.252183 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:37.252150 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5054fc8a-c200-4de3-8a55-8f1f4b968044" path="/var/lib/kubelet/pods/5054fc8a-c200-4de3-8a55-8f1f4b968044/volumes" Jul 10 19:38:38.687631 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:38.687590 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:39.049049 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.048961 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:39.431110 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.431078 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9"] Jul 10 19:38:39.431716 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.431693 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5054fc8a-c200-4de3-8a55-8f1f4b968044" containerName="istio-proxy" Jul 10 19:38:39.431716 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.431714 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5054fc8a-c200-4de3-8a55-8f1f4b968044" containerName="istio-proxy" Jul 10 19:38:39.431895 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.431737 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5054fc8a-c200-4de3-8a55-8f1f4b968044" containerName="istio-proxy" Jul 10 19:38:39.431895 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.431746 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5054fc8a-c200-4de3-8a55-8f1f4b968044" containerName="istio-proxy" Jul 10 19:38:39.431895 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.431834 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5054fc8a-c200-4de3-8a55-8f1f4b968044" containerName="istio-proxy" Jul 10 19:38:39.431895 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.431846 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5054fc8a-c200-4de3-8a55-8f1f4b968044" containerName="istio-proxy" Jul 10 19:38:39.435005 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.434986 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:38:39.438337 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.438319 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-3ec3c4-openshift-default-dockercfg-gdl4z\"" Jul 10 19:38:39.447558 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.447536 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9"] Jul 10 19:38:39.468957 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.468926 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:39.496031 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.495996 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf5ad96-432f-458c-973b-51875955b64e-istio-podinfo\") pod \"e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:38:39.496153 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.496047 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8xxm\" (UniqueName: \"kubernetes.io/projected/3bf5ad96-432f-458c-973b-51875955b64e-kube-api-access-k8xxm\") pod \"e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:38:39.496202 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.496179 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf5ad96-432f-458c-973b-51875955b64e-istio-envoy\") pod \"e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:38:39.496255 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.496223 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf5ad96-432f-458c-973b-51875955b64e-workload-socket\") pod \"e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:38:39.496291 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.496250 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf5ad96-432f-458c-973b-51875955b64e-istio-data\") pod \"e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:38:39.496291 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.496279 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bf5ad96-432f-458c-973b-51875955b64e-istio-token\") pod \"e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:38:39.496381 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.496310 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf5ad96-432f-458c-973b-51875955b64e-istiod-ca-cert\") pod \"e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:38:39.496492 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.496457 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf5ad96-432f-458c-973b-51875955b64e-credential-socket\") pod \"e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:38:39.496592 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.496513 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf5ad96-432f-458c-973b-51875955b64e-workload-certs\") pod \"e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:38:39.597959 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.597924 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf5ad96-432f-458c-973b-51875955b64e-workload-socket\") pod \"e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:38:39.598132 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.597964 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf5ad96-432f-458c-973b-51875955b64e-istio-data\") pod \"e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:38:39.598132 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.597991 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bf5ad96-432f-458c-973b-51875955b64e-istio-token\") pod \"e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:38:39.598132 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.598052 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf5ad96-432f-458c-973b-51875955b64e-istiod-ca-cert\") pod \"e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:38:39.598316 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.598280 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf5ad96-432f-458c-973b-51875955b64e-credential-socket\") pod \"e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:38:39.598427 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.598402 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf5ad96-432f-458c-973b-51875955b64e-workload-socket\") pod \"e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:38:39.598522 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.598431 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf5ad96-432f-458c-973b-51875955b64e-istio-data\") pod \"e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:38:39.598566 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.598537 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf5ad96-432f-458c-973b-51875955b64e-workload-certs\") pod \"e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:38:39.598600 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.598568 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf5ad96-432f-458c-973b-51875955b64e-istio-podinfo\") pod \"e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:38:39.598646 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.598606 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8xxm\" (UniqueName: \"kubernetes.io/projected/3bf5ad96-432f-458c-973b-51875955b64e-kube-api-access-k8xxm\") pod \"e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:38:39.598646 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.598614 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf5ad96-432f-458c-973b-51875955b64e-credential-socket\") pod \"e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:38:39.598746 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.598695 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf5ad96-432f-458c-973b-51875955b64e-istio-envoy\") pod \"e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:38:39.598921 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.598893 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf5ad96-432f-458c-973b-51875955b64e-workload-certs\") pod \"e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:38:39.599211 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.599182 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf5ad96-432f-458c-973b-51875955b64e-istiod-ca-cert\") pod \"e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:38:39.600972 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.600944 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf5ad96-432f-458c-973b-51875955b64e-istio-podinfo\") pod \"e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:38:39.601205 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.601175 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf5ad96-432f-458c-973b-51875955b64e-istio-envoy\") pod \"e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:38:39.606864 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.606838 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bf5ad96-432f-458c-973b-51875955b64e-istio-token\") pod \"e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:38:39.607323 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.607301 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8xxm\" (UniqueName: \"kubernetes.io/projected/3bf5ad96-432f-458c-973b-51875955b64e-kube-api-access-k8xxm\") pod \"e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:38:39.734142 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.734042 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:39.745006 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.744974 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:38:39.913179 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.913155 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9"] Jul 10 19:38:39.914801 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:38:39.914770 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf5ad96_432f_458c_973b_51875955b64e.slice/crio-99b423c310720207e2cdf84b2385b5a78853fa678ac3b7bb67360f8e2c7c2a64 WatchSource:0}: Error finding container 99b423c310720207e2cdf84b2385b5a78853fa678ac3b7bb67360f8e2c7c2a64: Status 404 returned error can't find the container with id 99b423c310720207e2cdf84b2385b5a78853fa678ac3b7bb67360f8e2c7c2a64 Jul 10 19:38:39.916974 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.916943 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:38:39.917031 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.917010 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:38:39.917066 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:39.917038 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:38:40.078255 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:40.078224 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" event={"ID":"3bf5ad96-432f-458c-973b-51875955b64e","Type":"ContainerStarted","Data":"62e1e54b1bb387d1dbeea93b4df9fb03fcce54ce1fd5ea5c6856957bc63f39ac"} Jul 10 19:38:40.078434 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:40.078279 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" event={"ID":"3bf5ad96-432f-458c-973b-51875955b64e","Type":"ContainerStarted","Data":"99b423c310720207e2cdf84b2385b5a78853fa678ac3b7bb67360f8e2c7c2a64"} Jul 10 19:38:40.100914 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:40.100852 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" podStartSLOduration=1.100836569 podStartE2EDuration="1.100836569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:38:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:38:40.099577569 +0000 UTC m=+2411.429718381" watchObservedRunningTime="2026-07-10 19:38:40.100836569 +0000 UTC m=+2411.430977377" Jul 10 19:38:40.745497 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:40.745454 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:38:40.750677 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:40.750654 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:38:41.083786 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:41.083710 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:38:41.084437 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:41.084416 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:38:42.066454 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:42.066406 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99" Jul 10 19:38:42.147462 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:42.137794 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:42.232937 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:42.232901 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:42.234531 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:42.234508 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z"] Jul 10 19:38:42.256666 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:42.256635 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z"] Jul 10 19:38:42.256797 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:42.256763 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z" Jul 10 19:38:42.259485 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:42.259463 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-3ec3c4-serving-cert\"" Jul 10 19:38:42.307808 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:42.307777 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:42.335310 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:42.335224 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/540ace4f-2fd3-4489-92d1-a50dfdbbeb50-openshift-service-ca\") pod \"maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z\" (UID: \"540ace4f-2fd3-4489-92d1-a50dfdbbeb50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z" Jul 10 19:38:42.335601 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:42.335579 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gqmv\" (UniqueName: \"kubernetes.io/projected/540ace4f-2fd3-4489-92d1-a50dfdbbeb50-kube-api-access-8gqmv\") pod \"maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z\" (UID: \"540ace4f-2fd3-4489-92d1-a50dfdbbeb50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z" Jul 10 19:38:42.335814 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:42.335796 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/540ace4f-2fd3-4489-92d1-a50dfdbbeb50-maas-api-tls\") pod \"maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z\" (UID: \"540ace4f-2fd3-4489-92d1-a50dfdbbeb50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z" Jul 10 19:38:42.437255 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:42.437219 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/540ace4f-2fd3-4489-92d1-a50dfdbbeb50-openshift-service-ca\") pod \"maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z\" (UID: \"540ace4f-2fd3-4489-92d1-a50dfdbbeb50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z" Jul 10 19:38:42.437539 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:42.437516 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gqmv\" (UniqueName: \"kubernetes.io/projected/540ace4f-2fd3-4489-92d1-a50dfdbbeb50-kube-api-access-8gqmv\") pod \"maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z\" (UID: \"540ace4f-2fd3-4489-92d1-a50dfdbbeb50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z" Jul 10 19:38:42.437712 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:42.437695 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/540ace4f-2fd3-4489-92d1-a50dfdbbeb50-maas-api-tls\") pod \"maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z\" (UID: \"540ace4f-2fd3-4489-92d1-a50dfdbbeb50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z" Jul 10 19:38:42.438107 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:38:42.437894 2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-3ec3c4-serving-cert: secret "maas-api-e2e-shared-b-3ec3c4-serving-cert" not found Jul 10 19:38:42.438107 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:38:42.437962 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/540ace4f-2fd3-4489-92d1-a50dfdbbeb50-maas-api-tls podName:540ace4f-2fd3-4489-92d1-a50dfdbbeb50 nodeName:}" failed. No retries permitted until 2026-07-10 19:38:42.937941003 +0000 UTC m=+2414.268081804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/540ace4f-2fd3-4489-92d1-a50dfdbbeb50-maas-api-tls") pod "maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z" (UID: "540ace4f-2fd3-4489-92d1-a50dfdbbeb50") : secret "maas-api-e2e-shared-b-3ec3c4-serving-cert" not found Jul 10 19:38:42.439002 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:42.438966 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/540ace4f-2fd3-4489-92d1-a50dfdbbeb50-openshift-service-ca\") pod \"maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z\" (UID: \"540ace4f-2fd3-4489-92d1-a50dfdbbeb50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z" Jul 10 19:38:42.448067 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:42.448044 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gqmv\" (UniqueName: \"kubernetes.io/projected/540ace4f-2fd3-4489-92d1-a50dfdbbeb50-kube-api-access-8gqmv\") pod \"maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z\" (UID: \"540ace4f-2fd3-4489-92d1-a50dfdbbeb50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z" Jul 10 19:38:42.711052 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:42.711021 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:42.810557 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:42.810513 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:42.910629 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:42.910590 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:42.944067 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:42.944028 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/540ace4f-2fd3-4489-92d1-a50dfdbbeb50-maas-api-tls\") pod \"maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z\" (UID: \"540ace4f-2fd3-4489-92d1-a50dfdbbeb50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z" Jul 10 19:38:42.947327 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:42.947293 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/540ace4f-2fd3-4489-92d1-a50dfdbbeb50-maas-api-tls\") pod \"maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z\" (UID: \"540ace4f-2fd3-4489-92d1-a50dfdbbeb50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z" Jul 10 19:38:43.168000 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:43.167967 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z" Jul 10 19:38:43.338478 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:43.331868 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z"] Jul 10 19:38:43.350491 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:38:43.350452 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod540ace4f_2fd3_4489_92d1_a50dfdbbeb50.slice/crio-a3b95e34a93d115779fc31485e3505a9241f33f07debdb0688119fe062e29ff4 WatchSource:0}: Error finding container a3b95e34a93d115779fc31485e3505a9241f33f07debdb0688119fe062e29ff4: Status 404 returned error can't find the container with id a3b95e34a93d115779fc31485e3505a9241f33f07debdb0688119fe062e29ff4 Jul 10 19:38:44.104330 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:44.104239 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z" event={"ID":"540ace4f-2fd3-4489-92d1-a50dfdbbeb50","Type":"ContainerStarted","Data":"51723bb66c4d2a71425e4d13aabf13f11b1b8737a5edc2e2bf5c42d7dd5ae31c"} Jul 10 19:38:44.104330 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:44.104289 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z" event={"ID":"540ace4f-2fd3-4489-92d1-a50dfdbbeb50","Type":"ContainerStarted","Data":"a3b95e34a93d115779fc31485e3505a9241f33f07debdb0688119fe062e29ff4"} Jul 10 19:38:44.104330 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:44.104318 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z" Jul 10 19:38:44.122686 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:44.122633 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z" podStartSLOduration=1.7306032230000001 podStartE2EDuration="2.122618096s" totalImagesPullingTime="392.014873ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:38:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:38:44.12225118 +0000 UTC m=+2415.452391990" watchObservedRunningTime="2026-07-10 19:38:44.122618096 +0000 UTC m=+2415.452758959" Jul 10 19:38:46.741502 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:46.741463 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:47.123239 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:47.123157 2560 generic.go:356] "Generic (PLEG): container finished" podID="b23f0def-02ec-422f-bc7e-b54897787bc1" containerID="e38a36d5f92db7c4e86709fda50620d5a1bf3fa11a9a61e8e3bf4307de97bf93" exitCode=0 Jul 10 19:38:47.123239 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:47.123205 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" event={"ID":"b23f0def-02ec-422f-bc7e-b54897787bc1","Type":"ContainerDied","Data":"e38a36d5f92db7c4e86709fda50620d5a1bf3fa11a9a61e8e3bf4307de97bf93"} Jul 10 19:38:47.123630 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:47.123607 2560 scope.go:117] "RemoveContainer" containerID="e38a36d5f92db7c4e86709fda50620d5a1bf3fa11a9a61e8e3bf4307de97bf93" Jul 10 19:38:47.124579 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:47.124550 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:38:47.124648 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:47.124618 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:38:47.124690 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:47.124647 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:38:48.038777 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:48.038733 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:48.134626 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:48.134587 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" event={"ID":"b23f0def-02ec-422f-bc7e-b54897787bc1","Type":"ContainerStarted","Data":"8bc4b2c8b194a3759275d9ac1c0c94e0b203d24a37760aa1e725e8e37107080b"} Jul 10 19:38:48.416194 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:48.416159 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:48.417372 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:48.417348 2560 patch_prober.go:28] interesting pod/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 10 19:38:48.417519 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:48.417394 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" podUID="b23f0def-02ec-422f-bc7e-b54897787bc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 10 19:38:49.416889 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:49.416853 2560 patch_prober.go:28] interesting pod/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 10 19:38:49.417284 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:49.416908 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" podUID="b23f0def-02ec-422f-bc7e-b54897787bc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 10 19:38:50.118022 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:50.117993 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z" Jul 10 19:38:50.416736 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:50.416702 2560 patch_prober.go:28] interesting pod/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 10 19:38:50.416891 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:50.416759 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" podUID="b23f0def-02ec-422f-bc7e-b54897787bc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 10 19:38:51.416603 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:51.416524 2560 patch_prober.go:28] interesting pod/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 10 19:38:51.416603 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:51.416581 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" podUID="b23f0def-02ec-422f-bc7e-b54897787bc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 10 19:38:52.427561 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:52.427528 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:52.428620 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:52.428600 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:52.432058 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:52.432039 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:38:55.467663 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:55.467628 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:55.568127 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:55.568074 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:56.917893 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:56.917859 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:57.124578 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:57.124544 2560 patch_prober.go:28] interesting pod/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 19:38:57.124744 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:57.124591 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" podUID="3bf5ad96-432f-458c-973b-51875955b64e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 19:38:58.197192 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:58.197158 2560 generic.go:356] "Generic (PLEG): container finished" podID="3bf5ad96-432f-458c-973b-51875955b64e" containerID="62e1e54b1bb387d1dbeea93b4df9fb03fcce54ce1fd5ea5c6856957bc63f39ac" exitCode=0 Jul 10 19:38:58.197630 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:58.197218 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" event={"ID":"3bf5ad96-432f-458c-973b-51875955b64e","Type":"ContainerDied","Data":"62e1e54b1bb387d1dbeea93b4df9fb03fcce54ce1fd5ea5c6856957bc63f39ac"} Jul 10 19:38:58.197630 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:58.197596 2560 scope.go:117] "RemoveContainer" containerID="62e1e54b1bb387d1dbeea93b4df9fb03fcce54ce1fd5ea5c6856957bc63f39ac" Jul 10 19:38:58.198538 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:58.198510 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:38:58.198607 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:58.198576 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:38:58.198646 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:58.198606 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 10 19:38:59.204243 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:59.204210 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" event={"ID":"3bf5ad96-432f-458c-973b-51875955b64e","Type":"ContainerStarted","Data":"bc14891d46bf60ed7c0b903b518c0b4cf6dd95a493b0d9cdd06e2394d0849604"} Jul 10 19:38:59.745505 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:59.745469 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:38:59.745676 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:59.745653 2560 patch_prober.go:28] interesting pod/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 10 19:38:59.745724 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:38:59.745696 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" podUID="3bf5ad96-432f-458c-973b-51875955b64e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 10 19:39:00.745659 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:39:00.745619 2560 patch_prober.go:28] interesting pod/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 10 19:39:00.746107 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:39:00.745678 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" podUID="3bf5ad96-432f-458c-973b-51875955b64e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 10 19:39:01.745782 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:39:01.745747 2560 patch_prober.go:28] interesting pod/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 10 19:39:01.746145 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:39:01.745813 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" podUID="3bf5ad96-432f-458c-973b-51875955b64e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 10 19:39:02.748839 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:39:02.748811 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:39:02.749298 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:39:02.749073 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:39:02.749731 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:39:02.749717 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:39:04.535768 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:39:04.535724 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:05.835298 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:39:05.835266 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:30.923877 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:39:30.923823 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:40.848782 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:39:40.848750 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:40:18.951825 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:40:18.951785 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:26.207181 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:41:26.207139 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:27.116468 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:41:27.116403 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:29.474407 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:41:29.474376 2560 scope.go:117] "RemoveContainer" containerID="4fe9e1fabdf85eb2a9b27e802241dfc6baa15e77639a2bb2c8b9df0c38c135b9" Jul 10 19:41:33.403802 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:41:33.403766 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:41:43.926535 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:41:43.926497 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:08.004779 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:42:08.004742 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:25.436141 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:42:25.436093 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:44.612362 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:42:44.612322 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:45.514154 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:42:45.514104 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:51.625188 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:42:51.625140 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:51.720798 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:42:51.720764 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:42:57.881873 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:42:57.881832 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:43:02.076928 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:43:02.076880 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:43:23.237312 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:43:23.237175 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:43:29.371750 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:43:29.371716 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l52t_2a22956a-4e96-41de-945c-a851b587a442/ovn-acl-logging/0.log" Jul 10 19:43:29.374333 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:43:29.374309 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l52t_2a22956a-4e96-41de-945c-a851b587a442/ovn-acl-logging/0.log" Jul 10 19:43:41.021395 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:43:41.021355 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:43:41.123806 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:43:41.123776 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:43:56.209002 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:43:56.208959 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:43:56.309394 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:43:56.309357 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:44:05.277392 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:05.277356 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:44:05.373663 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:05.373633 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:44:06.505529 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:06.505493 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:44:08.190627 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:08.190591 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:44:27.471173 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:27.471129 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:44:27.572016 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:27.571984 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:44:29.510827 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:29.510801 2560 scope.go:117] "RemoveContainer" containerID="e00a2b185b7bdeaf981d2ced5954cad2740e490e79d29f9e0892d28a4d09ebfd" Jul 10 19:44:37.353537 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:37.353506 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99"] Jul 10 19:44:37.354042 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:37.353757 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99" podUID="99e7d171-8e0f-4f45-bc67-952aac3b32f7" containerName="maas-api" containerID="cri-o://60e24656a801aa4cf09b0acd6d9139d1004547bcd29f8f77a216c1b55d106f66" gracePeriod=30 Jul 10 19:44:37.380870 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:37.380833 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:44:37.798171 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:37.798148 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99" Jul 10 19:44:37.890551 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:37.890467 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hxsq\" (UniqueName: \"kubernetes.io/projected/99e7d171-8e0f-4f45-bc67-952aac3b32f7-kube-api-access-5hxsq\") pod \"99e7d171-8e0f-4f45-bc67-952aac3b32f7\" (UID: \"99e7d171-8e0f-4f45-bc67-952aac3b32f7\") " Jul 10 19:44:37.890720 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:37.890558 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99e7d171-8e0f-4f45-bc67-952aac3b32f7-openshift-service-ca\") pod \"99e7d171-8e0f-4f45-bc67-952aac3b32f7\" (UID: \"99e7d171-8e0f-4f45-bc67-952aac3b32f7\") " Jul 10 19:44:37.890720 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:37.890643 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99e7d171-8e0f-4f45-bc67-952aac3b32f7-maas-api-tls\") pod \"99e7d171-8e0f-4f45-bc67-952aac3b32f7\" (UID: \"99e7d171-8e0f-4f45-bc67-952aac3b32f7\") " Jul 10 19:44:37.890904 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:37.890872 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99e7d171-8e0f-4f45-bc67-952aac3b32f7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "99e7d171-8e0f-4f45-bc67-952aac3b32f7" (UID: "99e7d171-8e0f-4f45-bc67-952aac3b32f7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:44:37.891049 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:37.891033 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99e7d171-8e0f-4f45-bc67-952aac3b32f7-openshift-service-ca\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:44:37.892699 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:37.892667 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99e7d171-8e0f-4f45-bc67-952aac3b32f7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "99e7d171-8e0f-4f45-bc67-952aac3b32f7" (UID: "99e7d171-8e0f-4f45-bc67-952aac3b32f7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:44:37.892699 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:37.892693 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99e7d171-8e0f-4f45-bc67-952aac3b32f7-kube-api-access-5hxsq" (OuterVolumeSpecName: "kube-api-access-5hxsq") pod "99e7d171-8e0f-4f45-bc67-952aac3b32f7" (UID: "99e7d171-8e0f-4f45-bc67-952aac3b32f7"). InnerVolumeSpecName "kube-api-access-5hxsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:44:37.937774 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:37.937740 2560 generic.go:356] "Generic (PLEG): container finished" podID="99e7d171-8e0f-4f45-bc67-952aac3b32f7" containerID="60e24656a801aa4cf09b0acd6d9139d1004547bcd29f8f77a216c1b55d106f66" exitCode=0 Jul 10 19:44:37.937924 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:37.937787 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99" Jul 10 19:44:37.937924 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:37.937818 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99" event={"ID":"99e7d171-8e0f-4f45-bc67-952aac3b32f7","Type":"ContainerDied","Data":"60e24656a801aa4cf09b0acd6d9139d1004547bcd29f8f77a216c1b55d106f66"} Jul 10 19:44:37.937924 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:37.937869 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99" event={"ID":"99e7d171-8e0f-4f45-bc67-952aac3b32f7","Type":"ContainerDied","Data":"702782781dee3f66fb60bd4ff6c45272377d6f5a703d4661a60f8c015e92b857"} Jul 10 19:44:37.937924 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:37.937891 2560 scope.go:117] "RemoveContainer" containerID="60e24656a801aa4cf09b0acd6d9139d1004547bcd29f8f77a216c1b55d106f66" Jul 10 19:44:37.947976 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:37.947961 2560 scope.go:117] "RemoveContainer" containerID="60e24656a801aa4cf09b0acd6d9139d1004547bcd29f8f77a216c1b55d106f66" Jul 10 19:44:37.948221 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:44:37.948204 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60e24656a801aa4cf09b0acd6d9139d1004547bcd29f8f77a216c1b55d106f66\": container with ID starting with 60e24656a801aa4cf09b0acd6d9139d1004547bcd29f8f77a216c1b55d106f66 not found: ID does not exist" containerID="60e24656a801aa4cf09b0acd6d9139d1004547bcd29f8f77a216c1b55d106f66" Jul 10 19:44:37.948273 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:37.948228 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60e24656a801aa4cf09b0acd6d9139d1004547bcd29f8f77a216c1b55d106f66"} err="failed to get container status \"60e24656a801aa4cf09b0acd6d9139d1004547bcd29f8f77a216c1b55d106f66\": rpc error: code = NotFound desc = could not find container \"60e24656a801aa4cf09b0acd6d9139d1004547bcd29f8f77a216c1b55d106f66\": container with ID starting with 60e24656a801aa4cf09b0acd6d9139d1004547bcd29f8f77a216c1b55d106f66 not found: ID does not exist" Jul 10 19:44:37.961177 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:37.961154 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99"] Jul 10 19:44:37.966920 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:37.966902 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7207cb-bd8d7b49b-b5d99"] Jul 10 19:44:37.992287 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:37.992266 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99e7d171-8e0f-4f45-bc67-952aac3b32f7-maas-api-tls\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:44:37.992394 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:37.992291 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hxsq\" (UniqueName: \"kubernetes.io/projected/99e7d171-8e0f-4f45-bc67-952aac3b32f7-kube-api-access-5hxsq\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:44:38.417110 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:38.417080 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:44:39.252937 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:39.252901 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99e7d171-8e0f-4f45-bc67-952aac3b32f7" path="/var/lib/kubelet/pods/99e7d171-8e0f-4f45-bc67-952aac3b32f7/volumes" Jul 10 19:44:42.053244 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:42.053210 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:44:48.999540 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:48.999494 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:44:49.113407 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:49.113378 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:44:52.698732 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:52.698652 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts"] Jul 10 19:44:52.699077 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:52.698898 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" podUID="b23f0def-02ec-422f-bc7e-b54897787bc1" containerName="istio-proxy" containerID="cri-o://8bc4b2c8b194a3759275d9ac1c0c94e0b203d24a37760aa1e725e8e37107080b" gracePeriod=30 Jul 10 19:44:52.744008 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:52.743974 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:44:52.937677 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:52.937638 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:44:53.737578 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:53.737539 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:44:53.880553 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:53.880501 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:44:57.952257 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:57.952233 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:44:57.987035 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:57.986972 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b23f0def-02ec-422f-bc7e-b54897787bc1-istio-data\") pod \"b23f0def-02ec-422f-bc7e-b54897787bc1\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " Jul 10 19:44:57.987035 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:57.987021 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b23f0def-02ec-422f-bc7e-b54897787bc1-istio-token\") pod \"b23f0def-02ec-422f-bc7e-b54897787bc1\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " Jul 10 19:44:57.987182 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:57.987068 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b23f0def-02ec-422f-bc7e-b54897787bc1-workload-socket\") pod \"b23f0def-02ec-422f-bc7e-b54897787bc1\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " Jul 10 19:44:57.987182 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:57.987138 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b23f0def-02ec-422f-bc7e-b54897787bc1-istio-podinfo\") pod \"b23f0def-02ec-422f-bc7e-b54897787bc1\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " Jul 10 19:44:57.987182 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:57.987171 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sx78\" (UniqueName: \"kubernetes.io/projected/b23f0def-02ec-422f-bc7e-b54897787bc1-kube-api-access-8sx78\") pod \"b23f0def-02ec-422f-bc7e-b54897787bc1\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " Jul 10 19:44:57.987319 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:57.987200 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b23f0def-02ec-422f-bc7e-b54897787bc1-workload-certs\") pod \"b23f0def-02ec-422f-bc7e-b54897787bc1\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " Jul 10 19:44:57.987319 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:57.987267 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b23f0def-02ec-422f-bc7e-b54897787bc1-credential-socket\") pod \"b23f0def-02ec-422f-bc7e-b54897787bc1\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " Jul 10 19:44:57.987319 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:57.987314 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b23f0def-02ec-422f-bc7e-b54897787bc1-istio-envoy\") pod \"b23f0def-02ec-422f-bc7e-b54897787bc1\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " Jul 10 19:44:57.987480 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:57.987326 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b23f0def-02ec-422f-bc7e-b54897787bc1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b23f0def-02ec-422f-bc7e-b54897787bc1" (UID: "b23f0def-02ec-422f-bc7e-b54897787bc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:44:57.987480 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:57.987367 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b23f0def-02ec-422f-bc7e-b54897787bc1-istiod-ca-cert\") pod \"b23f0def-02ec-422f-bc7e-b54897787bc1\" (UID: \"b23f0def-02ec-422f-bc7e-b54897787bc1\") " Jul 10 19:44:57.987608 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:57.987539 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b23f0def-02ec-422f-bc7e-b54897787bc1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b23f0def-02ec-422f-bc7e-b54897787bc1" (UID: "b23f0def-02ec-422f-bc7e-b54897787bc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:44:57.987741 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:57.987722 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b23f0def-02ec-422f-bc7e-b54897787bc1-credential-socket\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:44:57.987810 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:57.987747 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b23f0def-02ec-422f-bc7e-b54897787bc1-workload-socket\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:44:57.987810 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:57.987744 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b23f0def-02ec-422f-bc7e-b54897787bc1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b23f0def-02ec-422f-bc7e-b54897787bc1" (UID: "b23f0def-02ec-422f-bc7e-b54897787bc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:44:57.987810 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:57.987783 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b23f0def-02ec-422f-bc7e-b54897787bc1-istio-data" (OuterVolumeSpecName: "istio-data") pod "b23f0def-02ec-422f-bc7e-b54897787bc1" (UID: "b23f0def-02ec-422f-bc7e-b54897787bc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:44:57.987967 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:57.987822 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b23f0def-02ec-422f-bc7e-b54897787bc1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b23f0def-02ec-422f-bc7e-b54897787bc1" (UID: "b23f0def-02ec-422f-bc7e-b54897787bc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:44:57.989496 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:57.989458 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b23f0def-02ec-422f-bc7e-b54897787bc1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b23f0def-02ec-422f-bc7e-b54897787bc1" (UID: "b23f0def-02ec-422f-bc7e-b54897787bc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:44:57.989648 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:57.989627 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b23f0def-02ec-422f-bc7e-b54897787bc1-kube-api-access-8sx78" (OuterVolumeSpecName: "kube-api-access-8sx78") pod "b23f0def-02ec-422f-bc7e-b54897787bc1" (UID: "b23f0def-02ec-422f-bc7e-b54897787bc1"). InnerVolumeSpecName "kube-api-access-8sx78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:44:57.989711 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:57.989662 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b23f0def-02ec-422f-bc7e-b54897787bc1-istio-token" (OuterVolumeSpecName: "istio-token") pod "b23f0def-02ec-422f-bc7e-b54897787bc1" (UID: "b23f0def-02ec-422f-bc7e-b54897787bc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:44:57.989711 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:57.989658 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b23f0def-02ec-422f-bc7e-b54897787bc1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b23f0def-02ec-422f-bc7e-b54897787bc1" (UID: "b23f0def-02ec-422f-bc7e-b54897787bc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 19:44:58.038780 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:58.038752 2560 generic.go:356] "Generic (PLEG): container finished" podID="b23f0def-02ec-422f-bc7e-b54897787bc1" containerID="8bc4b2c8b194a3759275d9ac1c0c94e0b203d24a37760aa1e725e8e37107080b" exitCode=0 Jul 10 19:44:58.038881 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:58.038853 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" Jul 10 19:44:58.038924 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:58.038883 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" event={"ID":"b23f0def-02ec-422f-bc7e-b54897787bc1","Type":"ContainerDied","Data":"8bc4b2c8b194a3759275d9ac1c0c94e0b203d24a37760aa1e725e8e37107080b"} Jul 10 19:44:58.038956 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:58.038947 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts" event={"ID":"b23f0def-02ec-422f-bc7e-b54897787bc1","Type":"ContainerDied","Data":"ebb90a4f2478f1c934a3851b2dc2df8e88d68a1a03fe1ecb9a65c2b6c77c1fbd"} Jul 10 19:44:58.038992 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:58.038969 2560 scope.go:117] "RemoveContainer" containerID="8bc4b2c8b194a3759275d9ac1c0c94e0b203d24a37760aa1e725e8e37107080b" Jul 10 19:44:58.047883 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:58.047868 2560 scope.go:117] "RemoveContainer" containerID="e38a36d5f92db7c4e86709fda50620d5a1bf3fa11a9a61e8e3bf4307de97bf93" Jul 10 19:44:58.055420 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:58.055403 2560 scope.go:117] "RemoveContainer" containerID="8bc4b2c8b194a3759275d9ac1c0c94e0b203d24a37760aa1e725e8e37107080b" Jul 10 19:44:58.055671 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:44:58.055653 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bc4b2c8b194a3759275d9ac1c0c94e0b203d24a37760aa1e725e8e37107080b\": container with ID starting with 8bc4b2c8b194a3759275d9ac1c0c94e0b203d24a37760aa1e725e8e37107080b not found: ID does not exist" containerID="8bc4b2c8b194a3759275d9ac1c0c94e0b203d24a37760aa1e725e8e37107080b" Jul 10 19:44:58.055718 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:58.055677 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bc4b2c8b194a3759275d9ac1c0c94e0b203d24a37760aa1e725e8e37107080b"} err="failed to get container status \"8bc4b2c8b194a3759275d9ac1c0c94e0b203d24a37760aa1e725e8e37107080b\": rpc error: code = NotFound desc = could not find container \"8bc4b2c8b194a3759275d9ac1c0c94e0b203d24a37760aa1e725e8e37107080b\": container with ID starting with 8bc4b2c8b194a3759275d9ac1c0c94e0b203d24a37760aa1e725e8e37107080b not found: ID does not exist" Jul 10 19:44:58.055718 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:58.055692 2560 scope.go:117] "RemoveContainer" containerID="e38a36d5f92db7c4e86709fda50620d5a1bf3fa11a9a61e8e3bf4307de97bf93" Jul 10 19:44:58.055894 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:44:58.055878 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e38a36d5f92db7c4e86709fda50620d5a1bf3fa11a9a61e8e3bf4307de97bf93\": container with ID starting with e38a36d5f92db7c4e86709fda50620d5a1bf3fa11a9a61e8e3bf4307de97bf93 not found: ID does not exist" containerID="e38a36d5f92db7c4e86709fda50620d5a1bf3fa11a9a61e8e3bf4307de97bf93" Jul 10 19:44:58.055932 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:58.055900 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e38a36d5f92db7c4e86709fda50620d5a1bf3fa11a9a61e8e3bf4307de97bf93"} err="failed to get container status \"e38a36d5f92db7c4e86709fda50620d5a1bf3fa11a9a61e8e3bf4307de97bf93\": rpc error: code = NotFound desc = could not find container \"e38a36d5f92db7c4e86709fda50620d5a1bf3fa11a9a61e8e3bf4307de97bf93\": container with ID starting with e38a36d5f92db7c4e86709fda50620d5a1bf3fa11a9a61e8e3bf4307de97bf93 not found: ID does not exist" Jul 10 19:44:58.061546 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:58.061524 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts"] Jul 10 19:44:58.065112 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:58.065092 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-7207cb-openshift-default-6864c8579-rxkts"] Jul 10 19:44:58.088573 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:58.088555 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b23f0def-02ec-422f-bc7e-b54897787bc1-istio-envoy\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:44:58.088650 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:58.088575 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b23f0def-02ec-422f-bc7e-b54897787bc1-istiod-ca-cert\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:44:58.088650 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:58.088588 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b23f0def-02ec-422f-bc7e-b54897787bc1-istio-data\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:44:58.088650 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:58.088598 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b23f0def-02ec-422f-bc7e-b54897787bc1-istio-token\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:44:58.088650 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:58.088605 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b23f0def-02ec-422f-bc7e-b54897787bc1-istio-podinfo\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:44:58.088650 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:58.088613 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sx78\" (UniqueName: \"kubernetes.io/projected/b23f0def-02ec-422f-bc7e-b54897787bc1-kube-api-access-8sx78\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:44:58.088650 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:58.088624 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b23f0def-02ec-422f-bc7e-b54897787bc1-workload-certs\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:44:59.252928 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:44:59.252900 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b23f0def-02ec-422f-bc7e-b54897787bc1" path="/var/lib/kubelet/pods/b23f0def-02ec-422f-bc7e-b54897787bc1/volumes" Jul 10 19:45:00.146239 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:00.146210 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728545-fpmw8"] Jul 10 19:45:00.146601 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:00.146590 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b23f0def-02ec-422f-bc7e-b54897787bc1" containerName="istio-proxy" Jul 10 19:45:00.146656 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:00.146603 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b23f0def-02ec-422f-bc7e-b54897787bc1" containerName="istio-proxy" Jul 10 19:45:00.146656 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:00.146617 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99e7d171-8e0f-4f45-bc67-952aac3b32f7" containerName="maas-api" Jul 10 19:45:00.146656 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:00.146622 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="99e7d171-8e0f-4f45-bc67-952aac3b32f7" containerName="maas-api" Jul 10 19:45:00.146744 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:00.146694 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b23f0def-02ec-422f-bc7e-b54897787bc1" containerName="istio-proxy" Jul 10 19:45:00.146744 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:00.146702 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b23f0def-02ec-422f-bc7e-b54897787bc1" containerName="istio-proxy" Jul 10 19:45:00.146744 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:00.146708 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="99e7d171-8e0f-4f45-bc67-952aac3b32f7" containerName="maas-api" Jul 10 19:45:00.148603 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:00.148587 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728545-fpmw8" Jul 10 19:45:00.151642 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:00.151622 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fsnnw\"" Jul 10 19:45:00.157988 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:00.157974 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728545-fpmw8" Jul 10 19:45:00.163092 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:00.163068 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728545-fpmw8"] Jul 10 19:45:00.288142 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:00.288117 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728545-fpmw8"] Jul 10 19:45:00.290387 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:45:00.290350 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f414efb_542b_4a59_813c_2dd0f09db2e1.slice/crio-f1235beae7778f9d1cc6964d5d40f0bc6b06bdaf9e471fdae5ee52dc96c8eda6 WatchSource:0}: Error finding container f1235beae7778f9d1cc6964d5d40f0bc6b06bdaf9e471fdae5ee52dc96c8eda6: Status 404 returned error can't find the container with id f1235beae7778f9d1cc6964d5d40f0bc6b06bdaf9e471fdae5ee52dc96c8eda6 Jul 10 19:45:01.056271 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:01.056236 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728545-fpmw8" event={"ID":"0f414efb-542b-4a59-813c-2dd0f09db2e1","Type":"ContainerStarted","Data":"c66b9f64e7a7cd58ca199cf16776525607485374903cb653de6471933f5864b6"} Jul 10 19:45:01.056463 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:01.056289 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728545-fpmw8" event={"ID":"0f414efb-542b-4a59-813c-2dd0f09db2e1","Type":"ContainerStarted","Data":"f1235beae7778f9d1cc6964d5d40f0bc6b06bdaf9e471fdae5ee52dc96c8eda6"} Jul 10 19:45:01.074761 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:01.074714 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728545-fpmw8" podStartSLOduration=1.074701022 podStartE2EDuration="1.074701022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:45:01.072179066 +0000 UTC m=+2792.402319876" watchObservedRunningTime="2026-07-10 19:45:01.074701022 +0000 UTC m=+2792.404841831" Jul 10 19:45:02.062010 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:02.061972 2560 generic.go:356] "Generic (PLEG): container finished" podID="0f414efb-542b-4a59-813c-2dd0f09db2e1" containerID="c66b9f64e7a7cd58ca199cf16776525607485374903cb653de6471933f5864b6" exitCode=0 Jul 10 19:45:02.062348 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:02.062057 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728545-fpmw8" event={"ID":"0f414efb-542b-4a59-813c-2dd0f09db2e1","Type":"ContainerDied","Data":"c66b9f64e7a7cd58ca199cf16776525607485374903cb653de6471933f5864b6"} Jul 10 19:45:03.197923 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:03.197900 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728545-fpmw8" Jul 10 19:45:03.489006 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:03.488976 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z"] Jul 10 19:45:03.489186 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:03.489163 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z" podUID="540ace4f-2fd3-4489-92d1-a50dfdbbeb50" containerName="maas-api" containerID="cri-o://51723bb66c4d2a71425e4d13aabf13f11b1b8737a5edc2e2bf5c42d7dd5ae31c" gracePeriod=30 Jul 10 19:45:03.544602 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:03.544578 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:45:03.734678 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:03.734655 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z" Jul 10 19:45:03.838330 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:03.838273 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/540ace4f-2fd3-4489-92d1-a50dfdbbeb50-openshift-service-ca\") pod \"540ace4f-2fd3-4489-92d1-a50dfdbbeb50\" (UID: \"540ace4f-2fd3-4489-92d1-a50dfdbbeb50\") " Jul 10 19:45:03.838330 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:03.838305 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/540ace4f-2fd3-4489-92d1-a50dfdbbeb50-maas-api-tls\") pod \"540ace4f-2fd3-4489-92d1-a50dfdbbeb50\" (UID: \"540ace4f-2fd3-4489-92d1-a50dfdbbeb50\") " Jul 10 19:45:03.838507 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:03.838363 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gqmv\" (UniqueName: \"kubernetes.io/projected/540ace4f-2fd3-4489-92d1-a50dfdbbeb50-kube-api-access-8gqmv\") pod \"540ace4f-2fd3-4489-92d1-a50dfdbbeb50\" (UID: \"540ace4f-2fd3-4489-92d1-a50dfdbbeb50\") " Jul 10 19:45:03.838646 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:03.838627 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/540ace4f-2fd3-4489-92d1-a50dfdbbeb50-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "540ace4f-2fd3-4489-92d1-a50dfdbbeb50" (UID: "540ace4f-2fd3-4489-92d1-a50dfdbbeb50"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:45:03.840364 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:03.840338 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/540ace4f-2fd3-4489-92d1-a50dfdbbeb50-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "540ace4f-2fd3-4489-92d1-a50dfdbbeb50" (UID: "540ace4f-2fd3-4489-92d1-a50dfdbbeb50"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:45:03.840364 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:03.840347 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/540ace4f-2fd3-4489-92d1-a50dfdbbeb50-kube-api-access-8gqmv" (OuterVolumeSpecName: "kube-api-access-8gqmv") pod "540ace4f-2fd3-4489-92d1-a50dfdbbeb50" (UID: "540ace4f-2fd3-4489-92d1-a50dfdbbeb50"). InnerVolumeSpecName "kube-api-access-8gqmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:45:03.939296 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:03.939272 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gqmv\" (UniqueName: \"kubernetes.io/projected/540ace4f-2fd3-4489-92d1-a50dfdbbeb50-kube-api-access-8gqmv\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:45:03.939296 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:03.939293 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/540ace4f-2fd3-4489-92d1-a50dfdbbeb50-openshift-service-ca\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:45:03.939433 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:03.939302 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/540ace4f-2fd3-4489-92d1-a50dfdbbeb50-maas-api-tls\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:45:04.073536 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:04.073511 2560 generic.go:356] "Generic (PLEG): container finished" podID="540ace4f-2fd3-4489-92d1-a50dfdbbeb50" containerID="51723bb66c4d2a71425e4d13aabf13f11b1b8737a5edc2e2bf5c42d7dd5ae31c" exitCode=0 Jul 10 19:45:04.073640 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:04.073550 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z" Jul 10 19:45:04.073640 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:04.073598 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z" event={"ID":"540ace4f-2fd3-4489-92d1-a50dfdbbeb50","Type":"ContainerDied","Data":"51723bb66c4d2a71425e4d13aabf13f11b1b8737a5edc2e2bf5c42d7dd5ae31c"} Jul 10 19:45:04.073760 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:04.073656 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z" event={"ID":"540ace4f-2fd3-4489-92d1-a50dfdbbeb50","Type":"ContainerDied","Data":"a3b95e34a93d115779fc31485e3505a9241f33f07debdb0688119fe062e29ff4"} Jul 10 19:45:04.073760 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:04.073689 2560 scope.go:117] "RemoveContainer" containerID="51723bb66c4d2a71425e4d13aabf13f11b1b8737a5edc2e2bf5c42d7dd5ae31c" Jul 10 19:45:04.074958 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:04.074855 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728545-fpmw8" event={"ID":"0f414efb-542b-4a59-813c-2dd0f09db2e1","Type":"ContainerDied","Data":"f1235beae7778f9d1cc6964d5d40f0bc6b06bdaf9e471fdae5ee52dc96c8eda6"} Jul 10 19:45:04.074958 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:04.074875 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728545-fpmw8" Jul 10 19:45:04.074958 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:04.074885 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1235beae7778f9d1cc6964d5d40f0bc6b06bdaf9e471fdae5ee52dc96c8eda6" Jul 10 19:45:04.085555 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:04.085540 2560 scope.go:117] "RemoveContainer" containerID="51723bb66c4d2a71425e4d13aabf13f11b1b8737a5edc2e2bf5c42d7dd5ae31c" Jul 10 19:45:04.085804 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:45:04.085786 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51723bb66c4d2a71425e4d13aabf13f11b1b8737a5edc2e2bf5c42d7dd5ae31c\": container with ID starting with 51723bb66c4d2a71425e4d13aabf13f11b1b8737a5edc2e2bf5c42d7dd5ae31c not found: ID does not exist" containerID="51723bb66c4d2a71425e4d13aabf13f11b1b8737a5edc2e2bf5c42d7dd5ae31c" Jul 10 19:45:04.085852 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:04.085812 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51723bb66c4d2a71425e4d13aabf13f11b1b8737a5edc2e2bf5c42d7dd5ae31c"} err="failed to get container status \"51723bb66c4d2a71425e4d13aabf13f11b1b8737a5edc2e2bf5c42d7dd5ae31c\": rpc error: code = NotFound desc = could not find container \"51723bb66c4d2a71425e4d13aabf13f11b1b8737a5edc2e2bf5c42d7dd5ae31c\": container with ID starting with 51723bb66c4d2a71425e4d13aabf13f11b1b8737a5edc2e2bf5c42d7dd5ae31c not found: ID does not exist" Jul 10 19:45:04.097581 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:04.097534 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z"] Jul 10 19:45:04.100822 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:04.100804 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-3ec3c4-b9c94dfb5-c7h4z"] Jul 10 19:45:05.252335 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:05.252303 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="540ace4f-2fd3-4489-92d1-a50dfdbbeb50" path="/var/lib/kubelet/pods/540ace4f-2fd3-4489-92d1-a50dfdbbeb50/volumes" Jul 10 19:45:08.144233 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:08.144191 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:45:14.929476 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:14.929427 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:45:15.030280 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:15.030231 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:45:18.864116 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:18.864085 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9"] Jul 10 19:45:18.864567 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:18.864284 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" podUID="3bf5ad96-432f-458c-973b-51875955b64e" containerName="istio-proxy" containerID="cri-o://bc14891d46bf60ed7c0b903b518c0b4cf6dd95a493b0d9cdd06e2394d0849604" gracePeriod=30 Jul 10 19:45:18.931512 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:18.931477 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:45:19.111041 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:19.111002 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:45:24.112578 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.112553 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:45:24.173358 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.173328 2560 generic.go:356] "Generic (PLEG): container finished" podID="3bf5ad96-432f-458c-973b-51875955b64e" containerID="bc14891d46bf60ed7c0b903b518c0b4cf6dd95a493b0d9cdd06e2394d0849604" exitCode=0 Jul 10 19:45:24.173508 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.173372 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" Jul 10 19:45:24.173508 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.173408 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" event={"ID":"3bf5ad96-432f-458c-973b-51875955b64e","Type":"ContainerDied","Data":"bc14891d46bf60ed7c0b903b518c0b4cf6dd95a493b0d9cdd06e2394d0849604"} Jul 10 19:45:24.173508 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.173474 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9" event={"ID":"3bf5ad96-432f-458c-973b-51875955b64e","Type":"ContainerDied","Data":"99b423c310720207e2cdf84b2385b5a78853fa678ac3b7bb67360f8e2c7c2a64"} Jul 10 19:45:24.173508 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.173503 2560 scope.go:117] "RemoveContainer" containerID="bc14891d46bf60ed7c0b903b518c0b4cf6dd95a493b0d9cdd06e2394d0849604" Jul 10 19:45:24.181707 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.181691 2560 scope.go:117] "RemoveContainer" containerID="62e1e54b1bb387d1dbeea93b4df9fb03fcce54ce1fd5ea5c6856957bc63f39ac" Jul 10 19:45:24.189167 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.189149 2560 scope.go:117] "RemoveContainer" containerID="bc14891d46bf60ed7c0b903b518c0b4cf6dd95a493b0d9cdd06e2394d0849604" Jul 10 19:45:24.189427 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:45:24.189402 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc14891d46bf60ed7c0b903b518c0b4cf6dd95a493b0d9cdd06e2394d0849604\": container with ID starting with bc14891d46bf60ed7c0b903b518c0b4cf6dd95a493b0d9cdd06e2394d0849604 not found: ID does not exist" containerID="bc14891d46bf60ed7c0b903b518c0b4cf6dd95a493b0d9cdd06e2394d0849604" Jul 10 19:45:24.189538 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.189453 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc14891d46bf60ed7c0b903b518c0b4cf6dd95a493b0d9cdd06e2394d0849604"} err="failed to get container status \"bc14891d46bf60ed7c0b903b518c0b4cf6dd95a493b0d9cdd06e2394d0849604\": rpc error: code = NotFound desc = could not find container \"bc14891d46bf60ed7c0b903b518c0b4cf6dd95a493b0d9cdd06e2394d0849604\": container with ID starting with bc14891d46bf60ed7c0b903b518c0b4cf6dd95a493b0d9cdd06e2394d0849604 not found: ID does not exist" Jul 10 19:45:24.189538 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.189479 2560 scope.go:117] "RemoveContainer" containerID="62e1e54b1bb387d1dbeea93b4df9fb03fcce54ce1fd5ea5c6856957bc63f39ac" Jul 10 19:45:24.189741 ip-10-0-138-213 kubenswrapper[2560]: E0710 19:45:24.189724 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62e1e54b1bb387d1dbeea93b4df9fb03fcce54ce1fd5ea5c6856957bc63f39ac\": container with ID starting with 62e1e54b1bb387d1dbeea93b4df9fb03fcce54ce1fd5ea5c6856957bc63f39ac not found: ID does not exist" containerID="62e1e54b1bb387d1dbeea93b4df9fb03fcce54ce1fd5ea5c6856957bc63f39ac" Jul 10 19:45:24.189784 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.189747 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e1e54b1bb387d1dbeea93b4df9fb03fcce54ce1fd5ea5c6856957bc63f39ac"} err="failed to get container status \"62e1e54b1bb387d1dbeea93b4df9fb03fcce54ce1fd5ea5c6856957bc63f39ac\": rpc error: code = NotFound desc = could not find container \"62e1e54b1bb387d1dbeea93b4df9fb03fcce54ce1fd5ea5c6856957bc63f39ac\": container with ID starting with 62e1e54b1bb387d1dbeea93b4df9fb03fcce54ce1fd5ea5c6856957bc63f39ac not found: ID does not exist" Jul 10 19:45:24.212190 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.212169 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bf5ad96-432f-458c-973b-51875955b64e-istio-token\") pod \"3bf5ad96-432f-458c-973b-51875955b64e\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " Jul 10 19:45:24.212283 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.212217 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf5ad96-432f-458c-973b-51875955b64e-workload-certs\") pod \"3bf5ad96-432f-458c-973b-51875955b64e\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " Jul 10 19:45:24.212342 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.212289 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8xxm\" (UniqueName: \"kubernetes.io/projected/3bf5ad96-432f-458c-973b-51875955b64e-kube-api-access-k8xxm\") pod \"3bf5ad96-432f-458c-973b-51875955b64e\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " Jul 10 19:45:24.212395 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.212339 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf5ad96-432f-458c-973b-51875955b64e-istiod-ca-cert\") pod \"3bf5ad96-432f-458c-973b-51875955b64e\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " Jul 10 19:45:24.212472 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.212404 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf5ad96-432f-458c-973b-51875955b64e-istio-podinfo\") pod \"3bf5ad96-432f-458c-973b-51875955b64e\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " Jul 10 19:45:24.212472 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.212430 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf5ad96-432f-458c-973b-51875955b64e-istio-envoy\") pod \"3bf5ad96-432f-458c-973b-51875955b64e\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " Jul 10 19:45:24.212574 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.212483 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf5ad96-432f-458c-973b-51875955b64e-istio-data\") pod \"3bf5ad96-432f-458c-973b-51875955b64e\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " Jul 10 19:45:24.212574 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.212513 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf5ad96-432f-458c-973b-51875955b64e-workload-socket\") pod \"3bf5ad96-432f-458c-973b-51875955b64e\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " Jul 10 19:45:24.212574 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.212543 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf5ad96-432f-458c-973b-51875955b64e-credential-socket\") pod \"3bf5ad96-432f-458c-973b-51875955b64e\" (UID: \"3bf5ad96-432f-458c-973b-51875955b64e\") " Jul 10 19:45:24.212574 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.212555 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf5ad96-432f-458c-973b-51875955b64e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3bf5ad96-432f-458c-973b-51875955b64e" (UID: "3bf5ad96-432f-458c-973b-51875955b64e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:45:24.212845 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.212810 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf5ad96-432f-458c-973b-51875955b64e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3bf5ad96-432f-458c-973b-51875955b64e" (UID: "3bf5ad96-432f-458c-973b-51875955b64e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:45:24.212926 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.212904 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf5ad96-432f-458c-973b-51875955b64e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3bf5ad96-432f-458c-973b-51875955b64e" (UID: "3bf5ad96-432f-458c-973b-51875955b64e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:45:24.213058 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.213017 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf5ad96-432f-458c-973b-51875955b64e-istiod-ca-cert\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:45:24.213058 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.213040 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf5ad96-432f-458c-973b-51875955b64e-workload-socket\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:45:24.213058 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.213054 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf5ad96-432f-458c-973b-51875955b64e-workload-certs\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:45:24.213246 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.213016 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf5ad96-432f-458c-973b-51875955b64e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3bf5ad96-432f-458c-973b-51875955b64e" (UID: "3bf5ad96-432f-458c-973b-51875955b64e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:45:24.213591 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.213566 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf5ad96-432f-458c-973b-51875955b64e-istio-data" (OuterVolumeSpecName: "istio-data") pod "3bf5ad96-432f-458c-973b-51875955b64e" (UID: "3bf5ad96-432f-458c-973b-51875955b64e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:45:24.214412 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.214389 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf5ad96-432f-458c-973b-51875955b64e-kube-api-access-k8xxm" (OuterVolumeSpecName: "kube-api-access-k8xxm") pod "3bf5ad96-432f-458c-973b-51875955b64e" (UID: "3bf5ad96-432f-458c-973b-51875955b64e"). InnerVolumeSpecName "kube-api-access-k8xxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:45:24.214634 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.214616 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bf5ad96-432f-458c-973b-51875955b64e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3bf5ad96-432f-458c-973b-51875955b64e" (UID: "3bf5ad96-432f-458c-973b-51875955b64e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 19:45:24.214710 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.214688 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf5ad96-432f-458c-973b-51875955b64e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3bf5ad96-432f-458c-973b-51875955b64e" (UID: "3bf5ad96-432f-458c-973b-51875955b64e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:45:24.214770 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.214737 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf5ad96-432f-458c-973b-51875955b64e-istio-token" (OuterVolumeSpecName: "istio-token") pod "3bf5ad96-432f-458c-973b-51875955b64e" (UID: "3bf5ad96-432f-458c-973b-51875955b64e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:45:24.314227 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.314204 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8xxm\" (UniqueName: \"kubernetes.io/projected/3bf5ad96-432f-458c-973b-51875955b64e-kube-api-access-k8xxm\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:45:24.314227 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.314226 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf5ad96-432f-458c-973b-51875955b64e-istio-podinfo\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:45:24.314353 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.314235 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf5ad96-432f-458c-973b-51875955b64e-istio-envoy\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:45:24.314353 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.314243 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf5ad96-432f-458c-973b-51875955b64e-istio-data\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:45:24.314353 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.314250 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf5ad96-432f-458c-973b-51875955b64e-credential-socket\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:45:24.314353 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.314258 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bf5ad96-432f-458c-973b-51875955b64e-istio-token\") on node \"ip-10-0-138-213.ec2.internal\" DevicePath \"\"" Jul 10 19:45:24.495364 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.495340 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9"] Jul 10 19:45:24.499194 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.499174 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-3ec3c4-openshift-default-646c75b4bc-8lcm9"] Jul 10 19:45:24.653832 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.653809 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cfcf59dd-kwh88_e004b747-8a02-46be-93aa-0d77e56ab69a/copy-manifests/0.log" Jul 10 19:45:24.660130 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:24.660109 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cfcf59dd-kwh88_e004b747-8a02-46be-93aa-0d77e56ab69a/manager/0.log" Jul 10 19:45:25.252430 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:25.252390 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bf5ad96-432f-458c-973b-51875955b64e" path="/var/lib/kubelet/pods/3bf5ad96-432f-458c-973b-51875955b64e/volumes" Jul 10 19:45:25.370204 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:25.370179 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29728515-l42tt_83fa6ee5-9b25-4431-be98-9d359778d5da/cleanup/0.log" Jul 10 19:45:25.478752 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:25.478722 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29728530-zj5fd_0da78f1c-0aff-4a10-adf0-e0186c0c5e34/cleanup/0.log" Jul 10 19:45:25.581202 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:25.581139 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29728545-fpmw8_0f414efb-542b-4a59-813c-2dd0f09db2e1/cleanup/0.log" Jul 10 19:45:29.212918 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:29.212887 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4gzgzh_a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3/istio-proxy/0.log" Jul 10 19:45:29.328874 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:29.328845 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-pr25j_fb9e1aa3-6f16-42eb-a72d-aefe00037d58/discovery/0.log" Jul 10 19:45:29.653827 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:29.653797 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-82b42_df98b8de-e87e-4a94-adb1-2daf956c61c0/istio-proxy/1.log" Jul 10 19:45:29.767950 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:29.767912 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-lvdc7_0a6619a8-29b5-4241-bdb1-8a44bf57b1da/payload-pre-processing/0.log" Jul 10 19:45:29.883725 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:29.883653 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-sklxc_ef212931-5043-4f3a-804e-d2b071ee7a14/payload-processing/0.log" Jul 10 19:45:30.427989 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:30.427962 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-4vftx_96835982-08fa-4605-8290-ffd6d56343e7/main/0.log" Jul 10 19:45:30.638112 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:30.638086 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblvm4j_8a3b9969-6253-4250-a8c3-d87cbb56579a/main/0.log" Jul 10 19:45:34.422565 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:34.422533 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:45:34.547856 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:34.547817 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" limit=1024 Jul 10 19:45:37.409314 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:37.409286 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zv9t9_e3e3a914-d7de-4771-9d68-a7b05ab56b1f/global-pull-secret-syncer/0.log" Jul 10 19:45:37.466784 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:37.466759 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4hn8z_00d8d68a-3d1f-4b7f-b9a7-ef54b3489e4a/konnectivity-agent/0.log" Jul 10 19:45:37.596319 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:37.596288 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-213.ec2.internal_14b1acdcd9ea09cc31366e606e569025/haproxy/0.log" Jul 10 19:45:45.127437 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:45.127398 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a527d405-ebba-4ca0-a6fa-a7a29717965c/alertmanager/0.log" Jul 10 19:45:45.150980 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:45.150961 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a527d405-ebba-4ca0-a6fa-a7a29717965c/config-reloader/0.log" Jul 10 19:45:45.176892 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:45.176864 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a527d405-ebba-4ca0-a6fa-a7a29717965c/kube-rbac-proxy-web/0.log" Jul 10 19:45:45.200103 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:45.200083 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a527d405-ebba-4ca0-a6fa-a7a29717965c/kube-rbac-proxy/0.log" Jul 10 19:45:45.223716 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:45.223693 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a527d405-ebba-4ca0-a6fa-a7a29717965c/kube-rbac-proxy-metric/0.log" Jul 10 19:45:45.261851 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:45.261833 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a527d405-ebba-4ca0-a6fa-a7a29717965c/prom-label-proxy/0.log" Jul 10 19:45:45.285057 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:45.285033 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a527d405-ebba-4ca0-a6fa-a7a29717965c/init-config-reloader/0.log" Jul 10 19:45:45.369538 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:45.369475 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-t5qwt_144f82fa-744c-458e-86a7-7d16ebebce5d/kube-state-metrics/0.log" Jul 10 19:45:45.387978 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:45.387956 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-t5qwt_144f82fa-744c-458e-86a7-7d16ebebce5d/kube-rbac-proxy-main/0.log" Jul 10 19:45:45.409628 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:45.409603 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-t5qwt_144f82fa-744c-458e-86a7-7d16ebebce5d/kube-rbac-proxy-self/0.log" Jul 10 19:45:45.438945 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:45.438923 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-546f8cd949-89c7k_321cf2d7-c2b3-4b43-9c1a-ebb28da445d7/metrics-server/0.log" Jul 10 19:45:45.507832 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:45.507814 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8ldbp_3b650cfc-a576-4008-afe7-36d8d34462f6/node-exporter/0.log" Jul 10 19:45:45.530492 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:45.530469 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8ldbp_3b650cfc-a576-4008-afe7-36d8d34462f6/kube-rbac-proxy/0.log" Jul 10 19:45:45.552393 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:45.552379 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8ldbp_3b650cfc-a576-4008-afe7-36d8d34462f6/init-textfile/0.log" Jul 10 19:45:45.730609 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:45.730581 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-zzjcj_63ee3b4a-66e3-4fe3-88a8-329a848e3faf/kube-rbac-proxy-main/0.log" Jul 10 19:45:45.752371 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:45.752344 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-zzjcj_63ee3b4a-66e3-4fe3-88a8-329a848e3faf/kube-rbac-proxy-self/0.log" Jul 10 19:45:45.772934 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:45.772911 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-zzjcj_63ee3b4a-66e3-4fe3-88a8-329a848e3faf/openshift-state-metrics/0.log" Jul 10 19:45:45.818964 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:45.818939 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f21aeaa5-003b-4ba3-86b9-9a21451bc3fb/prometheus/0.log" Jul 10 19:45:45.843034 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:45.843011 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f21aeaa5-003b-4ba3-86b9-9a21451bc3fb/config-reloader/0.log" Jul 10 19:45:45.871234 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:45.871215 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f21aeaa5-003b-4ba3-86b9-9a21451bc3fb/thanos-sidecar/0.log" Jul 10 19:45:45.892269 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:45.892247 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f21aeaa5-003b-4ba3-86b9-9a21451bc3fb/kube-rbac-proxy-web/0.log" Jul 10 19:45:45.914445 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:45.914430 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f21aeaa5-003b-4ba3-86b9-9a21451bc3fb/kube-rbac-proxy/0.log" Jul 10 19:45:45.937721 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:45.937705 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f21aeaa5-003b-4ba3-86b9-9a21451bc3fb/kube-rbac-proxy-thanos/0.log" Jul 10 19:45:45.968525 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:45.968509 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f21aeaa5-003b-4ba3-86b9-9a21451bc3fb/init-config-reloader/0.log" Jul 10 19:45:46.049344 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:46.049292 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-t2x4s_0c3d7d0c-941d-43cc-9130-64169639313f/prometheus-operator-admission-webhook/0.log" Jul 10 19:45:46.080908 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:46.080888 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-94bd4b7b7-m4h5j_5efa515b-3b7c-4c71-820e-15c9cf7617b7/telemeter-client/0.log" Jul 10 19:45:46.104216 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:46.104191 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-94bd4b7b7-m4h5j_5efa515b-3b7c-4c71-820e-15c9cf7617b7/reload/0.log" Jul 10 19:45:46.132599 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:46.132573 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-94bd4b7b7-m4h5j_5efa515b-3b7c-4c71-820e-15c9cf7617b7/kube-rbac-proxy/0.log" Jul 10 19:45:47.020476 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.020433 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q9l4b/perf-node-gather-daemonset-thk75"] Jul 10 19:45:47.020828 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.020816 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f414efb-542b-4a59-813c-2dd0f09db2e1" containerName="cleanup" Jul 10 19:45:47.020872 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.020830 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f414efb-542b-4a59-813c-2dd0f09db2e1" containerName="cleanup" Jul 10 19:45:47.020872 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.020841 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="540ace4f-2fd3-4489-92d1-a50dfdbbeb50" containerName="maas-api" Jul 10 19:45:47.020872 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.020847 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="540ace4f-2fd3-4489-92d1-a50dfdbbeb50" containerName="maas-api" Jul 10 19:45:47.020872 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.020858 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bf5ad96-432f-458c-973b-51875955b64e" containerName="istio-proxy" Jul 10 19:45:47.020872 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.020863 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bf5ad96-432f-458c-973b-51875955b64e" containerName="istio-proxy" Jul 10 19:45:47.020872 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.020869 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bf5ad96-432f-458c-973b-51875955b64e" containerName="istio-proxy" Jul 10 19:45:47.021052 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.020875 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bf5ad96-432f-458c-973b-51875955b64e" containerName="istio-proxy" Jul 10 19:45:47.021052 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.020894 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b23f0def-02ec-422f-bc7e-b54897787bc1" containerName="istio-proxy" Jul 10 19:45:47.021052 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.020899 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b23f0def-02ec-422f-bc7e-b54897787bc1" containerName="istio-proxy" Jul 10 19:45:47.021052 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.020952 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f414efb-542b-4a59-813c-2dd0f09db2e1" containerName="cleanup" Jul 10 19:45:47.021052 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.020959 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bf5ad96-432f-458c-973b-51875955b64e" containerName="istio-proxy" Jul 10 19:45:47.021052 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.020966 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="540ace4f-2fd3-4489-92d1-a50dfdbbeb50" containerName="maas-api" Jul 10 19:45:47.024279 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.024262 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-thk75" Jul 10 19:45:47.027290 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.027272 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q9l4b\"/\"openshift-service-ca.crt\"" Jul 10 19:45:47.028536 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.028518 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-q9l4b\"/\"default-dockercfg-7kvh2\"" Jul 10 19:45:47.028642 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.028588 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q9l4b\"/\"kube-root-ca.crt\"" Jul 10 19:45:47.036549 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.036528 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q9l4b/perf-node-gather-daemonset-thk75"] Jul 10 19:45:47.103432 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.103403 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cac9105e-d70f-4925-b761-3476919b4c1c-podres\") pod \"perf-node-gather-daemonset-thk75\" (UID: \"cac9105e-d70f-4925-b761-3476919b4c1c\") " pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-thk75" Jul 10 19:45:47.103618 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.103509 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cac9105e-d70f-4925-b761-3476919b4c1c-lib-modules\") pod \"perf-node-gather-daemonset-thk75\" (UID: \"cac9105e-d70f-4925-b761-3476919b4c1c\") " pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-thk75" Jul 10 19:45:47.103618 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.103576 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cac9105e-d70f-4925-b761-3476919b4c1c-proc\") pod \"perf-node-gather-daemonset-thk75\" (UID: \"cac9105e-d70f-4925-b761-3476919b4c1c\") " pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-thk75" Jul 10 19:45:47.103740 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.103648 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cac9105e-d70f-4925-b761-3476919b4c1c-sys\") pod \"perf-node-gather-daemonset-thk75\" (UID: \"cac9105e-d70f-4925-b761-3476919b4c1c\") " pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-thk75" Jul 10 19:45:47.103740 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.103698 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpwrw\" (UniqueName: \"kubernetes.io/projected/cac9105e-d70f-4925-b761-3476919b4c1c-kube-api-access-wpwrw\") pod \"perf-node-gather-daemonset-thk75\" (UID: \"cac9105e-d70f-4925-b761-3476919b4c1c\") " pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-thk75" Jul 10 19:45:47.204778 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.204742 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cac9105e-d70f-4925-b761-3476919b4c1c-podres\") pod \"perf-node-gather-daemonset-thk75\" (UID: \"cac9105e-d70f-4925-b761-3476919b4c1c\") " pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-thk75" Jul 10 19:45:47.205122 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.204788 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cac9105e-d70f-4925-b761-3476919b4c1c-lib-modules\") pod \"perf-node-gather-daemonset-thk75\" (UID: \"cac9105e-d70f-4925-b761-3476919b4c1c\") " pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-thk75" Jul 10 19:45:47.205122 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.204830 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cac9105e-d70f-4925-b761-3476919b4c1c-proc\") pod \"perf-node-gather-daemonset-thk75\" (UID: \"cac9105e-d70f-4925-b761-3476919b4c1c\") " pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-thk75" Jul 10 19:45:47.205122 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.204880 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cac9105e-d70f-4925-b761-3476919b4c1c-sys\") pod \"perf-node-gather-daemonset-thk75\" (UID: \"cac9105e-d70f-4925-b761-3476919b4c1c\") " pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-thk75" Jul 10 19:45:47.205122 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.204920 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpwrw\" (UniqueName: \"kubernetes.io/projected/cac9105e-d70f-4925-b761-3476919b4c1c-kube-api-access-wpwrw\") pod \"perf-node-gather-daemonset-thk75\" (UID: \"cac9105e-d70f-4925-b761-3476919b4c1c\") " pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-thk75" Jul 10 19:45:47.205122 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.204969 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cac9105e-d70f-4925-b761-3476919b4c1c-lib-modules\") pod \"perf-node-gather-daemonset-thk75\" (UID: \"cac9105e-d70f-4925-b761-3476919b4c1c\") " pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-thk75" Jul 10 19:45:47.205122 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.204994 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cac9105e-d70f-4925-b761-3476919b4c1c-proc\") pod \"perf-node-gather-daemonset-thk75\" (UID: \"cac9105e-d70f-4925-b761-3476919b4c1c\") " pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-thk75" Jul 10 19:45:47.205122 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.205004 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cac9105e-d70f-4925-b761-3476919b4c1c-sys\") pod \"perf-node-gather-daemonset-thk75\" (UID: \"cac9105e-d70f-4925-b761-3476919b4c1c\") " pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-thk75" Jul 10 19:45:47.205122 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.205054 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cac9105e-d70f-4925-b761-3476919b4c1c-podres\") pod \"perf-node-gather-daemonset-thk75\" (UID: \"cac9105e-d70f-4925-b761-3476919b4c1c\") " pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-thk75" Jul 10 19:45:47.213790 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.213765 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpwrw\" (UniqueName: \"kubernetes.io/projected/cac9105e-d70f-4925-b761-3476919b4c1c-kube-api-access-wpwrw\") pod \"perf-node-gather-daemonset-thk75\" (UID: \"cac9105e-d70f-4925-b761-3476919b4c1c\") " pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-thk75" Jul 10 19:45:47.334763 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.334697 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-thk75" Jul 10 19:45:47.465436 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.465409 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q9l4b/perf-node-gather-daemonset-thk75"] Jul 10 19:45:47.468341 ip-10-0-138-213 kubenswrapper[2560]: W0710 19:45:47.468310 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcac9105e_d70f_4925_b761_3476919b4c1c.slice/crio-92d1e1fed0c653efc36afe6b53da925915714c02c0cb4158c9b76dfc0829a9be WatchSource:0}: Error finding container 92d1e1fed0c653efc36afe6b53da925915714c02c0cb4158c9b76dfc0829a9be: Status 404 returned error can't find the container with id 92d1e1fed0c653efc36afe6b53da925915714c02c0cb4158c9b76dfc0829a9be Jul 10 19:45:47.495421 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:47.495387 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-bjg48_34f5d206-d3be-47c7-a271-7486c9391651/networking-console-plugin/0.log" Jul 10 19:45:48.291265 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:48.291230 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-thk75" event={"ID":"cac9105e-d70f-4925-b761-3476919b4c1c","Type":"ContainerStarted","Data":"2e693c357be838ac3aa35299918f86a54bbec5ff71bf9b53165fdd8fdccd362f"} Jul 10 19:45:48.291745 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:48.291290 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-thk75" event={"ID":"cac9105e-d70f-4925-b761-3476919b4c1c","Type":"ContainerStarted","Data":"92d1e1fed0c653efc36afe6b53da925915714c02c0cb4158c9b76dfc0829a9be"} Jul 10 19:45:48.291745 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:48.291346 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-thk75" Jul 10 19:45:48.311829 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:48.311781 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-thk75" podStartSLOduration=2.311766999 podStartE2EDuration="2.311766999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:45:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:45:48.309982837 +0000 UTC m=+2839.640123679" watchObservedRunningTime="2026-07-10 19:45:48.311766999 +0000 UTC m=+2839.641907843" Jul 10 19:45:50.284957 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:50.284924 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d7l7x_e9ea737c-ac9d-4694-bd6b-980514db5371/dns/0.log" Jul 10 19:45:50.307354 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:50.307323 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d7l7x_e9ea737c-ac9d-4694-bd6b-980514db5371/kube-rbac-proxy/0.log" Jul 10 19:45:50.382238 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:50.382215 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l9qmm_d85934ea-40c2-4589-aca7-b0e8efca569c/dns-node-resolver/0.log" Jul 10 19:45:51.116297 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:51.116249 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x6sx7_fd6564ad-086b-4c7b-ada7-99f16bfe03c4/node-ca/0.log" Jul 10 19:45:52.167734 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:52.167703 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4gzgzh_a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3/istio-proxy/0.log" Jul 10 19:45:52.363896 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:52.363864 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-pr25j_fb9e1aa3-6f16-42eb-a72d-aefe00037d58/discovery/0.log" Jul 10 19:45:52.445131 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:52.445053 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-82b42_df98b8de-e87e-4a94-adb1-2daf956c61c0/istio-proxy/0.log" Jul 10 19:45:52.502043 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:52.502022 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-82b42_df98b8de-e87e-4a94-adb1-2daf956c61c0/istio-proxy/1.log" Jul 10 19:45:52.544819 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:52.544799 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-lvdc7_0a6619a8-29b5-4241-bdb1-8a44bf57b1da/payload-pre-processing/0.log" Jul 10 19:45:52.584696 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:52.584668 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-sklxc_ef212931-5043-4f3a-804e-d2b071ee7a14/payload-processing/0.log" Jul 10 19:45:53.403037 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:53.403007 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-z8vwd_d95e6de4-e5cc-4467-a6b9-4ac8d842d744/serve-healthcheck-canary/0.log" Jul 10 19:45:53.903657 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:53.903625 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-vshg7_88da616b-477f-491f-bf78-d80b9c187a24/insights-operator/0.log" Jul 10 19:45:53.905038 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:53.905018 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-vshg7_88da616b-477f-491f-bf78-d80b9c187a24/insights-operator/1.log" Jul 10 19:45:54.093519 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:54.093493 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sdt64_06a74dc2-5187-47af-9eeb-77f4ce8a8d7a/kube-rbac-proxy/0.log" Jul 10 19:45:54.115730 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:54.115704 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sdt64_06a74dc2-5187-47af-9eeb-77f4ce8a8d7a/exporter/0.log" Jul 10 19:45:54.139491 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:54.139469 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sdt64_06a74dc2-5187-47af-9eeb-77f4ce8a8d7a/extractor/0.log" Jul 10 19:45:54.305192 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:54.305173 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-thk75" Jul 10 19:45:57.070457 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:57.070423 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cfcf59dd-kwh88_e004b747-8a02-46be-93aa-0d77e56ab69a/manager/0.log" Jul 10 19:45:57.091407 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:57.091389 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cfcf59dd-kwh88_e004b747-8a02-46be-93aa-0d77e56ab69a/copy-manifests/0.log" Jul 10 19:45:57.652905 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:57.652880 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4gzgzh_a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3/istio-proxy/0.log" Jul 10 19:45:57.835042 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:57.835011 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-pr25j_fb9e1aa3-6f16-42eb-a72d-aefe00037d58/discovery/0.log" Jul 10 19:45:57.897158 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:57.897128 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-82b42_df98b8de-e87e-4a94-adb1-2daf956c61c0/istio-proxy/0.log" Jul 10 19:45:57.959878 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:57.959847 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-82b42_df98b8de-e87e-4a94-adb1-2daf956c61c0/istio-proxy/1.log" Jul 10 19:45:57.988278 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:57.988254 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-lvdc7_0a6619a8-29b5-4241-bdb1-8a44bf57b1da/payload-pre-processing/0.log" Jul 10 19:45:58.017755 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:58.017729 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-sklxc_ef212931-5043-4f3a-804e-d2b071ee7a14/payload-processing/0.log" Jul 10 19:45:58.600086 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:58.600056 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5b69967895-zbggq_60a4337a-8004-43ae-b41f-62b2ab657468/manager/0.log" Jul 10 19:45:59.278556 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:59.278524 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4gzgzh_a068c7cb-6506-42f4-bcb6-b49f0ec0d6c3/istio-proxy/0.log" Jul 10 19:45:59.473337 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:59.473309 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-pr25j_fb9e1aa3-6f16-42eb-a72d-aefe00037d58/discovery/0.log" Jul 10 19:45:59.526278 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:59.526252 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-82b42_df98b8de-e87e-4a94-adb1-2daf956c61c0/istio-proxy/0.log" Jul 10 19:45:59.580585 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:59.580494 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-82b42_df98b8de-e87e-4a94-adb1-2daf956c61c0/istio-proxy/1.log" Jul 10 19:45:59.607925 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:59.607895 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-lvdc7_0a6619a8-29b5-4241-bdb1-8a44bf57b1da/payload-pre-processing/0.log" Jul 10 19:45:59.643814 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:45:59.643775 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-sklxc_ef212931-5043-4f3a-804e-d2b071ee7a14/payload-processing/0.log" Jul 10 19:46:03.878763 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:46:03.878731 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-dd8w8_651e6091-4d1a-4adc-90bf-b0a7e5f8291c/kube-storage-version-migrator-operator/1.log" Jul 10 19:46:03.880526 ip-10-0-138-213 kubenswrapper[2560]: I0710 19:46:03.880504 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-dd8w8_651e6091-4d1a-4adc-90bf-b0a7e5f8291c/kube-storage-version-migrator-operator/0.log"