Jul 16 12:37:01.309763 ip-10-0-140-130 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 16 12:37:01.309773 ip-10-0-140-130 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 16 12:37:01.309780 ip-10-0-140-130 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 16 12:37:01.310016 ip-10-0-140-130 systemd[1]: Failed to start Kubernetes Kubelet. Jul 16 12:37:10.266080 ip-10-0-140-130 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 16 12:37:10.266098 ip-10-0-140-130 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 54d70e84e00b4c7d9da31caf5f9abce8 -- Jul 16 12:39:01.423709 ip-10-0-140-130 systemd[1]: Starting Kubernetes Kubelet... Jul 16 12:39:01.761156 ip-10-0-140-130 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 12:39:01.761156 ip-10-0-140-130 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 16 12:39:01.761156 ip-10-0-140-130 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 12:39:01.761156 ip-10-0-140-130 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 16 12:39:01.761156 ip-10-0-140-130 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 12:39:01.763145 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.763061 2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 16 12:39:01.768412 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768396 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 12:39:01.768412 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768411 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 12:39:01.768412 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768414 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 12:39:01.768500 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768417 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 12:39:01.768500 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768420 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 12:39:01.768500 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768423 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 12:39:01.768500 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768425 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 12:39:01.768500 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768428 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 12:39:01.768500 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768431 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 12:39:01.768500 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768433 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 12:39:01.768500 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768436 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 12:39:01.768500 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768440 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 12:39:01.768500 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768442 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 12:39:01.768500 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768444 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 12:39:01.768500 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768447 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 12:39:01.768500 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768455 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 12:39:01.768500 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768458 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 12:39:01.768500 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768461 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 12:39:01.768500 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768463 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 12:39:01.768500 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768466 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 12:39:01.768500 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768468 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 12:39:01.768500 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768470 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 12:39:01.768500 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768473 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 12:39:01.768969 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768475 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 12:39:01.768969 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768478 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 12:39:01.768969 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768480 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 12:39:01.768969 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768483 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 12:39:01.768969 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768486 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 12:39:01.768969 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768488 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 12:39:01.768969 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768490 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 12:39:01.768969 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768493 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 12:39:01.768969 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768495 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 12:39:01.768969 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768497 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 12:39:01.768969 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768500 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 12:39:01.768969 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768502 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 12:39:01.768969 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768505 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 12:39:01.768969 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768508 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 12:39:01.768969 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768510 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 12:39:01.768969 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768514 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 12:39:01.768969 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768516 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 12:39:01.768969 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768519 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 12:39:01.768969 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768521 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 12:39:01.769485 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768524 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 12:39:01.769485 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768526 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 12:39:01.769485 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768529 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 12:39:01.769485 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768531 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 12:39:01.769485 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768533 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 12:39:01.769485 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768536 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 12:39:01.769485 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768538 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 12:39:01.769485 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768540 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 12:39:01.769485 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768542 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 12:39:01.769485 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768545 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 12:39:01.769485 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768547 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 12:39:01.769485 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768549 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 12:39:01.769485 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768552 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 12:39:01.769485 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768555 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 12:39:01.769485 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768558 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 12:39:01.769485 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768560 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 12:39:01.769485 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768562 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 12:39:01.769485 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768565 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 12:39:01.769485 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768567 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 12:39:01.769485 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768570 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 12:39:01.769966 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768573 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 16 12:39:01.769966 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768575 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 12:39:01.769966 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768580 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 12:39:01.769966 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768584 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 12:39:01.769966 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768587 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 12:39:01.769966 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768590 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 12:39:01.769966 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768593 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 12:39:01.769966 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768595 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 12:39:01.769966 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768598 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 12:39:01.769966 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768601 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 12:39:01.769966 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768604 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 12:39:01.769966 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768606 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 12:39:01.769966 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768609 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 12:39:01.769966 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768611 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 12:39:01.769966 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768615 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 12:39:01.769966 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768618 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 12:39:01.769966 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768620 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 12:39:01.769966 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768623 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 12:39:01.769966 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768626 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 12:39:01.770450 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768629 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 12:39:01.770450 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768632 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 12:39:01.770450 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768635 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 12:39:01.770450 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768639 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 12:39:01.770450 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768643 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 12:39:01.770450 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768645 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 12:39:01.770450 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768649 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 12:39:01.770450 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768652 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 12:39:01.770450 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768655 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 12:39:01.770450 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768657 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 12:39:01.770450 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768659 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 12:39:01.770450 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768662 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 12:39:01.770450 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768664 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 12:39:01.770450 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768667 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 12:39:01.770450 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768669 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 12:39:01.770450 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768671 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 12:39:01.770450 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768674 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 12:39:01.770450 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.768676 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 12:39:01.770450 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769346 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 12:39:01.770450 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769354 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 12:39:01.770923 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769357 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 12:39:01.770923 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769361 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 12:39:01.770923 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769365 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 12:39:01.770923 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769368 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 12:39:01.770923 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769372 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 12:39:01.770923 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769375 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 12:39:01.770923 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769378 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 12:39:01.770923 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769381 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 12:39:01.770923 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769383 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 12:39:01.770923 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769401 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 12:39:01.770923 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769404 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 12:39:01.770923 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769406 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 12:39:01.770923 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769409 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 12:39:01.770923 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769411 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 12:39:01.770923 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769413 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 12:39:01.770923 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769416 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 12:39:01.770923 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769418 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 12:39:01.770923 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769421 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 12:39:01.770923 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769424 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 12:39:01.771410 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769428 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 12:39:01.771410 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769431 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 12:39:01.771410 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769433 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 12:39:01.771410 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769436 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 12:39:01.771410 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769438 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 12:39:01.771410 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769441 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 12:39:01.771410 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769443 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 12:39:01.771410 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769446 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 12:39:01.771410 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769448 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 12:39:01.771410 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769450 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 12:39:01.771410 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769453 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 12:39:01.771410 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769455 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 12:39:01.771410 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769458 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 12:39:01.771410 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769460 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 12:39:01.771410 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769462 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 12:39:01.771410 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769471 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 12:39:01.771410 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769473 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 12:39:01.771410 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769476 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 12:39:01.771410 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769479 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 12:39:01.771410 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769481 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 12:39:01.771913 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769483 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 12:39:01.771913 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769486 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 12:39:01.771913 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769488 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 12:39:01.771913 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769491 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 12:39:01.771913 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769493 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 12:39:01.771913 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769496 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 12:39:01.771913 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769499 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 12:39:01.771913 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769501 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 12:39:01.771913 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769504 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 12:39:01.771913 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769506 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 12:39:01.771913 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769508 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 12:39:01.771913 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769511 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 12:39:01.771913 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769514 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 12:39:01.771913 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769517 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 12:39:01.771913 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769519 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 12:39:01.771913 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769523 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 12:39:01.771913 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769525 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 12:39:01.771913 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769528 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 12:39:01.771913 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769530 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 12:39:01.771913 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769532 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 12:39:01.772420 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769535 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 12:39:01.772420 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769537 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 12:39:01.772420 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769539 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 12:39:01.772420 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769542 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 12:39:01.772420 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769544 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 12:39:01.772420 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769546 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 12:39:01.772420 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769549 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 12:39:01.772420 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769551 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 12:39:01.772420 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769553 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 12:39:01.772420 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769555 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 12:39:01.772420 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769558 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 12:39:01.772420 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769560 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 12:39:01.772420 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769563 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 12:39:01.772420 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769566 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 12:39:01.772420 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769568 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 12:39:01.772420 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769570 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 12:39:01.772420 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769572 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 12:39:01.772420 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769575 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 12:39:01.772420 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769578 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 12:39:01.772876 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769582 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 12:39:01.772876 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769584 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 12:39:01.772876 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769587 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 12:39:01.772876 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769590 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 12:39:01.772876 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769593 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 12:39:01.772876 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769596 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 16 12:39:01.772876 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769599 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 12:39:01.772876 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769602 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 12:39:01.772876 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769604 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 12:39:01.772876 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769607 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 12:39:01.772876 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769610 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 12:39:01.772876 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769612 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 12:39:01.772876 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769615 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 12:39:01.772876 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769618 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 12:39:01.772876 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769620 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 12:39:01.772876 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769623 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 12:39:01.772876 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769625 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 12:39:01.772876 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769627 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 12:39:01.772876 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769630 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 12:39:01.772876 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.769998 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 12:39:01.773372 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770002 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 12:39:01.773372 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770005 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 12:39:01.773372 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770008 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 12:39:01.773372 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770016 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 12:39:01.773372 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770021 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 12:39:01.773372 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770024 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 12:39:01.773372 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770027 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 12:39:01.773372 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770030 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 12:39:01.773372 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770033 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 12:39:01.773372 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770035 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 12:39:01.773372 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770038 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 12:39:01.773372 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770040 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 12:39:01.773372 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770043 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 12:39:01.773372 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770045 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 12:39:01.773372 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770048 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 12:39:01.773372 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770050 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 12:39:01.773372 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770053 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 12:39:01.773372 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770056 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 12:39:01.773372 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770059 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 12:39:01.773846 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770061 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 12:39:01.773846 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770063 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 12:39:01.773846 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770066 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 12:39:01.773846 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770068 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 12:39:01.773846 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770071 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 12:39:01.773846 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770073 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 12:39:01.773846 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770075 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 12:39:01.773846 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770078 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 12:39:01.773846 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770080 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 12:39:01.773846 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770083 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 12:39:01.773846 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770087 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 12:39:01.773846 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770089 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 12:39:01.773846 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770091 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 12:39:01.773846 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770094 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 12:39:01.773846 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770096 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 12:39:01.773846 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770099 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 12:39:01.773846 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770101 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 12:39:01.773846 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770109 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 12:39:01.773846 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770114 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 12:39:01.774310 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770117 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 12:39:01.774310 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770120 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 12:39:01.774310 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770122 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 12:39:01.774310 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770125 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 12:39:01.774310 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770128 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 12:39:01.774310 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770130 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 12:39:01.774310 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770133 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 12:39:01.774310 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770135 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 12:39:01.774310 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770138 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 12:39:01.774310 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770140 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 12:39:01.774310 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770143 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 12:39:01.774310 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770145 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 12:39:01.774310 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770148 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 12:39:01.774310 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770150 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 12:39:01.774310 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770152 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 12:39:01.774310 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770155 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 12:39:01.774310 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770157 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 12:39:01.774310 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770159 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 12:39:01.774310 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770162 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 12:39:01.774310 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770164 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 12:39:01.774844 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770166 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 12:39:01.774844 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770168 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 12:39:01.774844 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770171 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 12:39:01.774844 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770173 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 12:39:01.774844 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770176 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 12:39:01.774844 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770179 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 12:39:01.774844 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770181 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 12:39:01.774844 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770184 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 12:39:01.774844 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770186 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 12:39:01.774844 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770188 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 12:39:01.774844 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770191 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 12:39:01.774844 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770199 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 12:39:01.774844 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770202 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 12:39:01.774844 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770204 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 12:39:01.774844 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770206 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 12:39:01.774844 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770209 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 12:39:01.774844 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770212 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 12:39:01.774844 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770214 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 12:39:01.774844 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770217 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 12:39:01.774844 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770219 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 12:39:01.775332 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770221 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 12:39:01.775332 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770224 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 12:39:01.775332 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770226 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 12:39:01.775332 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770229 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 12:39:01.775332 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770231 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 12:39:01.775332 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770233 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 12:39:01.775332 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770235 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 12:39:01.775332 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770238 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 12:39:01.775332 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770240 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 12:39:01.775332 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770245 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 12:39:01.775332 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770248 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 12:39:01.775332 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770251 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 12:39:01.775332 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770253 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 12:39:01.775332 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770255 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 16 12:39:01.775332 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770258 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 12:39:01.775332 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770260 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 12:39:01.775332 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770263 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 12:39:01.775332 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770265 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 12:39:01.775332 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770267 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 12:39:01.775332 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770269 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 12:39:01.775826 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770336 2582 flags.go:64] FLAG: --address="0.0.0.0" Jul 16 12:39:01.775826 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770343 2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 16 12:39:01.775826 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770348 2582 flags.go:64] FLAG: --anonymous-auth="true" Jul 16 12:39:01.775826 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770352 2582 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 16 12:39:01.775826 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770379 2582 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 16 12:39:01.775826 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770400 2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 16 12:39:01.775826 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770404 2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 16 12:39:01.775826 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770408 2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 16 12:39:01.775826 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770411 2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 16 12:39:01.775826 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770414 2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 16 12:39:01.775826 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770417 2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 16 12:39:01.775826 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770420 2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 16 12:39:01.775826 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770424 2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 16 12:39:01.775826 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770426 2582 flags.go:64] FLAG: --cgroup-root="" Jul 16 12:39:01.775826 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770429 2582 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 16 12:39:01.775826 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770432 2582 flags.go:64] FLAG: --client-ca-file="" Jul 16 12:39:01.775826 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770435 2582 flags.go:64] FLAG: --cloud-provider="external" Jul 16 12:39:01.775826 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770437 2582 flags.go:64] FLAG: --cluster-dns="[]" Jul 16 12:39:01.775826 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770442 2582 flags.go:64] FLAG: --cluster-domain="" Jul 16 12:39:01.775826 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770445 2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 16 12:39:01.775826 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770448 2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 16 12:39:01.775826 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770452 2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 16 12:39:01.775826 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770455 2582 flags.go:64] FLAG: --container-log-max-files="5" Jul 16 12:39:01.775826 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770459 2582 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 16 12:39:01.776435 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770463 2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 16 12:39:01.776435 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770466 2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 16 12:39:01.776435 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770469 2582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 16 12:39:01.776435 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770472 2582 flags.go:64] FLAG: --contention-profiling="false" Jul 16 12:39:01.776435 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770475 2582 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 16 12:39:01.776435 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770477 2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 16 12:39:01.776435 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770480 2582 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 16 12:39:01.776435 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770483 2582 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 16 12:39:01.776435 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770487 2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 16 12:39:01.776435 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770489 2582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 16 12:39:01.776435 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770492 2582 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 16 12:39:01.776435 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770495 2582 flags.go:64] FLAG: --enable-load-reader="false" Jul 16 12:39:01.776435 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770497 2582 flags.go:64] FLAG: --enable-server="true" Jul 16 12:39:01.776435 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770501 2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 16 12:39:01.776435 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770508 2582 flags.go:64] FLAG: --event-burst="100" Jul 16 12:39:01.776435 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770511 2582 flags.go:64] FLAG: --event-qps="50" Jul 16 12:39:01.776435 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770514 2582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 16 12:39:01.776435 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770517 2582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 16 12:39:01.776435 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770520 2582 flags.go:64] FLAG: --eviction-hard="" Jul 16 12:39:01.776435 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770523 2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 16 12:39:01.776435 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770526 2582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 16 12:39:01.776435 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770528 2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 16 12:39:01.776435 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770531 2582 flags.go:64] FLAG: --eviction-soft="" Jul 16 12:39:01.776435 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770534 2582 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 16 12:39:01.776435 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770537 2582 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 16 12:39:01.777085 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770539 2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 16 12:39:01.777085 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770542 2582 flags.go:64] FLAG: --experimental-mounter-path="" Jul 16 12:39:01.777085 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770545 2582 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 16 12:39:01.777085 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770548 2582 flags.go:64] FLAG: --fail-swap-on="true" Jul 16 12:39:01.777085 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770551 2582 flags.go:64] FLAG: --feature-gates="" Jul 16 12:39:01.777085 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770559 2582 flags.go:64] FLAG: --file-check-frequency="20s" Jul 16 12:39:01.777085 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770562 2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 16 12:39:01.777085 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770565 2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 16 12:39:01.777085 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770568 2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 16 12:39:01.777085 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770570 2582 flags.go:64] FLAG: --healthz-port="10248" Jul 16 12:39:01.777085 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770573 2582 flags.go:64] FLAG: --help="false" Jul 16 12:39:01.777085 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770576 2582 flags.go:64] FLAG: --hostname-override="ip-10-0-140-130.ec2.internal" Jul 16 12:39:01.777085 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770579 2582 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 16 12:39:01.777085 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770582 2582 flags.go:64] FLAG: --http-check-frequency="20s" Jul 16 12:39:01.777085 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770585 2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 16 12:39:01.777085 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770588 2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 16 12:39:01.777085 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770591 2582 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 16 12:39:01.777085 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770594 2582 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 16 12:39:01.777085 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770596 2582 flags.go:64] FLAG: --image-service-endpoint="" Jul 16 12:39:01.777085 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770599 2582 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 16 12:39:01.777085 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770602 2582 flags.go:64] FLAG: --kube-api-burst="100" Jul 16 12:39:01.777085 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770605 2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 16 12:39:01.777085 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770610 2582 flags.go:64] FLAG: --kube-api-qps="50" Jul 16 12:39:01.777085 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770612 2582 flags.go:64] FLAG: --kube-reserved="" Jul 16 12:39:01.777705 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770615 2582 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 16 12:39:01.777705 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770618 2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 16 12:39:01.777705 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770621 2582 flags.go:64] FLAG: --kubelet-cgroups="" Jul 16 12:39:01.777705 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770624 2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 16 12:39:01.777705 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770627 2582 flags.go:64] FLAG: --lock-file="" Jul 16 12:39:01.777705 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770630 2582 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 16 12:39:01.777705 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770632 2582 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 16 12:39:01.777705 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770635 2582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 16 12:39:01.777705 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770641 2582 flags.go:64] FLAG: --log-json-split-stream="false" Jul 16 12:39:01.777705 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770643 2582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 16 12:39:01.777705 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770646 2582 flags.go:64] FLAG: --log-text-split-stream="false" Jul 16 12:39:01.777705 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770649 2582 flags.go:64] FLAG: --logging-format="text" Jul 16 12:39:01.777705 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770653 2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 16 12:39:01.777705 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770656 2582 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 16 12:39:01.777705 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770659 2582 flags.go:64] FLAG: --manifest-url="" Jul 16 12:39:01.777705 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770662 2582 flags.go:64] FLAG: --manifest-url-header="" Jul 16 12:39:01.777705 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770666 2582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 16 12:39:01.777705 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770669 2582 flags.go:64] FLAG: --max-open-files="1000000" Jul 16 12:39:01.777705 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770673 2582 flags.go:64] FLAG: --max-pods="110" Jul 16 12:39:01.777705 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770675 2582 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 16 12:39:01.777705 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770678 2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 16 12:39:01.777705 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770681 2582 flags.go:64] FLAG: --memory-manager-policy="None" Jul 16 12:39:01.777705 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770684 2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 16 12:39:01.777705 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770687 2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 16 12:39:01.777705 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770689 2582 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 16 12:39:01.778311 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770692 2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 16 12:39:01.778311 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770698 2582 flags.go:64] FLAG: --node-status-max-images="50" Jul 16 12:39:01.778311 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770701 2582 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 16 12:39:01.778311 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770704 2582 flags.go:64] FLAG: --oom-score-adj="-999" Jul 16 12:39:01.778311 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770707 2582 flags.go:64] FLAG: --pod-cidr="" Jul 16 12:39:01.778311 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770711 2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 16 12:39:01.778311 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770717 2582 flags.go:64] FLAG: --pod-manifest-path="" Jul 16 12:39:01.778311 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770719 2582 flags.go:64] FLAG: --pod-max-pids="-1" Jul 16 12:39:01.778311 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770722 2582 flags.go:64] FLAG: --pods-per-core="0" Jul 16 12:39:01.778311 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770725 2582 flags.go:64] FLAG: --port="10250" Jul 16 12:39:01.778311 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770728 2582 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 16 12:39:01.778311 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770731 2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0533825078e3a419d" Jul 16 12:39:01.778311 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770734 2582 flags.go:64] FLAG: --qos-reserved="" Jul 16 12:39:01.778311 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770737 2582 flags.go:64] FLAG: --read-only-port="10255" Jul 16 12:39:01.778311 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770740 2582 flags.go:64] FLAG: --register-node="true" Jul 16 12:39:01.778311 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770743 2582 flags.go:64] FLAG: --register-with-taints="" Jul 16 12:39:01.778311 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770746 2582 flags.go:64] FLAG: --registry-burst="10" Jul 16 12:39:01.778311 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770748 2582 flags.go:64] FLAG: --registry-qps="5" Jul 16 12:39:01.778311 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770751 2582 flags.go:64] FLAG: --reserved-cpus="" Jul 16 12:39:01.778311 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770755 2582 flags.go:64] FLAG: --reserved-memory="" Jul 16 12:39:01.778311 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770759 2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 16 12:39:01.778311 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770761 2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 16 12:39:01.778311 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770764 2582 flags.go:64] FLAG: --rotate-certificates="false" Jul 16 12:39:01.778311 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770767 2582 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 16 12:39:01.778311 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770770 2582 flags.go:64] FLAG: --runonce="false" Jul 16 12:39:01.778929 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770773 2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 16 12:39:01.778929 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770775 2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 16 12:39:01.778929 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770778 2582 flags.go:64] FLAG: --seccomp-default="false" Jul 16 12:39:01.778929 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770781 2582 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 16 12:39:01.778929 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770784 2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 16 12:39:01.778929 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770787 2582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 16 12:39:01.778929 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770790 2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 16 12:39:01.778929 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770792 2582 flags.go:64] FLAG: --storage-driver-password="root" Jul 16 12:39:01.778929 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770796 2582 flags.go:64] FLAG: --storage-driver-secure="false" Jul 16 12:39:01.778929 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770799 2582 flags.go:64] FLAG: --storage-driver-table="stats" Jul 16 12:39:01.778929 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770802 2582 flags.go:64] FLAG: --storage-driver-user="root" Jul 16 12:39:01.778929 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770804 2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 16 12:39:01.778929 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770807 2582 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 16 12:39:01.778929 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770812 2582 flags.go:64] FLAG: --system-cgroups="" Jul 16 12:39:01.778929 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770815 2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 16 12:39:01.778929 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770820 2582 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 16 12:39:01.778929 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770823 2582 flags.go:64] FLAG: --tls-cert-file="" Jul 16 12:39:01.778929 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770825 2582 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 16 12:39:01.778929 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770829 2582 flags.go:64] FLAG: --tls-min-version="" Jul 16 12:39:01.778929 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770832 2582 flags.go:64] FLAG: --tls-private-key-file="" Jul 16 12:39:01.778929 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770835 2582 flags.go:64] FLAG: --topology-manager-policy="none" Jul 16 12:39:01.778929 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770837 2582 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 16 12:39:01.778929 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770840 2582 flags.go:64] FLAG: --topology-manager-scope="container" Jul 16 12:39:01.778929 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770843 2582 flags.go:64] FLAG: --v="2" Jul 16 12:39:01.778929 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770847 2582 flags.go:64] FLAG: --version="false" Jul 16 12:39:01.779600 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770850 2582 flags.go:64] FLAG: --vmodule="" Jul 16 12:39:01.779600 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770855 2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 16 12:39:01.779600 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.770858 2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 16 12:39:01.779600 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770988 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 12:39:01.779600 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770993 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 12:39:01.779600 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770996 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 12:39:01.779600 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.770999 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 12:39:01.779600 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771001 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 12:39:01.779600 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771004 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 12:39:01.779600 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771006 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 12:39:01.779600 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771009 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 12:39:01.779600 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771011 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 12:39:01.779600 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771014 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 12:39:01.779600 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771017 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 12:39:01.779600 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771019 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 12:39:01.779600 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771022 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 12:39:01.779600 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771024 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 12:39:01.779600 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771027 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 12:39:01.779600 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771029 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 12:39:01.779600 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771032 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 12:39:01.780085 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771036 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 16 12:39:01.780085 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771039 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 12:39:01.780085 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771041 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 12:39:01.780085 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771044 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 12:39:01.780085 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771046 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 12:39:01.780085 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771050 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 12:39:01.780085 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771054 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 12:39:01.780085 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771056 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 12:39:01.780085 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771058 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 12:39:01.780085 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771061 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 12:39:01.780085 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771064 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 12:39:01.780085 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771066 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 12:39:01.780085 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771068 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 12:39:01.780085 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771072 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 12:39:01.780085 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771074 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 12:39:01.780085 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771076 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 12:39:01.780085 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771079 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 12:39:01.780085 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771081 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 12:39:01.780085 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771084 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 12:39:01.780561 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771086 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 12:39:01.780561 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771090 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 12:39:01.780561 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771093 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 12:39:01.780561 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771096 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 12:39:01.780561 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771099 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 12:39:01.780561 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771102 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 12:39:01.780561 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771105 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 12:39:01.780561 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771108 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 12:39:01.780561 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771111 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 12:39:01.780561 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771113 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 12:39:01.780561 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771116 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 12:39:01.780561 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771119 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 12:39:01.780561 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771121 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 12:39:01.780561 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771125 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 12:39:01.780561 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771127 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 12:39:01.780561 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771136 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 12:39:01.780561 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771138 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 12:39:01.780561 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771141 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 12:39:01.780561 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771143 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 12:39:01.781023 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771146 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 12:39:01.781023 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771148 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 12:39:01.781023 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771152 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 12:39:01.781023 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771155 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 12:39:01.781023 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771158 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 12:39:01.781023 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771160 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 12:39:01.781023 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771163 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 12:39:01.781023 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771166 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 12:39:01.781023 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771168 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 12:39:01.781023 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771172 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 12:39:01.781023 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771174 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 12:39:01.781023 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771177 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 12:39:01.781023 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771179 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 12:39:01.781023 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771181 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 12:39:01.781023 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771184 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 12:39:01.781023 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771186 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 12:39:01.781023 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771189 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 12:39:01.781023 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771191 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 12:39:01.781023 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771193 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 12:39:01.781023 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771196 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 12:39:01.781535 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771198 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 12:39:01.781535 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771201 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 12:39:01.781535 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771204 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 12:39:01.781535 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771206 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 12:39:01.781535 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771208 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 12:39:01.781535 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771211 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 12:39:01.781535 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771215 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 12:39:01.781535 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771217 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 12:39:01.781535 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771220 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 12:39:01.781535 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771223 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 12:39:01.781535 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771226 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 12:39:01.781535 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771228 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 12:39:01.781535 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771231 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 12:39:01.781535 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771233 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 12:39:01.781535 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771235 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 12:39:01.781535 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771238 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 12:39:01.781535 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771241 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 12:39:01.781535 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771243 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 12:39:01.781535 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771246 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 12:39:01.781535 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771248 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 12:39:01.782026 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771251 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 12:39:01.782026 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771253 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 12:39:01.782026 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771256 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 12:39:01.782026 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.771258 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 12:39:01.782026 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.771744 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 12:39:01.782026 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.778309 2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 16 12:39:01.782026 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.778327 2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 16 12:39:01.782026 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.779055 2582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 16 12:39:01.782026 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.779065 2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 16 12:39:01.782026 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779129 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 12:39:01.782026 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779133 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 12:39:01.782026 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779137 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 12:39:01.782026 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779140 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 12:39:01.782026 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779143 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 12:39:01.782383 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779147 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 12:39:01.782383 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779151 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 12:39:01.782383 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779154 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 12:39:01.782383 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779157 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 12:39:01.782383 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779160 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 12:39:01.782383 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779163 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 12:39:01.782383 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779166 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 12:39:01.782383 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779169 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 12:39:01.782383 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779171 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 12:39:01.782383 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779174 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 12:39:01.782383 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779176 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 12:39:01.782383 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779178 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 12:39:01.782383 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779181 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 12:39:01.782383 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779183 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 12:39:01.782383 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779186 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 12:39:01.782383 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779188 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 12:39:01.782383 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779191 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 12:39:01.782383 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779193 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 12:39:01.782383 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779196 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 12:39:01.783093 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779198 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 12:39:01.783093 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779201 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 12:39:01.783093 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779203 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 12:39:01.783093 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779213 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 12:39:01.783093 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779216 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 12:39:01.783093 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779224 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 12:39:01.783093 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779227 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 12:39:01.783093 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779229 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 12:39:01.783093 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779231 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 12:39:01.783093 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779233 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 12:39:01.783093 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779236 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 12:39:01.783093 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779240 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 12:39:01.783093 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779243 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 12:39:01.783093 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779246 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 12:39:01.783093 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779248 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 12:39:01.783093 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779251 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 12:39:01.783093 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779253 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 12:39:01.783093 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779255 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 12:39:01.783093 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779258 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 12:39:01.783093 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779260 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 12:39:01.783927 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779263 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 12:39:01.783927 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779265 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 12:39:01.783927 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779268 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 12:39:01.783927 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779270 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 12:39:01.783927 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779273 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 12:39:01.783927 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779275 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 12:39:01.783927 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779277 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 12:39:01.783927 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779280 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 12:39:01.783927 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779282 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 12:39:01.783927 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779285 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 12:39:01.783927 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779287 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 12:39:01.783927 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779290 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 12:39:01.783927 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779292 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 12:39:01.783927 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779295 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 12:39:01.783927 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779297 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 12:39:01.783927 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779299 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 12:39:01.783927 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779302 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 12:39:01.783927 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779304 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 12:39:01.783927 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779313 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 12:39:01.783927 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779316 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 12:39:01.784562 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779318 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 16 12:39:01.784562 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779320 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 12:39:01.784562 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779322 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 12:39:01.784562 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779325 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 12:39:01.784562 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779327 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 12:39:01.784562 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779329 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 12:39:01.784562 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779332 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 12:39:01.784562 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779334 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 12:39:01.784562 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779337 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 12:39:01.784562 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779339 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 12:39:01.784562 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779341 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 12:39:01.784562 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779344 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 12:39:01.784562 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779346 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 12:39:01.784562 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779348 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 12:39:01.784562 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779351 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 12:39:01.784562 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779353 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 12:39:01.784562 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779355 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 12:39:01.784562 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779358 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 12:39:01.784562 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779360 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 12:39:01.784562 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779362 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 12:39:01.785104 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779365 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 12:39:01.785104 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779367 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 12:39:01.785104 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779370 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 12:39:01.785104 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779372 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 12:39:01.785104 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779374 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 12:39:01.785104 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779377 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 12:39:01.785104 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779379 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 12:39:01.785104 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779381 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 12:39:01.785104 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779401 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 12:39:01.785104 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779406 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 12:39:01.785104 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779409 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 12:39:01.785104 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779417 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 12:39:01.785104 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779421 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 12:39:01.785104 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779423 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 12:39:01.785104 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779425 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 12:39:01.785582 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.779430 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 12:39:01.785582 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779573 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 12:39:01.785582 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779579 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 12:39:01.785582 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779583 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 12:39:01.785582 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779586 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 12:39:01.785582 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779588 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 12:39:01.785582 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779591 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 12:39:01.785582 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779594 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 12:39:01.785582 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779597 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 12:39:01.785582 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779599 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 12:39:01.785582 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779602 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 12:39:01.785582 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779605 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 12:39:01.785582 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779608 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 12:39:01.785582 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779610 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 12:39:01.785582 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779613 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 12:39:01.785947 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779615 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 12:39:01.785947 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779618 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 12:39:01.785947 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779620 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 12:39:01.785947 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779622 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 12:39:01.785947 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779625 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 12:39:01.785947 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779627 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 12:39:01.785947 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779630 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 12:39:01.785947 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779632 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 12:39:01.785947 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779634 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 12:39:01.785947 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779637 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 12:39:01.785947 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779639 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 12:39:01.785947 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779641 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 12:39:01.785947 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779643 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 12:39:01.785947 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779646 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 12:39:01.785947 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779654 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 12:39:01.785947 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779656 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 12:39:01.785947 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779658 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 12:39:01.785947 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779661 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 12:39:01.785947 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779663 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 12:39:01.785947 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779666 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 12:39:01.786664 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779668 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 12:39:01.786664 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779671 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 12:39:01.786664 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779673 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 12:39:01.786664 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779675 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 12:39:01.786664 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779678 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 12:39:01.786664 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779680 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 12:39:01.786664 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779683 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 12:39:01.786664 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779685 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 12:39:01.786664 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779687 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 12:39:01.786664 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779690 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 12:39:01.786664 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779693 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 12:39:01.786664 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779696 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 12:39:01.786664 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779699 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 12:39:01.786664 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779701 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 12:39:01.786664 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779704 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 12:39:01.786664 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779706 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 12:39:01.786664 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779708 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 12:39:01.786664 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779711 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 12:39:01.786664 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779713 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 12:39:01.787439 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779715 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 12:39:01.787439 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779718 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 12:39:01.787439 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779720 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 12:39:01.787439 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779722 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 12:39:01.787439 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779724 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 12:39:01.787439 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779727 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 12:39:01.787439 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779729 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 12:39:01.787439 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779732 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 12:39:01.787439 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779739 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 12:39:01.787439 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779742 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 12:39:01.787439 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779744 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 12:39:01.787439 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779747 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 12:39:01.787439 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779749 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 12:39:01.787439 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779751 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 16 12:39:01.787439 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779753 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 12:39:01.787439 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779756 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 12:39:01.787439 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779758 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 12:39:01.787439 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779761 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 12:39:01.787439 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779763 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 12:39:01.787439 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779765 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 12:39:01.787934 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779768 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 12:39:01.787934 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779770 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 12:39:01.787934 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779773 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 12:39:01.787934 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779775 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 12:39:01.787934 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779777 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 12:39:01.787934 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779779 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 12:39:01.787934 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779782 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 12:39:01.787934 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779784 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 12:39:01.787934 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779787 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 12:39:01.787934 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779789 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 12:39:01.787934 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779792 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 12:39:01.787934 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779794 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 12:39:01.787934 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779797 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 12:39:01.787934 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779799 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 12:39:01.787934 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779801 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 12:39:01.787934 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779803 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 12:39:01.787934 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779807 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 12:39:01.787934 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779810 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 12:39:01.787934 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779813 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 12:39:01.788636 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779816 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 12:39:01.788636 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779819 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 12:39:01.788636 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779826 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 12:39:01.788636 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779832 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 12:39:01.788636 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779834 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 12:39:01.788636 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779837 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 12:39:01.788636 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:01.779839 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 12:39:01.788636 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.779843 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 12:39:01.788636 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.779944 2582 server.go:962] "Client rotation is on, will bootstrap in background" Jul 16 12:39:01.788636 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.782449 2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 16 12:39:01.788636 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.785065 2582 server.go:1019] "Starting client certificate rotation" Jul 16 12:39:01.788636 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.785185 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 12:39:01.788636 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.785244 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 12:39:01.788636 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.787426 2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 12:39:01.789306 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.789290 2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 12:39:01.798279 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.798263 2582 log.go:25] "Validated CRI v1 runtime API" Jul 16 12:39:01.801297 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.801282 2582 log.go:25] "Validated CRI v1 image API" Jul 16 12:39:01.803278 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.803262 2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 16 12:39:01.805326 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.805307 2582 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9a9328ca-937f-4b11-a9f0-e9751cb5c708:/dev/nvme0n1p4 ba265ac9-ab8f-4003-977f-56ccedcfa154:/dev/nvme0n1p3] Jul 16 12:39:01.805420 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.805324 2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 16 12:39:01.811709 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.811562 2582 manager.go:216] Machine: {Timestamp:2026-07-16 12:39:01.809373093 +0000 UTC m=+0.287118719 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099232 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2797cbf5d1271c467b270ad8b9465a SystemUUID:ec2797cb-f5d1-271c-467b-270ad8b9465a BootID:54d70e84-e00b-4c7d-9da3-1caf5f9abce8 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:72:9c:de:86:c7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:72:9c:de:86:c7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:44:86:a8:68:7f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 16 12:39:01.811709 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.811697 2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 16 12:39:01.811838 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.811798 2582 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 16 12:39:01.812735 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.812714 2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 16 12:39:01.812877 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.812738 2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-130.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 16 12:39:01.813445 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.813434 2582 topology_manager.go:138] "Creating topology manager with none policy" Jul 16 12:39:01.813475 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.813449 2582 container_manager_linux.go:309] "Creating device plugin manager" Jul 16 12:39:01.813475 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.813456 2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 12:39:01.814164 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.814155 2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 12:39:01.814612 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.814596 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 12:39:01.815126 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.815115 2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 16 12:39:01.817946 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.817931 2582 state_mem.go:36] "Initialized new in-memory state store" Jul 16 12:39:01.818062 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.818052 2582 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 16 12:39:01.818104 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.818083 2582 kubelet.go:486] "Attempting to sync node with API server" Jul 16 12:39:01.818104 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.818092 2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 16 12:39:01.818104 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.818103 2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 16 12:39:01.818184 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.818111 2582 kubelet.go:398] "Adding apiserver pod source" Jul 16 12:39:01.818184 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.818119 2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 16 12:39:01.818598 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.818580 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 12:39:01.818699 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.818610 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 12:39:01.821370 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.821327 2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 16 12:39:01.823035 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.823012 2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 16 12:39:01.823111 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.823045 2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 16 12:39:01.823634 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.823622 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 16 12:39:01.823684 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.823638 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 16 12:39:01.823684 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.823651 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 16 12:39:01.823684 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.823657 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 16 12:39:01.823684 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.823665 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 16 12:39:01.823684 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.823670 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 16 12:39:01.823684 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.823679 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 16 12:39:01.823684 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.823685 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 16 12:39:01.823977 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.823691 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 16 12:39:01.823977 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.823697 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 16 12:39:01.823977 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.823705 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 16 12:39:01.823977 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.823710 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 16 12:39:01.824376 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.824366 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 16 12:39:01.824376 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.824376 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 16 12:39:01.827536 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.827513 2582 server.go:1268] "Started kubelet" Jul 16 12:39:01.827645 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.827596 2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 16 12:39:01.827889 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.827845 2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 16 12:39:01.827941 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.827900 2582 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 16 12:39:01.828361 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.828344 2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-130.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 16 12:39:01.828452 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:01.828431 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 16 12:39:01.828502 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:01.828479 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-130.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 16 12:39:01.829410 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.828761 2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 16 12:39:01.829039 ip-10-0-140-130 systemd[1]: Started Kubernetes Kubelet. Jul 16 12:39:01.830421 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.830375 2582 server.go:310] "Adding debug handlers to kubelet server" Jul 16 12:39:01.833075 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.833058 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 16 12:39:01.833154 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.833072 2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 16 12:39:01.833215 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:01.833164 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-130.ec2.internal\" not found" Jul 16 12:39:01.834078 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.834060 2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 16 12:39:01.834078 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.834053 2582 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 16 12:39:01.834203 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.834087 2582 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 16 12:39:01.834203 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.834107 2582 factory.go:55] Registering systemd factory Jul 16 12:39:01.834203 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.834120 2582 factory.go:223] Registration of the systemd container factory successfully Jul 16 12:39:01.834203 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.834182 2582 reconstruct.go:97] "Volume reconstruction finished" Jul 16 12:39:01.834203 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.834195 2582 reconciler.go:29] "Reconciler: start to sync state" Jul 16 12:39:01.834441 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:01.834253 2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 16 12:39:01.834441 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.834345 2582 factory.go:153] Registering CRI-O factory Jul 16 12:39:01.834441 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.834364 2582 factory.go:223] Registration of the crio container factory successfully Jul 16 12:39:01.834441 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.834426 2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 16 12:39:01.834441 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.834444 2582 factory.go:103] Registering Raw factory Jul 16 12:39:01.834663 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.834454 2582 manager.go:1196] Started watching for new ooms in manager Jul 16 12:39:01.835134 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.835122 2582 manager.go:349] Starting recovery of all containers Jul 16 12:39:01.836328 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:01.836305 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 16 12:39:01.836551 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:01.836532 2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-130.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 16 12:39:01.837173 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:01.836258 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-130.ec2.internal.18c2c57cb6e926e5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-130.ec2.internal,UID:ip-10-0-140-130.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-130.ec2.internal,},FirstTimestamp:2026-07-16 12:39:01.827487461 +0000 UTC m=+0.305233067,LastTimestamp:2026-07-16 12:39:01.827487461 +0000 UTC m=+0.305233067,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-130.ec2.internal,}" Jul 16 12:39:01.843077 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.842897 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 16 12:39:01.846543 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.846523 2582 manager.go:354] Recovery completed Jul 16 12:39:01.850988 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.850939 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:39:01.853966 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.853940 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:39:01.854050 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.853975 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:39:01.854050 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.853987 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:39:01.854938 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.854925 2582 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 16 12:39:01.854938 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.854930 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:39:01.854938 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.854938 2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 16 12:39:01.855056 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.854953 2582 state_mem.go:36] "Initialized new in-memory state store" Jul 16 12:39:01.855621 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.855604 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:39:01.855673 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.855630 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:39:01.855673 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.855640 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:39:01.855764 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:01.855556 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-130.ec2.internal.18c2c57cb87d0a45 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-130.ec2.internal,UID:ip-10-0-140-130.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-130.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-130.ec2.internal,},FirstTimestamp:2026-07-16 12:39:01.853956677 +0000 UTC m=+0.331702284,LastTimestamp:2026-07-16 12:39:01.853956677 +0000 UTC m=+0.331702284,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-130.ec2.internal,}" Jul 16 12:39:01.856258 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.856224 2582 policy_none.go:49] "None policy: Start" Jul 16 12:39:01.856258 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.856257 2582 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 16 12:39:01.856327 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.856267 2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 16 12:39:01.857499 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.857489 2582 policy_none.go:47] "Start" Jul 16 12:39:01.866342 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:01.866287 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-130.ec2.internal.18c2c57cb87d696c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-130.ec2.internal,UID:ip-10-0-140-130.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-130.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-130.ec2.internal,},FirstTimestamp:2026-07-16 12:39:01.853981036 +0000 UTC m=+0.331726642,LastTimestamp:2026-07-16 12:39:01.853981036 +0000 UTC m=+0.331726642,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-130.ec2.internal,}" Jul 16 12:39:01.870660 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.870641 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c94rs" Jul 16 12:39:01.876781 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:01.876672 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-130.ec2.internal.18c2c57cb87d91d5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-130.ec2.internal,UID:ip-10-0-140-130.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-130.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-130.ec2.internal,},FirstTimestamp:2026-07-16 12:39:01.853991381 +0000 UTC m=+0.331736989,LastTimestamp:2026-07-16 12:39:01.853991381 +0000 UTC m=+0.331736989,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-130.ec2.internal,}" Jul 16 12:39:01.878105 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.878084 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c94rs" Jul 16 12:39:01.889356 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.889341 2582 manager.go:337] "Starting Device Plugin manager" Jul 16 12:39:01.889449 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:01.889369 2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 16 12:39:01.889449 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.889379 2582 server.go:87] "Starting device plugin registration server" Jul 16 12:39:01.889626 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.889612 2582 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 16 12:39:01.889690 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.889629 2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 16 12:39:01.889746 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.889722 2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 16 12:39:01.889814 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.889798 2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 16 12:39:01.889814 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.889811 2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 16 12:39:01.890252 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:01.890232 2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 16 12:39:01.890310 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:01.890289 2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-130.ec2.internal\" not found" Jul 16 12:39:01.971045 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.971017 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 16 12:39:01.972207 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.971061 2582 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 16 12:39:01.972207 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.971082 2582 kubelet.go:2448] "Starting kubelet main sync loop" Jul 16 12:39:01.972207 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:01.971120 2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 16 12:39:01.973249 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.973235 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 12:39:01.990321 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.990301 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:39:01.991598 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.991584 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:39:01.991648 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.991614 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:39:01.991648 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.991624 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:39:01.991648 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:01.991645 2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-130.ec2.internal" Jul 16 12:39:02.001578 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.001565 2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-130.ec2.internal" Jul 16 12:39:02.001625 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:02.001585 2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-130.ec2.internal\": node \"ip-10-0-140-130.ec2.internal\" not found" Jul 16 12:39:02.042339 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:02.042280 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-130.ec2.internal\" not found" Jul 16 12:39:02.071478 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.071450 2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-130.ec2.internal"] Jul 16 12:39:02.071595 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.071521 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:39:02.072500 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.072486 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:39:02.072571 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.072513 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:39:02.072571 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.072523 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:39:02.074106 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.074092 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:39:02.074257 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.074243 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal" Jul 16 12:39:02.074318 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.074276 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:39:02.074829 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.074812 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:39:02.074829 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.074824 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:39:02.074969 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.074840 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:39:02.074969 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.074848 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:39:02.074969 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.074851 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:39:02.074969 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.074872 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:39:02.076536 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.076523 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-130.ec2.internal" Jul 16 12:39:02.076592 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.076545 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:39:02.077134 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.077119 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:39:02.077225 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.077160 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:39:02.077225 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.077175 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:39:02.104964 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:02.104944 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-130.ec2.internal\" not found" node="ip-10-0-140-130.ec2.internal" Jul 16 12:39:02.109530 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:02.109516 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-130.ec2.internal\" not found" node="ip-10-0-140-130.ec2.internal" Jul 16 12:39:02.136371 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.136343 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/eec32a5cfb1937ffec65aa300c58f36d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal\" (UID: \"eec32a5cfb1937ffec65aa300c58f36d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal" Jul 16 12:39:02.136488 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.136380 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eec32a5cfb1937ffec65aa300c58f36d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal\" (UID: \"eec32a5cfb1937ffec65aa300c58f36d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal" Jul 16 12:39:02.136488 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.136445 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/42e2219b6620be69c895eb04c7d0d0b4-config\") pod \"kube-apiserver-proxy-ip-10-0-140-130.ec2.internal\" (UID: \"42e2219b6620be69c895eb04c7d0d0b4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-130.ec2.internal" Jul 16 12:39:02.142460 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:02.142445 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-130.ec2.internal\" not found" Jul 16 12:39:02.237109 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.237080 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/eec32a5cfb1937ffec65aa300c58f36d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal\" (UID: \"eec32a5cfb1937ffec65aa300c58f36d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal" Jul 16 12:39:02.237199 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.237112 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eec32a5cfb1937ffec65aa300c58f36d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal\" (UID: \"eec32a5cfb1937ffec65aa300c58f36d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal" Jul 16 12:39:02.237199 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.237131 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/42e2219b6620be69c895eb04c7d0d0b4-config\") pod \"kube-apiserver-proxy-ip-10-0-140-130.ec2.internal\" (UID: \"42e2219b6620be69c895eb04c7d0d0b4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-130.ec2.internal" Jul 16 12:39:02.237199 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.237168 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/eec32a5cfb1937ffec65aa300c58f36d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal\" (UID: \"eec32a5cfb1937ffec65aa300c58f36d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal" Jul 16 12:39:02.237199 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.237181 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eec32a5cfb1937ffec65aa300c58f36d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal\" (UID: \"eec32a5cfb1937ffec65aa300c58f36d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal" Jul 16 12:39:02.237337 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.237183 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/42e2219b6620be69c895eb04c7d0d0b4-config\") pod \"kube-apiserver-proxy-ip-10-0-140-130.ec2.internal\" (UID: \"42e2219b6620be69c895eb04c7d0d0b4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-130.ec2.internal" Jul 16 12:39:02.243183 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:02.243165 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-130.ec2.internal\" not found" Jul 16 12:39:02.344077 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:02.343992 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-130.ec2.internal\" not found" Jul 16 12:39:02.408235 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.408197 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal" Jul 16 12:39:02.412907 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.412758 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-130.ec2.internal" Jul 16 12:39:02.444551 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:02.444522 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-130.ec2.internal\" not found" Jul 16 12:39:02.545121 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:02.545082 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-130.ec2.internal\" not found" Jul 16 12:39:02.645993 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:02.645966 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-130.ec2.internal\" not found" Jul 16 12:39:02.746757 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:02.746723 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-130.ec2.internal\" not found" Jul 16 12:39:02.786933 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.786902 2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 16 12:39:02.787467 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.787063 2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 12:39:02.846836 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:02.846808 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-130.ec2.internal\" not found" Jul 16 12:39:02.883463 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.883379 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 12:34:02 +0000 UTC" deadline="2028-01-10 16:47:36.600676153 +0000 UTC" Jul 16 12:39:02.883463 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.883426 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13036h8m33.71725343s" Jul 16 12:39:02.904250 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.904230 2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 12:39:02.935488 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.935464 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal" Jul 16 12:39:02.949360 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.949338 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 12:39:02.951398 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.951384 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-130.ec2.internal" Jul 16 12:39:02.960011 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:02.959998 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 12:39:03.175602 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.175576 2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 12:39:03.234346 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:03.234302 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42e2219b6620be69c895eb04c7d0d0b4.slice/crio-6c8974ba47a940dd5b9ae0f02064e9ee204996d66c5ff4c255a2af514eff2688 WatchSource:0}: Error finding container 6c8974ba47a940dd5b9ae0f02064e9ee204996d66c5ff4c255a2af514eff2688: Status 404 returned error can't find the container with id 6c8974ba47a940dd5b9ae0f02064e9ee204996d66c5ff4c255a2af514eff2688 Jul 16 12:39:03.238613 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.238594 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 12:39:03.286090 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.286063 2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 12:39:03.294850 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:03.294824 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeec32a5cfb1937ffec65aa300c58f36d.slice/crio-db61a6e49c5dda269b8f8ff9671f5244f2034108f0dbacd472f7fa9d415559c8 WatchSource:0}: Error finding container db61a6e49c5dda269b8f8ff9671f5244f2034108f0dbacd472f7fa9d415559c8: Status 404 returned error can't find the container with id db61a6e49c5dda269b8f8ff9671f5244f2034108f0dbacd472f7fa9d415559c8 Jul 16 12:39:03.821134 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.821094 2582 apiserver.go:52] "Watching apiserver" Jul 16 12:39:03.834242 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.834214 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 16 12:39:03.842992 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.842965 2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 16 12:39:03.843391 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.843363 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-986g4","kube-system/konnectivity-agent-4ns96","kube-system/kube-apiserver-proxy-ip-10-0-140-130.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gdt","openshift-dns/node-resolver-lv8x6","openshift-image-registry/node-ca-76lb5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal","openshift-multus/multus-5s6sz","openshift-multus/multus-additional-cni-plugins-m8pdn","kube-system/global-pull-secret-syncer-zd992","openshift-cluster-node-tuning-operator/tuned-dc8zj","openshift-multus/network-metrics-daemon-q5cp8","openshift-network-diagnostics/network-check-target-xgvpv","openshift-network-operator/iptables-alerter-mv2w2"] Jul 16 12:39:03.845200 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.845172 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:39:03.845290 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:03.845243 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd992" podUID="b9a976d1-7bb2-47d6-b285-788a6ff40cef" Jul 16 12:39:03.846182 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.846161 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgvpv" Jul 16 12:39:03.846271 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:03.846218 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgvpv" podUID="b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa" Jul 16 12:39:03.847259 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.847238 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mv2w2" Jul 16 12:39:03.848663 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.848643 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gdt" Jul 16 12:39:03.849916 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.849898 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lv8x6" Jul 16 12:39:03.851200 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.851183 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 16 12:39:03.851305 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.851185 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 16 12:39:03.851497 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.851477 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 16 12:39:03.851497 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.851488 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 16 12:39:03.852052 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.852029 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 16 12:39:03.852190 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.852077 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-76lb5" Jul 16 12:39:03.852190 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.852101 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 16 12:39:03.852190 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.852171 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rzhqh\"" Jul 16 12:39:03.853242 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.853225 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 16 12:39:03.853575 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.853558 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m8pdn" Jul 16 12:39:03.855880 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.855859 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5s6sz" Jul 16 12:39:03.856634 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.856617 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-p8kkx\"" Jul 16 12:39:03.856741 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.856675 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 16 12:39:03.856932 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.856915 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hmblm\"" Jul 16 12:39:03.857286 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.857270 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4ns96" Jul 16 12:39:03.858934 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.858912 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:03.862663 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.862642 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:03.864502 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.864445 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5cp8" Jul 16 12:39:03.864578 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:03.864545 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5cp8" podUID="fdd6ebe0-1750-4a12-a4c7-023cfadcbabc" Jul 16 12:39:03.869524 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.869499 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 16 12:39:03.869678 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.869658 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 16 12:39:03.869732 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.869691 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 16 12:39:03.869732 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.869724 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 16 12:39:03.869812 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.869747 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 16 12:39:03.870898 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.870875 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 16 12:39:03.871013 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.870875 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 16 12:39:03.871013 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.871004 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 16 12:39:03.871241 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.871014 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 16 12:39:03.871241 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.871128 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 16 12:39:03.871241 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.871131 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 16 12:39:03.871407 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.871304 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hdmlt\"" Jul 16 12:39:03.871487 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.871443 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 16 12:39:03.871618 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.871603 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 16 12:39:03.871688 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.871658 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 16 12:39:03.871907 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.871891 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2w4h5\"" Jul 16 12:39:03.871984 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.871924 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 16 12:39:03.871984 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.871956 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-55nf8\"" Jul 16 12:39:03.872095 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.871994 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 16 12:39:03.872095 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.872004 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gwvkk\"" Jul 16 12:39:03.872258 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.872240 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-stqqp\"" Jul 16 12:39:03.872327 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.872246 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 16 12:39:03.872388 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.872366 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 16 12:39:03.872447 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.872393 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bthhj\"" Jul 16 12:39:03.872529 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.872494 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 16 12:39:03.876398 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.876380 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 12:39:03.892506 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.892487 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wr2cr" Jul 16 12:39:03.901050 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.901031 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wr2cr" Jul 16 12:39:03.935030 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.935005 2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 16 12:39:03.946517 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.946482 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04c215b4-7f84-4763-9ad8-3db823a6c2c6-cnibin\") pod \"multus-additional-cni-plugins-m8pdn\" (UID: \"04c215b4-7f84-4763-9ad8-3db823a6c2c6\") " pod="openshift-multus/multus-additional-cni-plugins-m8pdn" Jul 16 12:39:03.946617 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.946529 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-host-run-k8s-cni-cncf-io\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:03.946617 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.946587 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b95cb0aa-837a-41a4-913c-7e3b15f04fdf-host\") pod \"node-ca-76lb5\" (UID: \"b95cb0aa-837a-41a4-913c-7e3b15f04fdf\") " pod="openshift-image-registry/node-ca-76lb5" Jul 16 12:39:03.946747 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.946623 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04c215b4-7f84-4763-9ad8-3db823a6c2c6-system-cni-dir\") pod \"multus-additional-cni-plugins-m8pdn\" (UID: \"04c215b4-7f84-4763-9ad8-3db823a6c2c6\") " pod="openshift-multus/multus-additional-cni-plugins-m8pdn" Jul 16 12:39:03.946747 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.946652 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-multus-socket-dir-parent\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:03.946747 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.946697 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45kfk\" (UniqueName: \"kubernetes.io/projected/b95cb0aa-837a-41a4-913c-7e3b15f04fdf-kube-api-access-45kfk\") pod \"node-ca-76lb5\" (UID: \"b95cb0aa-837a-41a4-913c-7e3b15f04fdf\") " pod="openshift-image-registry/node-ca-76lb5" Jul 16 12:39:03.946747 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.946737 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-var-lib-openvswitch\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:03.947361 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.946782 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/95be9185-aa73-4b64-9301-04e5ca0c661e-socket-dir\") pod \"aws-ebs-csi-driver-node-r6gdt\" (UID: \"95be9185-aa73-4b64-9301-04e5ca0c661e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gdt" Jul 16 12:39:03.947361 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.946803 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-host-run-netns\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:03.947361 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.946880 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9a976d1-7bb2-47d6-b285-788a6ff40cef-original-pull-secret\") pod \"global-pull-secret-syncer-zd992\" (UID: \"b9a976d1-7bb2-47d6-b285-788a6ff40cef\") " pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:39:03.947361 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.946938 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04c215b4-7f84-4763-9ad8-3db823a6c2c6-cni-binary-copy\") pod \"multus-additional-cni-plugins-m8pdn\" (UID: \"04c215b4-7f84-4763-9ad8-3db823a6c2c6\") " pod="openshift-multus/multus-additional-cni-plugins-m8pdn" Jul 16 12:39:03.947361 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.946982 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-cni-binary-copy\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:03.947361 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.947008 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-host-var-lib-cni-bin\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:03.947361 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.947048 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-etc-kubernetes\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:03.947361 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.947086 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-log-socket\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:03.947361 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.947113 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e332a21-d3fd-468d-a704-8a98f55f1596-tmp-dir\") pod \"node-resolver-lv8x6\" (UID: \"7e332a21-d3fd-468d-a704-8a98f55f1596\") " pod="openshift-dns/node-resolver-lv8x6" Jul 16 12:39:03.947361 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.947151 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhlsr\" (UniqueName: \"kubernetes.io/projected/7e332a21-d3fd-468d-a704-8a98f55f1596-kube-api-access-qhlsr\") pod \"node-resolver-lv8x6\" (UID: \"7e332a21-d3fd-468d-a704-8a98f55f1596\") " pod="openshift-dns/node-resolver-lv8x6" Jul 16 12:39:03.947361 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.947176 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-ovnkube-config\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:03.947361 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.947210 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-multus-daemon-config\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:03.947361 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.947265 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a80d1676-7c47-4427-992c-234fbc36cf7b-agent-certs\") pod \"konnectivity-agent-4ns96\" (UID: \"a80d1676-7c47-4427-992c-234fbc36cf7b\") " pod="kube-system/konnectivity-agent-4ns96" Jul 16 12:39:03.947361 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.947286 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-host-run-ovn-kubernetes\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:03.947361 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.947309 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-env-overrides\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:03.947361 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.947327 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-ovn-node-metrics-cert\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:03.948205 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.947344 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-etc-sysconfig\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:03.948205 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.947359 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-etc-sysctl-conf\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:03.948205 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.947374 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-run\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:03.948205 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.947397 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-os-release\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:03.948205 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.947415 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7e332a21-d3fd-468d-a704-8a98f55f1596-hosts-file\") pod \"node-resolver-lv8x6\" (UID: \"7e332a21-d3fd-468d-a704-8a98f55f1596\") " pod="openshift-dns/node-resolver-lv8x6" Jul 16 12:39:03.948205 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.947458 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-systemd-units\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:03.948205 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.947501 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-run-ovn\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:03.948205 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.947535 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-lib-modules\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:03.948205 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.947560 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/95be9185-aa73-4b64-9301-04e5ca0c661e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r6gdt\" (UID: \"95be9185-aa73-4b64-9301-04e5ca0c661e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gdt" Jul 16 12:39:03.948205 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.947587 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-etc-kubernetes\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:03.948205 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.947612 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-etc-tuned\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:03.948205 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.947674 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgvzh\" (UniqueName: \"kubernetes.io/projected/a30e0419-799d-45fb-a586-f06018bc1ee4-kube-api-access-sgvzh\") pod \"iptables-alerter-mv2w2\" (UID: \"a30e0419-799d-45fb-a586-f06018bc1ee4\") " pod="openshift-network-operator/iptables-alerter-mv2w2" Jul 16 12:39:03.948205 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.947720 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/95be9185-aa73-4b64-9301-04e5ca0c661e-etc-selinux\") pod \"aws-ebs-csi-driver-node-r6gdt\" (UID: \"95be9185-aa73-4b64-9301-04e5ca0c661e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gdt" Jul 16 12:39:03.948205 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.947746 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-host-var-lib-cni-multus\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:03.948205 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.947782 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-tmp\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:03.948205 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.947808 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04c215b4-7f84-4763-9ad8-3db823a6c2c6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m8pdn\" (UID: \"04c215b4-7f84-4763-9ad8-3db823a6c2c6\") " pod="openshift-multus/multus-additional-cni-plugins-m8pdn" Jul 16 12:39:03.948934 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.947852 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-run-openvswitch\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:03.948934 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.947915 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-host\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:03.948934 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.947956 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcdkr\" (UniqueName: \"kubernetes.io/projected/fdd6ebe0-1750-4a12-a4c7-023cfadcbabc-kube-api-access-wcdkr\") pod \"network-metrics-daemon-q5cp8\" (UID: \"fdd6ebe0-1750-4a12-a4c7-023cfadcbabc\") " pod="openshift-multus/network-metrics-daemon-q5cp8" Jul 16 12:39:03.948934 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.948012 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-hostroot\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:03.948934 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.948058 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r9hb\" (UniqueName: \"kubernetes.io/projected/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-kube-api-access-4r9hb\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:03.948934 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.948087 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-host-kubelet\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:03.948934 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.948138 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-host-run-netns\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:03.948934 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.948178 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-host-cni-bin\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:03.948934 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.948211 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zv4q\" (UniqueName: \"kubernetes.io/projected/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-kube-api-access-4zv4q\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:03.948934 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.948264 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-multus-cni-dir\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:03.948934 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.948284 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-multus-conf-dir\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:03.948934 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.948309 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-host-cni-netd\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:03.948934 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.948905 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-etc-sysctl-d\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:03.949573 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.948951 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b9a976d1-7bb2-47d6-b285-788a6ff40cef-kubelet-config\") pod \"global-pull-secret-syncer-zd992\" (UID: \"b9a976d1-7bb2-47d6-b285-788a6ff40cef\") " pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:39:03.949573 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.948989 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/04c215b4-7f84-4763-9ad8-3db823a6c2c6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m8pdn\" (UID: \"04c215b4-7f84-4763-9ad8-3db823a6c2c6\") " pod="openshift-multus/multus-additional-cni-plugins-m8pdn" Jul 16 12:39:03.949573 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.949024 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-cnibin\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:03.949573 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.949049 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-host-var-lib-kubelet\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:03.949573 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.949079 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-host-run-multus-certs\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:03.949573 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.949121 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-host-slash\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:03.949573 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.949160 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:03.949573 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.949192 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzjjn\" (UniqueName: \"kubernetes.io/projected/b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa-kube-api-access-nzjjn\") pod \"network-check-target-xgvpv\" (UID: \"b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa\") " pod="openshift-network-diagnostics/network-check-target-xgvpv" Jul 16 12:39:03.949573 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.949222 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/95be9185-aa73-4b64-9301-04e5ca0c661e-device-dir\") pod \"aws-ebs-csi-driver-node-r6gdt\" (UID: \"95be9185-aa73-4b64-9301-04e5ca0c661e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gdt" Jul 16 12:39:03.949573 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.949253 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/04c215b4-7f84-4763-9ad8-3db823a6c2c6-os-release\") pod \"multus-additional-cni-plugins-m8pdn\" (UID: \"04c215b4-7f84-4763-9ad8-3db823a6c2c6\") " pod="openshift-multus/multus-additional-cni-plugins-m8pdn" Jul 16 12:39:03.949573 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.949289 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/04c215b4-7f84-4763-9ad8-3db823a6c2c6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m8pdn\" (UID: \"04c215b4-7f84-4763-9ad8-3db823a6c2c6\") " pod="openshift-multus/multus-additional-cni-plugins-m8pdn" Jul 16 12:39:03.949573 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.949325 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rptqs\" (UniqueName: \"kubernetes.io/projected/04c215b4-7f84-4763-9ad8-3db823a6c2c6-kube-api-access-rptqs\") pod \"multus-additional-cni-plugins-m8pdn\" (UID: \"04c215b4-7f84-4763-9ad8-3db823a6c2c6\") " pod="openshift-multus/multus-additional-cni-plugins-m8pdn" Jul 16 12:39:03.949573 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.949359 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a80d1676-7c47-4427-992c-234fbc36cf7b-konnectivity-ca\") pod \"konnectivity-agent-4ns96\" (UID: \"a80d1676-7c47-4427-992c-234fbc36cf7b\") " pod="kube-system/konnectivity-agent-4ns96" Jul 16 12:39:03.949573 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.949383 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-var-lib-kubelet\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:03.949573 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.949417 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/95be9185-aa73-4b64-9301-04e5ca0c661e-registration-dir\") pod \"aws-ebs-csi-driver-node-r6gdt\" (UID: \"95be9185-aa73-4b64-9301-04e5ca0c661e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gdt" Jul 16 12:39:03.949573 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.949477 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-node-log\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:03.952560 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.950485 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvrdz\" (UniqueName: \"kubernetes.io/projected/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-kube-api-access-vvrdz\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:03.952560 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.950513 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b9a976d1-7bb2-47d6-b285-788a6ff40cef-dbus\") pod \"global-pull-secret-syncer-zd992\" (UID: \"b9a976d1-7bb2-47d6-b285-788a6ff40cef\") " pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:39:03.952560 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.950660 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/95be9185-aa73-4b64-9301-04e5ca0c661e-sys-fs\") pod \"aws-ebs-csi-driver-node-r6gdt\" (UID: \"95be9185-aa73-4b64-9301-04e5ca0c661e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gdt" Jul 16 12:39:03.952560 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.950741 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2fld\" (UniqueName: \"kubernetes.io/projected/95be9185-aa73-4b64-9301-04e5ca0c661e-kube-api-access-j2fld\") pod \"aws-ebs-csi-driver-node-r6gdt\" (UID: \"95be9185-aa73-4b64-9301-04e5ca0c661e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gdt" Jul 16 12:39:03.952560 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.950821 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-system-cni-dir\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:03.952560 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.950885 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b95cb0aa-837a-41a4-913c-7e3b15f04fdf-serviceca\") pod \"node-ca-76lb5\" (UID: \"b95cb0aa-837a-41a4-913c-7e3b15f04fdf\") " pod="openshift-image-registry/node-ca-76lb5" Jul 16 12:39:03.952560 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.950945 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-run-systemd\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:03.952560 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.950980 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-ovnkube-script-lib\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:03.952560 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.951013 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-etc-modprobe-d\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:03.952560 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.951044 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-etc-systemd\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:03.952560 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.951107 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-etc-openvswitch\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:03.952560 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.951133 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-sys\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:03.952560 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.951185 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd6ebe0-1750-4a12-a4c7-023cfadcbabc-metrics-certs\") pod \"network-metrics-daemon-q5cp8\" (UID: \"fdd6ebe0-1750-4a12-a4c7-023cfadcbabc\") " pod="openshift-multus/network-metrics-daemon-q5cp8" Jul 16 12:39:03.952560 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.951256 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a30e0419-799d-45fb-a586-f06018bc1ee4-iptables-alerter-script\") pod \"iptables-alerter-mv2w2\" (UID: \"a30e0419-799d-45fb-a586-f06018bc1ee4\") " pod="openshift-network-operator/iptables-alerter-mv2w2" Jul 16 12:39:03.952560 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.951302 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a30e0419-799d-45fb-a586-f06018bc1ee4-host-slash\") pod \"iptables-alerter-mv2w2\" (UID: \"a30e0419-799d-45fb-a586-f06018bc1ee4\") " pod="openshift-network-operator/iptables-alerter-mv2w2" Jul 16 12:39:03.976174 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.976119 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal" event={"ID":"eec32a5cfb1937ffec65aa300c58f36d","Type":"ContainerStarted","Data":"db61a6e49c5dda269b8f8ff9671f5244f2034108f0dbacd472f7fa9d415559c8"} Jul 16 12:39:03.978354 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:03.978327 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-130.ec2.internal" event={"ID":"42e2219b6620be69c895eb04c7d0d0b4","Type":"ContainerStarted","Data":"6c8974ba47a940dd5b9ae0f02064e9ee204996d66c5ff4c255a2af514eff2688"} Jul 16 12:39:04.021521 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.021489 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 12:39:04.051717 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.051679 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcdkr\" (UniqueName: \"kubernetes.io/projected/fdd6ebe0-1750-4a12-a4c7-023cfadcbabc-kube-api-access-wcdkr\") pod \"network-metrics-daemon-q5cp8\" (UID: \"fdd6ebe0-1750-4a12-a4c7-023cfadcbabc\") " pod="openshift-multus/network-metrics-daemon-q5cp8" Jul 16 12:39:04.051879 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.051732 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-hostroot\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.051879 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.051764 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r9hb\" (UniqueName: \"kubernetes.io/projected/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-kube-api-access-4r9hb\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.051879 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.051793 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-host-kubelet\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.051879 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.051820 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-host-run-netns\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.051879 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.051844 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-host-cni-bin\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.051879 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.051869 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zv4q\" (UniqueName: \"kubernetes.io/projected/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-kube-api-access-4zv4q\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.052210 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.051893 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-multus-cni-dir\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.052210 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.051916 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-multus-conf-dir\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.052210 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.051942 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-host-cni-netd\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.052210 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.051967 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-etc-sysctl-d\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:04.052210 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.051993 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b9a976d1-7bb2-47d6-b285-788a6ff40cef-kubelet-config\") pod \"global-pull-secret-syncer-zd992\" (UID: \"b9a976d1-7bb2-47d6-b285-788a6ff40cef\") " pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:39:04.052210 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.052056 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/04c215b4-7f84-4763-9ad8-3db823a6c2c6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m8pdn\" (UID: \"04c215b4-7f84-4763-9ad8-3db823a6c2c6\") " pod="openshift-multus/multus-additional-cni-plugins-m8pdn" Jul 16 12:39:04.052210 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.052082 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-cnibin\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.052210 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.052107 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-host-var-lib-kubelet\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.052210 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.052142 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-host-run-multus-certs\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.052210 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.052167 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-host-slash\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.052210 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.052199 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.052781 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.052227 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzjjn\" (UniqueName: \"kubernetes.io/projected/b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa-kube-api-access-nzjjn\") pod \"network-check-target-xgvpv\" (UID: \"b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa\") " pod="openshift-network-diagnostics/network-check-target-xgvpv" Jul 16 12:39:04.052781 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.052257 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/95be9185-aa73-4b64-9301-04e5ca0c661e-device-dir\") pod \"aws-ebs-csi-driver-node-r6gdt\" (UID: \"95be9185-aa73-4b64-9301-04e5ca0c661e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gdt" Jul 16 12:39:04.052781 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.052281 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/04c215b4-7f84-4763-9ad8-3db823a6c2c6-os-release\") pod \"multus-additional-cni-plugins-m8pdn\" (UID: \"04c215b4-7f84-4763-9ad8-3db823a6c2c6\") " pod="openshift-multus/multus-additional-cni-plugins-m8pdn" Jul 16 12:39:04.052781 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.052306 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/04c215b4-7f84-4763-9ad8-3db823a6c2c6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m8pdn\" (UID: \"04c215b4-7f84-4763-9ad8-3db823a6c2c6\") " pod="openshift-multus/multus-additional-cni-plugins-m8pdn" Jul 16 12:39:04.052781 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.052362 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rptqs\" (UniqueName: \"kubernetes.io/projected/04c215b4-7f84-4763-9ad8-3db823a6c2c6-kube-api-access-rptqs\") pod \"multus-additional-cni-plugins-m8pdn\" (UID: \"04c215b4-7f84-4763-9ad8-3db823a6c2c6\") " pod="openshift-multus/multus-additional-cni-plugins-m8pdn" Jul 16 12:39:04.052781 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.052390 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a80d1676-7c47-4427-992c-234fbc36cf7b-konnectivity-ca\") pod \"konnectivity-agent-4ns96\" (UID: \"a80d1676-7c47-4427-992c-234fbc36cf7b\") " pod="kube-system/konnectivity-agent-4ns96" Jul 16 12:39:04.052781 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.052417 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-var-lib-kubelet\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:04.052781 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.052483 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/95be9185-aa73-4b64-9301-04e5ca0c661e-registration-dir\") pod \"aws-ebs-csi-driver-node-r6gdt\" (UID: \"95be9185-aa73-4b64-9301-04e5ca0c661e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gdt" Jul 16 12:39:04.052781 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.052510 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-node-log\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.052781 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.052537 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvrdz\" (UniqueName: \"kubernetes.io/projected/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-kube-api-access-vvrdz\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:04.052781 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.052584 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b9a976d1-7bb2-47d6-b285-788a6ff40cef-dbus\") pod \"global-pull-secret-syncer-zd992\" (UID: \"b9a976d1-7bb2-47d6-b285-788a6ff40cef\") " pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:39:04.052781 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.052612 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/95be9185-aa73-4b64-9301-04e5ca0c661e-sys-fs\") pod \"aws-ebs-csi-driver-node-r6gdt\" (UID: \"95be9185-aa73-4b64-9301-04e5ca0c661e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gdt" Jul 16 12:39:04.052781 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.052661 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2fld\" (UniqueName: \"kubernetes.io/projected/95be9185-aa73-4b64-9301-04e5ca0c661e-kube-api-access-j2fld\") pod \"aws-ebs-csi-driver-node-r6gdt\" (UID: \"95be9185-aa73-4b64-9301-04e5ca0c661e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gdt" Jul 16 12:39:04.052781 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.052688 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-system-cni-dir\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.052781 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.052713 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b95cb0aa-837a-41a4-913c-7e3b15f04fdf-serviceca\") pod \"node-ca-76lb5\" (UID: \"b95cb0aa-837a-41a4-913c-7e3b15f04fdf\") " pod="openshift-image-registry/node-ca-76lb5" Jul 16 12:39:04.052781 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.052738 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-run-systemd\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.052781 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.052768 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-ovnkube-script-lib\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.053390 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.052795 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-etc-modprobe-d\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:04.053390 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.052821 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-etc-systemd\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:04.053390 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.052844 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-hostroot\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.053390 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.052862 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-etc-openvswitch\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.053390 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.052886 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-sys\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:04.053390 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.052922 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd6ebe0-1750-4a12-a4c7-023cfadcbabc-metrics-certs\") pod \"network-metrics-daemon-q5cp8\" (UID: \"fdd6ebe0-1750-4a12-a4c7-023cfadcbabc\") " pod="openshift-multus/network-metrics-daemon-q5cp8" Jul 16 12:39:04.053390 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.052972 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a30e0419-799d-45fb-a586-f06018bc1ee4-iptables-alerter-script\") pod \"iptables-alerter-mv2w2\" (UID: \"a30e0419-799d-45fb-a586-f06018bc1ee4\") " pod="openshift-network-operator/iptables-alerter-mv2w2" Jul 16 12:39:04.053390 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053001 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a30e0419-799d-45fb-a586-f06018bc1ee4-host-slash\") pod \"iptables-alerter-mv2w2\" (UID: \"a30e0419-799d-45fb-a586-f06018bc1ee4\") " pod="openshift-network-operator/iptables-alerter-mv2w2" Jul 16 12:39:04.053390 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053027 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04c215b4-7f84-4763-9ad8-3db823a6c2c6-cnibin\") pod \"multus-additional-cni-plugins-m8pdn\" (UID: \"04c215b4-7f84-4763-9ad8-3db823a6c2c6\") " pod="openshift-multus/multus-additional-cni-plugins-m8pdn" Jul 16 12:39:04.053390 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053053 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-host-run-k8s-cni-cncf-io\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.053390 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053080 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b95cb0aa-837a-41a4-913c-7e3b15f04fdf-host\") pod \"node-ca-76lb5\" (UID: \"b95cb0aa-837a-41a4-913c-7e3b15f04fdf\") " pod="openshift-image-registry/node-ca-76lb5" Jul 16 12:39:04.053390 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053106 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04c215b4-7f84-4763-9ad8-3db823a6c2c6-system-cni-dir\") pod \"multus-additional-cni-plugins-m8pdn\" (UID: \"04c215b4-7f84-4763-9ad8-3db823a6c2c6\") " pod="openshift-multus/multus-additional-cni-plugins-m8pdn" Jul 16 12:39:04.053390 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053115 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-host-kubelet\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.053390 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053134 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-multus-socket-dir-parent\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.053390 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053157 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45kfk\" (UniqueName: \"kubernetes.io/projected/b95cb0aa-837a-41a4-913c-7e3b15f04fdf-kube-api-access-45kfk\") pod \"node-ca-76lb5\" (UID: \"b95cb0aa-837a-41a4-913c-7e3b15f04fdf\") " pod="openshift-image-registry/node-ca-76lb5" Jul 16 12:39:04.053390 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053174 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-var-lib-openvswitch\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.053390 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053177 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-host-run-netns\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.054236 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053190 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/95be9185-aa73-4b64-9301-04e5ca0c661e-socket-dir\") pod \"aws-ebs-csi-driver-node-r6gdt\" (UID: \"95be9185-aa73-4b64-9301-04e5ca0c661e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gdt" Jul 16 12:39:04.054236 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053216 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-host-run-netns\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.054236 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053227 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-host-cni-bin\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.054236 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053233 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9a976d1-7bb2-47d6-b285-788a6ff40cef-original-pull-secret\") pod \"global-pull-secret-syncer-zd992\" (UID: \"b9a976d1-7bb2-47d6-b285-788a6ff40cef\") " pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:39:04.054236 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053271 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04c215b4-7f84-4763-9ad8-3db823a6c2c6-cni-binary-copy\") pod \"multus-additional-cni-plugins-m8pdn\" (UID: \"04c215b4-7f84-4763-9ad8-3db823a6c2c6\") " pod="openshift-multus/multus-additional-cni-plugins-m8pdn" Jul 16 12:39:04.054236 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053290 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-cni-binary-copy\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.054236 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053307 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-host-var-lib-cni-bin\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.054236 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053323 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-etc-kubernetes\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.054236 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053342 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-multus-cni-dir\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.054236 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053356 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-log-socket\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.054236 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053383 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e332a21-d3fd-468d-a704-8a98f55f1596-tmp-dir\") pod \"node-resolver-lv8x6\" (UID: \"7e332a21-d3fd-468d-a704-8a98f55f1596\") " pod="openshift-dns/node-resolver-lv8x6" Jul 16 12:39:04.054236 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053390 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-multus-conf-dir\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.054236 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053411 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhlsr\" (UniqueName: \"kubernetes.io/projected/7e332a21-d3fd-468d-a704-8a98f55f1596-kube-api-access-qhlsr\") pod \"node-resolver-lv8x6\" (UID: \"7e332a21-d3fd-468d-a704-8a98f55f1596\") " pod="openshift-dns/node-resolver-lv8x6" Jul 16 12:39:04.054236 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053433 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-host-cni-netd\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.054236 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053438 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-ovnkube-config\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.054236 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053501 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-multus-daemon-config\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.054236 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053532 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a80d1676-7c47-4427-992c-234fbc36cf7b-agent-certs\") pod \"konnectivity-agent-4ns96\" (UID: \"a80d1676-7c47-4427-992c-234fbc36cf7b\") " pod="kube-system/konnectivity-agent-4ns96" Jul 16 12:39:04.054236 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053562 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-host-run-ovn-kubernetes\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.055008 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053590 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-env-overrides\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.055008 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053620 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-ovn-node-metrics-cert\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.055008 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053648 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-etc-sysconfig\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:04.055008 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053676 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-etc-sysctl-conf\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:04.055008 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053703 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-run\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:04.055008 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053733 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-os-release\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.055008 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053760 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7e332a21-d3fd-468d-a704-8a98f55f1596-hosts-file\") pod \"node-resolver-lv8x6\" (UID: \"7e332a21-d3fd-468d-a704-8a98f55f1596\") " pod="openshift-dns/node-resolver-lv8x6" Jul 16 12:39:04.055008 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053790 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-systemd-units\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.055008 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053818 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-run-ovn\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.055008 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053845 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-lib-modules\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:04.055008 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053873 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/95be9185-aa73-4b64-9301-04e5ca0c661e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r6gdt\" (UID: \"95be9185-aa73-4b64-9301-04e5ca0c661e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gdt" Jul 16 12:39:04.055008 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053904 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-etc-kubernetes\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:04.055008 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053933 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-etc-tuned\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:04.055008 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053961 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgvzh\" (UniqueName: \"kubernetes.io/projected/a30e0419-799d-45fb-a586-f06018bc1ee4-kube-api-access-sgvzh\") pod \"iptables-alerter-mv2w2\" (UID: \"a30e0419-799d-45fb-a586-f06018bc1ee4\") " pod="openshift-network-operator/iptables-alerter-mv2w2" Jul 16 12:39:04.055008 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.053992 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/95be9185-aa73-4b64-9301-04e5ca0c661e-etc-selinux\") pod \"aws-ebs-csi-driver-node-r6gdt\" (UID: \"95be9185-aa73-4b64-9301-04e5ca0c661e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gdt" Jul 16 12:39:04.055008 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.054023 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-host-var-lib-cni-multus\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.055008 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.054050 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-tmp\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:04.055545 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.054104 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-etc-sysctl-conf\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:04.055545 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.054132 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-cnibin\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.055545 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.054182 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04c215b4-7f84-4763-9ad8-3db823a6c2c6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m8pdn\" (UID: \"04c215b4-7f84-4763-9ad8-3db823a6c2c6\") " pod="openshift-multus/multus-additional-cni-plugins-m8pdn" Jul 16 12:39:04.055545 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.054231 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-run-openvswitch\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.055545 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.054249 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-etc-sysctl-d\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:04.055545 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.054268 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-host\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:04.055545 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.054337 2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 16 12:39:04.055545 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.054451 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-host\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:04.055545 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.054501 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-host-var-lib-kubelet\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.055545 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.054556 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-host-run-multus-certs\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.055545 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.054584 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-host-slash\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.055545 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.054622 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.055545 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.054795 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-ovnkube-config\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.055545 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.054828 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/95be9185-aa73-4b64-9301-04e5ca0c661e-device-dir\") pod \"aws-ebs-csi-driver-node-r6gdt\" (UID: \"95be9185-aa73-4b64-9301-04e5ca0c661e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gdt" Jul 16 12:39:04.055545 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.054884 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/04c215b4-7f84-4763-9ad8-3db823a6c2c6-os-release\") pod \"multus-additional-cni-plugins-m8pdn\" (UID: \"04c215b4-7f84-4763-9ad8-3db823a6c2c6\") " pod="openshift-multus/multus-additional-cni-plugins-m8pdn" Jul 16 12:39:04.055545 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.055234 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-multus-daemon-config\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.055545 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.055243 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/04c215b4-7f84-4763-9ad8-3db823a6c2c6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m8pdn\" (UID: \"04c215b4-7f84-4763-9ad8-3db823a6c2c6\") " pod="openshift-multus/multus-additional-cni-plugins-m8pdn" Jul 16 12:39:04.055545 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.055275 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-host-run-ovn-kubernetes\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.056365 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.055552 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-env-overrides\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.056365 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.055737 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-run\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:04.056365 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.055819 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-os-release\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.056365 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.055877 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7e332a21-d3fd-468d-a704-8a98f55f1596-hosts-file\") pod \"node-resolver-lv8x6\" (UID: \"7e332a21-d3fd-468d-a704-8a98f55f1596\") " pod="openshift-dns/node-resolver-lv8x6" Jul 16 12:39:04.056365 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.055960 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a30e0419-799d-45fb-a586-f06018bc1ee4-host-slash\") pod \"iptables-alerter-mv2w2\" (UID: \"a30e0419-799d-45fb-a586-f06018bc1ee4\") " pod="openshift-network-operator/iptables-alerter-mv2w2" Jul 16 12:39:04.056365 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.056010 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-systemd-units\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.056365 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.056116 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-lib-modules\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:04.056365 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.056163 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/95be9185-aa73-4b64-9301-04e5ca0c661e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r6gdt\" (UID: \"95be9185-aa73-4b64-9301-04e5ca0c661e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gdt" Jul 16 12:39:04.056365 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.056210 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-etc-kubernetes\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:04.057029 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.056574 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-run-ovn\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.057029 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.056757 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-system-cni-dir\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.057029 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.056825 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-var-lib-kubelet\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:04.057029 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.056880 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/95be9185-aa73-4b64-9301-04e5ca0c661e-registration-dir\") pod \"aws-ebs-csi-driver-node-r6gdt\" (UID: \"95be9185-aa73-4b64-9301-04e5ca0c661e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gdt" Jul 16 12:39:04.057029 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.056890 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/95be9185-aa73-4b64-9301-04e5ca0c661e-etc-selinux\") pod \"aws-ebs-csi-driver-node-r6gdt\" (UID: \"95be9185-aa73-4b64-9301-04e5ca0c661e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gdt" Jul 16 12:39:04.057029 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.056938 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-node-log\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.057029 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.056996 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-host-var-lib-cni-multus\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.057386 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.057048 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-etc-sysconfig\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:04.057386 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.052554 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b9a976d1-7bb2-47d6-b285-788a6ff40cef-kubelet-config\") pod \"global-pull-secret-syncer-zd992\" (UID: \"b9a976d1-7bb2-47d6-b285-788a6ff40cef\") " pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:39:04.057386 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.057210 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b9a976d1-7bb2-47d6-b285-788a6ff40cef-dbus\") pod \"global-pull-secret-syncer-zd992\" (UID: \"b9a976d1-7bb2-47d6-b285-788a6ff40cef\") " pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:39:04.057386 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.057265 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/95be9185-aa73-4b64-9301-04e5ca0c661e-sys-fs\") pod \"aws-ebs-csi-driver-node-r6gdt\" (UID: \"95be9185-aa73-4b64-9301-04e5ca0c661e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gdt" Jul 16 12:39:04.057598 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.057515 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-multus-socket-dir-parent\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.057598 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.057534 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04c215b4-7f84-4763-9ad8-3db823a6c2c6-cni-binary-copy\") pod \"multus-additional-cni-plugins-m8pdn\" (UID: \"04c215b4-7f84-4763-9ad8-3db823a6c2c6\") " pod="openshift-multus/multus-additional-cni-plugins-m8pdn" Jul 16 12:39:04.057598 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.057567 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04c215b4-7f84-4763-9ad8-3db823a6c2c6-cnibin\") pod \"multus-additional-cni-plugins-m8pdn\" (UID: \"04c215b4-7f84-4763-9ad8-3db823a6c2c6\") " pod="openshift-multus/multus-additional-cni-plugins-m8pdn" Jul 16 12:39:04.057756 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.057631 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-host-run-k8s-cni-cncf-io\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.057756 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.057677 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b95cb0aa-837a-41a4-913c-7e3b15f04fdf-host\") pod \"node-ca-76lb5\" (UID: \"b95cb0aa-837a-41a4-913c-7e3b15f04fdf\") " pod="openshift-image-registry/node-ca-76lb5" Jul 16 12:39:04.057756 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.057719 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04c215b4-7f84-4763-9ad8-3db823a6c2c6-system-cni-dir\") pod \"multus-additional-cni-plugins-m8pdn\" (UID: \"04c215b4-7f84-4763-9ad8-3db823a6c2c6\") " pod="openshift-multus/multus-additional-cni-plugins-m8pdn" Jul 16 12:39:04.057907 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.057764 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-log-socket\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.057907 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.057778 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-var-lib-openvswitch\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.057907 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.057890 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/95be9185-aa73-4b64-9301-04e5ca0c661e-socket-dir\") pod \"aws-ebs-csi-driver-node-r6gdt\" (UID: \"95be9185-aa73-4b64-9301-04e5ca0c661e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gdt" Jul 16 12:39:04.058049 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.057940 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-host-run-netns\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.058049 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:04.058026 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:39:04.058144 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:04.058088 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9a976d1-7bb2-47d6-b285-788a6ff40cef-original-pull-secret podName:b9a976d1-7bb2-47d6-b285-788a6ff40cef nodeName:}" failed. No retries permitted until 2026-07-16 12:39:04.558066232 +0000 UTC m=+3.035811859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9a976d1-7bb2-47d6-b285-788a6ff40cef-original-pull-secret") pod "global-pull-secret-syncer-zd992" (UID: "b9a976d1-7bb2-47d6-b285-788a6ff40cef") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:39:04.058198 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.058174 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/04c215b4-7f84-4763-9ad8-3db823a6c2c6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m8pdn\" (UID: \"04c215b4-7f84-4763-9ad8-3db823a6c2c6\") " pod="openshift-multus/multus-additional-cni-plugins-m8pdn" Jul 16 12:39:04.058269 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.058248 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-host-var-lib-cni-bin\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.058319 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.058305 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-etc-kubernetes\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.058510 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.058454 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-cni-binary-copy\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.058510 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.058486 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04c215b4-7f84-4763-9ad8-3db823a6c2c6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m8pdn\" (UID: \"04c215b4-7f84-4763-9ad8-3db823a6c2c6\") " pod="openshift-multus/multus-additional-cni-plugins-m8pdn" Jul 16 12:39:04.058645 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.058548 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-etc-systemd\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:04.058645 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.058554 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-etc-openvswitch\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.058645 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.058548 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-sys\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:04.058795 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.058720 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-etc-modprobe-d\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:04.058852 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.058828 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e332a21-d3fd-468d-a704-8a98f55f1596-tmp-dir\") pod \"node-resolver-lv8x6\" (UID: \"7e332a21-d3fd-468d-a704-8a98f55f1596\") " pod="openshift-dns/node-resolver-lv8x6" Jul 16 12:39:04.058905 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.058894 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-run-systemd\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.058990 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.058964 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b95cb0aa-837a-41a4-913c-7e3b15f04fdf-serviceca\") pod \"node-ca-76lb5\" (UID: \"b95cb0aa-837a-41a4-913c-7e3b15f04fdf\") " pod="openshift-image-registry/node-ca-76lb5" Jul 16 12:39:04.059495 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:04.059437 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:39:04.059575 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:04.059527 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd6ebe0-1750-4a12-a4c7-023cfadcbabc-metrics-certs podName:fdd6ebe0-1750-4a12-a4c7-023cfadcbabc nodeName:}" failed. No retries permitted until 2026-07-16 12:39:04.559507773 +0000 UTC m=+3.037253376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdd6ebe0-1750-4a12-a4c7-023cfadcbabc-metrics-certs") pod "network-metrics-daemon-q5cp8" (UID: "fdd6ebe0-1750-4a12-a4c7-023cfadcbabc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:39:04.059575 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.059535 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-ovnkube-script-lib\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.059699 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.059622 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-run-openvswitch\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.059813 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.059793 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-tmp\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:04.060139 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.060110 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a30e0419-799d-45fb-a586-f06018bc1ee4-iptables-alerter-script\") pod \"iptables-alerter-mv2w2\" (UID: \"a30e0419-799d-45fb-a586-f06018bc1ee4\") " pod="openshift-network-operator/iptables-alerter-mv2w2" Jul 16 12:39:04.060219 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.060144 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-etc-tuned\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:04.060497 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.060453 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a80d1676-7c47-4427-992c-234fbc36cf7b-konnectivity-ca\") pod \"konnectivity-agent-4ns96\" (UID: \"a80d1676-7c47-4427-992c-234fbc36cf7b\") " pod="kube-system/konnectivity-agent-4ns96" Jul 16 12:39:04.066405 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:04.066377 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:39:04.066405 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:04.066406 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:39:04.066584 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:04.066421 2582 projected.go:196] Error preparing data for projected volume kube-api-access-nzjjn for pod openshift-network-diagnostics/network-check-target-xgvpv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:39:04.066584 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:04.066528 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa-kube-api-access-nzjjn podName:b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa nodeName:}" failed. No retries permitted until 2026-07-16 12:39:04.566509732 +0000 UTC m=+3.044255342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nzjjn" (UniqueName: "kubernetes.io/projected/b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa-kube-api-access-nzjjn") pod "network-check-target-xgvpv" (UID: "b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:39:04.070426 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.070396 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a80d1676-7c47-4427-992c-234fbc36cf7b-agent-certs\") pod \"konnectivity-agent-4ns96\" (UID: \"a80d1676-7c47-4427-992c-234fbc36cf7b\") " pod="kube-system/konnectivity-agent-4ns96" Jul 16 12:39:04.072503 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.072413 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcdkr\" (UniqueName: \"kubernetes.io/projected/fdd6ebe0-1750-4a12-a4c7-023cfadcbabc-kube-api-access-wcdkr\") pod \"network-metrics-daemon-q5cp8\" (UID: \"fdd6ebe0-1750-4a12-a4c7-023cfadcbabc\") " pod="openshift-multus/network-metrics-daemon-q5cp8" Jul 16 12:39:04.073180 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.073156 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2fld\" (UniqueName: \"kubernetes.io/projected/95be9185-aa73-4b64-9301-04e5ca0c661e-kube-api-access-j2fld\") pod \"aws-ebs-csi-driver-node-r6gdt\" (UID: \"95be9185-aa73-4b64-9301-04e5ca0c661e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gdt" Jul 16 12:39:04.073658 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.073593 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r9hb\" (UniqueName: \"kubernetes.io/projected/431ba1c9-15f3-4a03-ab1d-8a2b983a0704-kube-api-access-4r9hb\") pod \"multus-5s6sz\" (UID: \"431ba1c9-15f3-4a03-ab1d-8a2b983a0704\") " pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.073997 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.073952 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-ovn-node-metrics-cert\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.074854 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.074828 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rptqs\" (UniqueName: \"kubernetes.io/projected/04c215b4-7f84-4763-9ad8-3db823a6c2c6-kube-api-access-rptqs\") pod \"multus-additional-cni-plugins-m8pdn\" (UID: \"04c215b4-7f84-4763-9ad8-3db823a6c2c6\") " pod="openshift-multus/multus-additional-cni-plugins-m8pdn" Jul 16 12:39:04.074995 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.074828 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgvzh\" (UniqueName: \"kubernetes.io/projected/a30e0419-799d-45fb-a586-f06018bc1ee4-kube-api-access-sgvzh\") pod \"iptables-alerter-mv2w2\" (UID: \"a30e0419-799d-45fb-a586-f06018bc1ee4\") " pod="openshift-network-operator/iptables-alerter-mv2w2" Jul 16 12:39:04.074995 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.074830 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvrdz\" (UniqueName: \"kubernetes.io/projected/e6d8dd65-4edb-485a-90d0-6e98e5c075fe-kube-api-access-vvrdz\") pod \"tuned-dc8zj\" (UID: \"e6d8dd65-4edb-485a-90d0-6e98e5c075fe\") " pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:04.077209 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.077186 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45kfk\" (UniqueName: \"kubernetes.io/projected/b95cb0aa-837a-41a4-913c-7e3b15f04fdf-kube-api-access-45kfk\") pod \"node-ca-76lb5\" (UID: \"b95cb0aa-837a-41a4-913c-7e3b15f04fdf\") " pod="openshift-image-registry/node-ca-76lb5" Jul 16 12:39:04.077901 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.077842 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zv4q\" (UniqueName: \"kubernetes.io/projected/ffe9c110-1bbe-42a5-bd97-b2f3c70e680d-kube-api-access-4zv4q\") pod \"ovnkube-node-986g4\" (UID: \"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d\") " pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.080135 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.079728 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhlsr\" (UniqueName: \"kubernetes.io/projected/7e332a21-d3fd-468d-a704-8a98f55f1596-kube-api-access-qhlsr\") pod \"node-resolver-lv8x6\" (UID: \"7e332a21-d3fd-468d-a704-8a98f55f1596\") " pod="openshift-dns/node-resolver-lv8x6" Jul 16 12:39:04.158904 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.158650 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mv2w2" Jul 16 12:39:04.166879 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.166853 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gdt" Jul 16 12:39:04.171573 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:04.171550 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda30e0419_799d_45fb_a586_f06018bc1ee4.slice/crio-d3fc932a6b27956596c0098a0b2be2178a049fd527c6738fa1ffdfaa1c187502 WatchSource:0}: Error finding container d3fc932a6b27956596c0098a0b2be2178a049fd527c6738fa1ffdfaa1c187502: Status 404 returned error can't find the container with id d3fc932a6b27956596c0098a0b2be2178a049fd527c6738fa1ffdfaa1c187502 Jul 16 12:39:04.176118 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.176097 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:04.180083 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:04.180035 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95be9185_aa73_4b64_9301_04e5ca0c661e.slice/crio-22b416f807762167b51e1955f732e68c1852075dca5c561503000fe844a6374c WatchSource:0}: Error finding container 22b416f807762167b51e1955f732e68c1852075dca5c561503000fe844a6374c: Status 404 returned error can't find the container with id 22b416f807762167b51e1955f732e68c1852075dca5c561503000fe844a6374c Jul 16 12:39:04.180361 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.180341 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lv8x6" Jul 16 12:39:04.189570 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.189488 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-76lb5" Jul 16 12:39:04.197164 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.194732 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m8pdn" Jul 16 12:39:04.197164 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.197021 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5s6sz" Jul 16 12:39:04.203458 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.203436 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4ns96" Jul 16 12:39:04.207080 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.206918 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" Jul 16 12:39:04.220154 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:04.220074 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04c215b4_7f84_4763_9ad8_3db823a6c2c6.slice/crio-8d0b5fc6895627384c096055fe3bade47f862560d9c46dc7b0964419b3a0e3c9 WatchSource:0}: Error finding container 8d0b5fc6895627384c096055fe3bade47f862560d9c46dc7b0964419b3a0e3c9: Status 404 returned error can't find the container with id 8d0b5fc6895627384c096055fe3bade47f862560d9c46dc7b0964419b3a0e3c9 Jul 16 12:39:04.228419 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:04.228390 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod431ba1c9_15f3_4a03_ab1d_8a2b983a0704.slice/crio-186d6b30c79f40e757b08b277f9117cead7ee23583144c76ce95afc778f17878 WatchSource:0}: Error finding container 186d6b30c79f40e757b08b277f9117cead7ee23583144c76ce95afc778f17878: Status 404 returned error can't find the container with id 186d6b30c79f40e757b08b277f9117cead7ee23583144c76ce95afc778f17878 Jul 16 12:39:04.232224 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:04.232197 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6d8dd65_4edb_485a_90d0_6e98e5c075fe.slice/crio-8e4e5da3973ad88bd27a2f3a16d5ef18bb2772e8b02b537431eef51d1100b643 WatchSource:0}: Error finding container 8e4e5da3973ad88bd27a2f3a16d5ef18bb2772e8b02b537431eef51d1100b643: Status 404 returned error can't find the container with id 8e4e5da3973ad88bd27a2f3a16d5ef18bb2772e8b02b537431eef51d1100b643 Jul 16 12:39:04.238324 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:04.238271 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda80d1676_7c47_4427_992c_234fbc36cf7b.slice/crio-78e1fe9a2a93c6290153b40f081abaf36b88ed20a4d9ca106560eff118438df3 WatchSource:0}: Error finding container 78e1fe9a2a93c6290153b40f081abaf36b88ed20a4d9ca106560eff118438df3: Status 404 returned error can't find the container with id 78e1fe9a2a93c6290153b40f081abaf36b88ed20a4d9ca106560eff118438df3 Jul 16 12:39:04.559018 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.558986 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9a976d1-7bb2-47d6-b285-788a6ff40cef-original-pull-secret\") pod \"global-pull-secret-syncer-zd992\" (UID: \"b9a976d1-7bb2-47d6-b285-788a6ff40cef\") " pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:39:04.559189 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:04.559165 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:39:04.559271 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:04.559259 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9a976d1-7bb2-47d6-b285-788a6ff40cef-original-pull-secret podName:b9a976d1-7bb2-47d6-b285-788a6ff40cef nodeName:}" failed. No retries permitted until 2026-07-16 12:39:05.559232247 +0000 UTC m=+4.036977844 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9a976d1-7bb2-47d6-b285-788a6ff40cef-original-pull-secret") pod "global-pull-secret-syncer-zd992" (UID: "b9a976d1-7bb2-47d6-b285-788a6ff40cef") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:39:04.659744 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.659651 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzjjn\" (UniqueName: \"kubernetes.io/projected/b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa-kube-api-access-nzjjn\") pod \"network-check-target-xgvpv\" (UID: \"b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa\") " pod="openshift-network-diagnostics/network-check-target-xgvpv" Jul 16 12:39:04.659744 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.659696 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd6ebe0-1750-4a12-a4c7-023cfadcbabc-metrics-certs\") pod \"network-metrics-daemon-q5cp8\" (UID: \"fdd6ebe0-1750-4a12-a4c7-023cfadcbabc\") " pod="openshift-multus/network-metrics-daemon-q5cp8" Jul 16 12:39:04.659952 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:04.659828 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:39:04.659952 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:04.659852 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:39:04.659952 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:04.659877 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:39:04.659952 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:04.659889 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd6ebe0-1750-4a12-a4c7-023cfadcbabc-metrics-certs podName:fdd6ebe0-1750-4a12-a4c7-023cfadcbabc nodeName:}" failed. No retries permitted until 2026-07-16 12:39:05.659872182 +0000 UTC m=+4.137617783 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdd6ebe0-1750-4a12-a4c7-023cfadcbabc-metrics-certs") pod "network-metrics-daemon-q5cp8" (UID: "fdd6ebe0-1750-4a12-a4c7-023cfadcbabc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:39:04.659952 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:04.659892 2582 projected.go:196] Error preparing data for projected volume kube-api-access-nzjjn for pod openshift-network-diagnostics/network-check-target-xgvpv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:39:04.659952 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:04.659948 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa-kube-api-access-nzjjn podName:b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa nodeName:}" failed. No retries permitted until 2026-07-16 12:39:05.659930747 +0000 UTC m=+4.137676348 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nzjjn" (UniqueName: "kubernetes.io/projected/b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa-kube-api-access-nzjjn") pod "network-check-target-xgvpv" (UID: "b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:39:04.903020 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.902941 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 12:34:04 +0000 UTC" deadline="2028-01-07 16:38:54.766113843 +0000 UTC" Jul 16 12:39:04.903401 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.903024 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12963h59m49.863099635s" Jul 16 12:39:04.972566 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.971829 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgvpv" Jul 16 12:39:04.972566 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:04.971961 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgvpv" podUID="b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa" Jul 16 12:39:04.972566 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.972049 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5cp8" Jul 16 12:39:04.972566 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:04.972145 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5cp8" podUID="fdd6ebe0-1750-4a12-a4c7-023cfadcbabc" Jul 16 12:39:04.982782 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.982743 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5s6sz" event={"ID":"431ba1c9-15f3-4a03-ab1d-8a2b983a0704","Type":"ContainerStarted","Data":"186d6b30c79f40e757b08b277f9117cead7ee23583144c76ce95afc778f17878"} Jul 16 12:39:04.984370 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.984338 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m8pdn" event={"ID":"04c215b4-7f84-4763-9ad8-3db823a6c2c6","Type":"ContainerStarted","Data":"8d0b5fc6895627384c096055fe3bade47f862560d9c46dc7b0964419b3a0e3c9"} Jul 16 12:39:04.985901 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.985874 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-76lb5" event={"ID":"b95cb0aa-837a-41a4-913c-7e3b15f04fdf","Type":"ContainerStarted","Data":"f7e83b61995ca0911249be8105d63abef17b66552f9cf19557cf336039c62620"} Jul 16 12:39:04.990095 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.990006 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-986g4" event={"ID":"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d","Type":"ContainerStarted","Data":"b78ebeebd41618d563f61a95cc0f10f5302142970563322f02c4b2adec14fad7"} Jul 16 12:39:04.992848 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.992816 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gdt" event={"ID":"95be9185-aa73-4b64-9301-04e5ca0c661e","Type":"ContainerStarted","Data":"22b416f807762167b51e1955f732e68c1852075dca5c561503000fe844a6374c"} Jul 16 12:39:04.994636 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.994585 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4ns96" event={"ID":"a80d1676-7c47-4427-992c-234fbc36cf7b","Type":"ContainerStarted","Data":"78e1fe9a2a93c6290153b40f081abaf36b88ed20a4d9ca106560eff118438df3"} Jul 16 12:39:04.996488 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.996466 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" event={"ID":"e6d8dd65-4edb-485a-90d0-6e98e5c075fe","Type":"ContainerStarted","Data":"8e4e5da3973ad88bd27a2f3a16d5ef18bb2772e8b02b537431eef51d1100b643"} Jul 16 12:39:04.998809 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:04.998782 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lv8x6" event={"ID":"7e332a21-d3fd-468d-a704-8a98f55f1596","Type":"ContainerStarted","Data":"e21b8ecdd5e19f8e06a536198e8a8896b2c9518e9dbbe84f35b1d9f9f522fe69"} Jul 16 12:39:05.001665 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:05.001597 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mv2w2" event={"ID":"a30e0419-799d-45fb-a586-f06018bc1ee4","Type":"ContainerStarted","Data":"d3fc932a6b27956596c0098a0b2be2178a049fd527c6738fa1ffdfaa1c187502"} Jul 16 12:39:05.568088 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:05.567488 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9a976d1-7bb2-47d6-b285-788a6ff40cef-original-pull-secret\") pod \"global-pull-secret-syncer-zd992\" (UID: \"b9a976d1-7bb2-47d6-b285-788a6ff40cef\") " pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:39:05.568088 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:05.567658 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:39:05.568088 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:05.567727 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9a976d1-7bb2-47d6-b285-788a6ff40cef-original-pull-secret podName:b9a976d1-7bb2-47d6-b285-788a6ff40cef nodeName:}" failed. No retries permitted until 2026-07-16 12:39:07.567707013 +0000 UTC m=+6.045452627 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9a976d1-7bb2-47d6-b285-788a6ff40cef-original-pull-secret") pod "global-pull-secret-syncer-zd992" (UID: "b9a976d1-7bb2-47d6-b285-788a6ff40cef") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:39:05.668832 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:05.668004 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzjjn\" (UniqueName: \"kubernetes.io/projected/b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa-kube-api-access-nzjjn\") pod \"network-check-target-xgvpv\" (UID: \"b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa\") " pod="openshift-network-diagnostics/network-check-target-xgvpv" Jul 16 12:39:05.668832 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:05.668065 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd6ebe0-1750-4a12-a4c7-023cfadcbabc-metrics-certs\") pod \"network-metrics-daemon-q5cp8\" (UID: \"fdd6ebe0-1750-4a12-a4c7-023cfadcbabc\") " pod="openshift-multus/network-metrics-daemon-q5cp8" Jul 16 12:39:05.668832 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:05.668231 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:39:05.668832 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:05.668298 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd6ebe0-1750-4a12-a4c7-023cfadcbabc-metrics-certs podName:fdd6ebe0-1750-4a12-a4c7-023cfadcbabc nodeName:}" failed. No retries permitted until 2026-07-16 12:39:07.668275994 +0000 UTC m=+6.146021606 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdd6ebe0-1750-4a12-a4c7-023cfadcbabc-metrics-certs") pod "network-metrics-daemon-q5cp8" (UID: "fdd6ebe0-1750-4a12-a4c7-023cfadcbabc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:39:05.668832 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:05.668398 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:39:05.668832 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:05.668419 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:39:05.668832 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:05.668432 2582 projected.go:196] Error preparing data for projected volume kube-api-access-nzjjn for pod openshift-network-diagnostics/network-check-target-xgvpv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:39:05.668832 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:05.668487 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa-kube-api-access-nzjjn podName:b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa nodeName:}" failed. No retries permitted until 2026-07-16 12:39:07.668473948 +0000 UTC m=+6.146219560 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nzjjn" (UniqueName: "kubernetes.io/projected/b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa-kube-api-access-nzjjn") pod "network-check-target-xgvpv" (UID: "b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:39:05.903934 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:05.903818 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 12:34:04 +0000 UTC" deadline="2027-12-20 06:09:12.777265266 +0000 UTC" Jul 16 12:39:05.903934 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:05.903861 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12521h30m6.873410155s" Jul 16 12:39:05.980345 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:05.980312 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:39:05.980508 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:05.980451 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd992" podUID="b9a976d1-7bb2-47d6-b285-788a6ff40cef" Jul 16 12:39:06.971724 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:06.971694 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgvpv" Jul 16 12:39:06.972169 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:06.971823 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgvpv" podUID="b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa" Jul 16 12:39:06.972169 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:06.971889 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5cp8" Jul 16 12:39:06.972169 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:06.971983 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5cp8" podUID="fdd6ebe0-1750-4a12-a4c7-023cfadcbabc" Jul 16 12:39:07.586358 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:07.586313 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9a976d1-7bb2-47d6-b285-788a6ff40cef-original-pull-secret\") pod \"global-pull-secret-syncer-zd992\" (UID: \"b9a976d1-7bb2-47d6-b285-788a6ff40cef\") " pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:39:07.586596 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:07.586498 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:39:07.586596 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:07.586569 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9a976d1-7bb2-47d6-b285-788a6ff40cef-original-pull-secret podName:b9a976d1-7bb2-47d6-b285-788a6ff40cef nodeName:}" failed. No retries permitted until 2026-07-16 12:39:11.586548369 +0000 UTC m=+10.064293968 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9a976d1-7bb2-47d6-b285-788a6ff40cef-original-pull-secret") pod "global-pull-secret-syncer-zd992" (UID: "b9a976d1-7bb2-47d6-b285-788a6ff40cef") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:39:07.687093 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:07.687051 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzjjn\" (UniqueName: \"kubernetes.io/projected/b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa-kube-api-access-nzjjn\") pod \"network-check-target-xgvpv\" (UID: \"b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa\") " pod="openshift-network-diagnostics/network-check-target-xgvpv" Jul 16 12:39:07.687276 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:07.687119 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd6ebe0-1750-4a12-a4c7-023cfadcbabc-metrics-certs\") pod \"network-metrics-daemon-q5cp8\" (UID: \"fdd6ebe0-1750-4a12-a4c7-023cfadcbabc\") " pod="openshift-multus/network-metrics-daemon-q5cp8" Jul 16 12:39:07.687276 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:07.687243 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:39:07.687383 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:07.687306 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd6ebe0-1750-4a12-a4c7-023cfadcbabc-metrics-certs podName:fdd6ebe0-1750-4a12-a4c7-023cfadcbabc nodeName:}" failed. No retries permitted until 2026-07-16 12:39:11.687287525 +0000 UTC m=+10.165033124 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdd6ebe0-1750-4a12-a4c7-023cfadcbabc-metrics-certs") pod "network-metrics-daemon-q5cp8" (UID: "fdd6ebe0-1750-4a12-a4c7-023cfadcbabc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:39:07.687807 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:07.687768 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:39:07.687807 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:07.687802 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:39:07.687807 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:07.687816 2582 projected.go:196] Error preparing data for projected volume kube-api-access-nzjjn for pod openshift-network-diagnostics/network-check-target-xgvpv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:39:07.688027 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:07.687878 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa-kube-api-access-nzjjn podName:b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa nodeName:}" failed. No retries permitted until 2026-07-16 12:39:11.687860224 +0000 UTC m=+10.165605822 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nzjjn" (UniqueName: "kubernetes.io/projected/b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa-kube-api-access-nzjjn") pod "network-check-target-xgvpv" (UID: "b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:39:07.971359 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:07.971236 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:39:07.971557 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:07.971426 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd992" podUID="b9a976d1-7bb2-47d6-b285-788a6ff40cef" Jul 16 12:39:08.972350 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:08.972115 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgvpv" Jul 16 12:39:08.972807 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:08.972467 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgvpv" podUID="b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa" Jul 16 12:39:08.972807 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:08.972556 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5cp8" Jul 16 12:39:08.972807 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:08.972637 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5cp8" podUID="fdd6ebe0-1750-4a12-a4c7-023cfadcbabc" Jul 16 12:39:09.971899 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:09.971790 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:39:09.972052 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:09.971907 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd992" podUID="b9a976d1-7bb2-47d6-b285-788a6ff40cef" Jul 16 12:39:10.972086 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:10.972004 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgvpv" Jul 16 12:39:10.972542 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:10.972111 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgvpv" podUID="b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa" Jul 16 12:39:10.972542 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:10.972436 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5cp8" Jul 16 12:39:10.972542 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:10.972523 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5cp8" podUID="fdd6ebe0-1750-4a12-a4c7-023cfadcbabc" Jul 16 12:39:11.619272 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:11.619240 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9a976d1-7bb2-47d6-b285-788a6ff40cef-original-pull-secret\") pod \"global-pull-secret-syncer-zd992\" (UID: \"b9a976d1-7bb2-47d6-b285-788a6ff40cef\") " pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:39:11.619475 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:11.619364 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:39:11.619475 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:11.619435 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9a976d1-7bb2-47d6-b285-788a6ff40cef-original-pull-secret podName:b9a976d1-7bb2-47d6-b285-788a6ff40cef nodeName:}" failed. No retries permitted until 2026-07-16 12:39:19.619416863 +0000 UTC m=+18.097162474 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9a976d1-7bb2-47d6-b285-788a6ff40cef-original-pull-secret") pod "global-pull-secret-syncer-zd992" (UID: "b9a976d1-7bb2-47d6-b285-788a6ff40cef") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:39:11.719604 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:11.719571 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzjjn\" (UniqueName: \"kubernetes.io/projected/b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa-kube-api-access-nzjjn\") pod \"network-check-target-xgvpv\" (UID: \"b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa\") " pod="openshift-network-diagnostics/network-check-target-xgvpv" Jul 16 12:39:11.719782 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:11.719616 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd6ebe0-1750-4a12-a4c7-023cfadcbabc-metrics-certs\") pod \"network-metrics-daemon-q5cp8\" (UID: \"fdd6ebe0-1750-4a12-a4c7-023cfadcbabc\") " pod="openshift-multus/network-metrics-daemon-q5cp8" Jul 16 12:39:11.719782 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:11.719738 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:39:11.719782 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:11.719782 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd6ebe0-1750-4a12-a4c7-023cfadcbabc-metrics-certs podName:fdd6ebe0-1750-4a12-a4c7-023cfadcbabc nodeName:}" failed. No retries permitted until 2026-07-16 12:39:19.719768691 +0000 UTC m=+18.197514286 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdd6ebe0-1750-4a12-a4c7-023cfadcbabc-metrics-certs") pod "network-metrics-daemon-q5cp8" (UID: "fdd6ebe0-1750-4a12-a4c7-023cfadcbabc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:39:11.719958 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:11.719839 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:39:11.719958 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:11.719849 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:39:11.719958 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:11.719858 2582 projected.go:196] Error preparing data for projected volume kube-api-access-nzjjn for pod openshift-network-diagnostics/network-check-target-xgvpv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:39:11.719958 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:11.719882 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa-kube-api-access-nzjjn podName:b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa nodeName:}" failed. No retries permitted until 2026-07-16 12:39:19.719874021 +0000 UTC m=+18.197619629 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nzjjn" (UniqueName: "kubernetes.io/projected/b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa-kube-api-access-nzjjn") pod "network-check-target-xgvpv" (UID: "b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:39:11.973686 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:11.972499 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:39:11.973686 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:11.972619 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd992" podUID="b9a976d1-7bb2-47d6-b285-788a6ff40cef" Jul 16 12:39:12.971818 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:12.971784 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgvpv" Jul 16 12:39:12.971995 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:12.971889 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgvpv" podUID="b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa" Jul 16 12:39:12.972235 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:12.972218 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5cp8" Jul 16 12:39:12.972305 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:12.972291 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5cp8" podUID="fdd6ebe0-1750-4a12-a4c7-023cfadcbabc" Jul 16 12:39:13.971741 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:13.971708 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:39:13.972164 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:13.971836 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd992" podUID="b9a976d1-7bb2-47d6-b285-788a6ff40cef" Jul 16 12:39:14.971851 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:14.971824 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgvpv" Jul 16 12:39:14.972189 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:14.971829 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5cp8" Jul 16 12:39:14.972189 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:14.971921 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgvpv" podUID="b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa" Jul 16 12:39:14.972189 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:14.972003 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5cp8" podUID="fdd6ebe0-1750-4a12-a4c7-023cfadcbabc" Jul 16 12:39:15.972158 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:15.972123 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:39:15.972631 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:15.972222 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd992" podUID="b9a976d1-7bb2-47d6-b285-788a6ff40cef" Jul 16 12:39:16.972314 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:16.972274 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5cp8" Jul 16 12:39:16.972769 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:16.972330 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgvpv" Jul 16 12:39:16.972769 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:16.972433 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5cp8" podUID="fdd6ebe0-1750-4a12-a4c7-023cfadcbabc" Jul 16 12:39:16.972769 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:16.972510 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgvpv" podUID="b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa" Jul 16 12:39:17.971579 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:17.971542 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:39:17.971751 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:17.971676 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd992" podUID="b9a976d1-7bb2-47d6-b285-788a6ff40cef" Jul 16 12:39:18.971760 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:18.971725 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgvpv" Jul 16 12:39:18.972284 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:18.971739 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5cp8" Jul 16 12:39:18.972284 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:18.971871 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgvpv" podUID="b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa" Jul 16 12:39:18.972284 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:18.971965 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5cp8" podUID="fdd6ebe0-1750-4a12-a4c7-023cfadcbabc" Jul 16 12:39:19.681457 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:19.681414 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9a976d1-7bb2-47d6-b285-788a6ff40cef-original-pull-secret\") pod \"global-pull-secret-syncer-zd992\" (UID: \"b9a976d1-7bb2-47d6-b285-788a6ff40cef\") " pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:39:19.681768 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:19.681578 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:39:19.681768 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:19.681666 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9a976d1-7bb2-47d6-b285-788a6ff40cef-original-pull-secret podName:b9a976d1-7bb2-47d6-b285-788a6ff40cef nodeName:}" failed. No retries permitted until 2026-07-16 12:39:35.681639578 +0000 UTC m=+34.159385195 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9a976d1-7bb2-47d6-b285-788a6ff40cef-original-pull-secret") pod "global-pull-secret-syncer-zd992" (UID: "b9a976d1-7bb2-47d6-b285-788a6ff40cef") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:39:19.781863 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:19.781830 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzjjn\" (UniqueName: \"kubernetes.io/projected/b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa-kube-api-access-nzjjn\") pod \"network-check-target-xgvpv\" (UID: \"b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa\") " pod="openshift-network-diagnostics/network-check-target-xgvpv" Jul 16 12:39:19.782016 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:19.781875 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd6ebe0-1750-4a12-a4c7-023cfadcbabc-metrics-certs\") pod \"network-metrics-daemon-q5cp8\" (UID: \"fdd6ebe0-1750-4a12-a4c7-023cfadcbabc\") " pod="openshift-multus/network-metrics-daemon-q5cp8" Jul 16 12:39:19.782016 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:19.781976 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:39:19.782016 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:19.781996 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:39:19.782016 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:19.782015 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:39:19.782142 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:19.782025 2582 projected.go:196] Error preparing data for projected volume kube-api-access-nzjjn for pod openshift-network-diagnostics/network-check-target-xgvpv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:39:19.782142 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:19.782028 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd6ebe0-1750-4a12-a4c7-023cfadcbabc-metrics-certs podName:fdd6ebe0-1750-4a12-a4c7-023cfadcbabc nodeName:}" failed. No retries permitted until 2026-07-16 12:39:35.782014886 +0000 UTC m=+34.259760480 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdd6ebe0-1750-4a12-a4c7-023cfadcbabc-metrics-certs") pod "network-metrics-daemon-q5cp8" (UID: "fdd6ebe0-1750-4a12-a4c7-023cfadcbabc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:39:19.782142 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:19.782064 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa-kube-api-access-nzjjn podName:b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa nodeName:}" failed. No retries permitted until 2026-07-16 12:39:35.782052813 +0000 UTC m=+34.259798426 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nzjjn" (UniqueName: "kubernetes.io/projected/b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa-kube-api-access-nzjjn") pod "network-check-target-xgvpv" (UID: "b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:39:19.971638 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:19.971567 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:39:19.971792 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:19.971705 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd992" podUID="b9a976d1-7bb2-47d6-b285-788a6ff40cef" Jul 16 12:39:20.972243 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:20.972210 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5cp8" Jul 16 12:39:20.972690 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:20.972340 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5cp8" podUID="fdd6ebe0-1750-4a12-a4c7-023cfadcbabc" Jul 16 12:39:20.972690 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:20.972412 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgvpv" Jul 16 12:39:20.972690 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:20.972496 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgvpv" podUID="b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa" Jul 16 12:39:21.972609 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:21.972575 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:39:21.973046 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:21.972697 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd992" podUID="b9a976d1-7bb2-47d6-b285-788a6ff40cef" Jul 16 12:39:22.972130 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:22.972097 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5cp8" Jul 16 12:39:22.972130 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:22.972121 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgvpv" Jul 16 12:39:22.972350 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:22.972217 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5cp8" podUID="fdd6ebe0-1750-4a12-a4c7-023cfadcbabc" Jul 16 12:39:22.972350 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:22.972330 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgvpv" podUID="b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa" Jul 16 12:39:23.972197 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:23.972155 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:39:23.972709 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:23.972299 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd992" podUID="b9a976d1-7bb2-47d6-b285-788a6ff40cef" Jul 16 12:39:24.971745 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:24.971582 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgvpv" Jul 16 12:39:24.971921 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:24.971649 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5cp8" Jul 16 12:39:24.971921 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:24.971815 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgvpv" podUID="b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa" Jul 16 12:39:24.971994 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:24.971942 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5cp8" podUID="fdd6ebe0-1750-4a12-a4c7-023cfadcbabc" Jul 16 12:39:25.047746 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:25.047275 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lv8x6" event={"ID":"7e332a21-d3fd-468d-a704-8a98f55f1596","Type":"ContainerStarted","Data":"509e297f214f7841b5c31a8bc0bfa22fbb5e7cf8949f7b664762ba60f7a096df"} Jul 16 12:39:25.048989 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:25.048959 2582 generic.go:356] "Generic (PLEG): container finished" podID="eec32a5cfb1937ffec65aa300c58f36d" containerID="ff352f28b3f01719c36acfb5324cfc2288d954d5e5e56c899be6dcbbb804edcd" exitCode=0 Jul 16 12:39:25.049116 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:25.049046 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal" event={"ID":"eec32a5cfb1937ffec65aa300c58f36d","Type":"ContainerDied","Data":"ff352f28b3f01719c36acfb5324cfc2288d954d5e5e56c899be6dcbbb804edcd"} Jul 16 12:39:25.050551 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:25.050524 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-130.ec2.internal" event={"ID":"42e2219b6620be69c895eb04c7d0d0b4","Type":"ContainerStarted","Data":"f4b9f4fa964da903ddb7a682223a364830df55ec6ce952467669ec178b51c533"} Jul 16 12:39:25.052582 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:25.052545 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5s6sz" event={"ID":"431ba1c9-15f3-4a03-ab1d-8a2b983a0704","Type":"ContainerStarted","Data":"3acbaaf7412dc7eb317aa5ad5140d85db5cdf7552ad76d143be63ad48595ccb4"} Jul 16 12:39:25.054156 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:25.054132 2582 generic.go:356] "Generic (PLEG): container finished" podID="04c215b4-7f84-4763-9ad8-3db823a6c2c6" containerID="a9a80ea93472a384da79d471eba797f2db9108deca42f9fdf9391833d3c0c675" exitCode=0 Jul 16 12:39:25.054246 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:25.054203 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m8pdn" event={"ID":"04c215b4-7f84-4763-9ad8-3db823a6c2c6","Type":"ContainerDied","Data":"a9a80ea93472a384da79d471eba797f2db9108deca42f9fdf9391833d3c0c675"} Jul 16 12:39:25.055883 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:25.055862 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-76lb5" event={"ID":"b95cb0aa-837a-41a4-913c-7e3b15f04fdf","Type":"ContainerStarted","Data":"e80e6c71d61ecf3b8f495d0b5319a242c07ca78b85a07bb65727b25d88cb1c86"} Jul 16 12:39:25.059142 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:25.059123 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-986g4" event={"ID":"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d","Type":"ContainerStarted","Data":"410975d3b5460717492131420835f3e30d8877116c76cfa374ff811d2c5bd94a"} Jul 16 12:39:25.059215 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:25.059194 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-986g4" event={"ID":"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d","Type":"ContainerStarted","Data":"420d608b0fe1a8492a2f9dd551d31766f6ca3966ecaecede12b4376446b886ed"} Jul 16 12:39:25.059256 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:25.059237 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-986g4" event={"ID":"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d","Type":"ContainerStarted","Data":"7fc34e94f104aa50b5c93e669f94cbf37861b1f5aea43bf4cb2fd6370f2074b2"} Jul 16 12:39:25.059293 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:25.059266 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-986g4" event={"ID":"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d","Type":"ContainerStarted","Data":"9982ffece29dbd0ec92f374714c98040cef6e1ba4e4a670a7a4f8e71b3fe456c"} Jul 16 12:39:25.059334 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:25.059298 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-986g4" event={"ID":"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d","Type":"ContainerStarted","Data":"214a92142944368e544cd486c47efbd7463340ae068cd136836445281f14236e"} Jul 16 12:39:25.059367 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:25.059355 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-986g4" event={"ID":"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d","Type":"ContainerStarted","Data":"552b2a53cf6f8735d1fe36311de1ec039efcb5f7e4928850ba6cc09b4ff3c157"} Jul 16 12:39:25.061153 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:25.061128 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gdt" event={"ID":"95be9185-aa73-4b64-9301-04e5ca0c661e","Type":"ContainerStarted","Data":"d3d53cbd9d1b13f6cb2632ba4dde52dd7a781b96b84e77ce5f33ccf0ca461f18"} Jul 16 12:39:25.062575 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:25.062556 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4ns96" event={"ID":"a80d1676-7c47-4427-992c-234fbc36cf7b","Type":"ContainerStarted","Data":"71d4ad6302af6e9db59eaf40a1de90c5c18b3e3db72e5e6e339641174624b01b"} Jul 16 12:39:25.063949 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:25.063931 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" event={"ID":"e6d8dd65-4edb-485a-90d0-6e98e5c075fe","Type":"ContainerStarted","Data":"15a5f00dc4459f4b9c3b5bdcf7f1d9186cf38131e447a4414611f6e7deb57b80"} Jul 16 12:39:25.066170 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:25.066125 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lv8x6" podStartSLOduration=3.218077692 podStartE2EDuration="23.066114097s" totalImagesPullingTime="19.848036405s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:39:25.06524841 +0000 UTC m=+23.542994049" watchObservedRunningTime="2026-07-16 12:39:25.066114097 +0000 UTC m=+23.543859713" Jul 16 12:39:25.095025 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:25.094985 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-130.ec2.internal" podStartSLOduration=22.094968199 podStartE2EDuration="22.094968199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:39:25.094593129 +0000 UTC m=+23.572338761" watchObservedRunningTime="2026-07-16 12:39:25.094968199 +0000 UTC m=+23.572713816" Jul 16 12:39:25.133882 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:25.133844 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-76lb5" podStartSLOduration=3.292923779 podStartE2EDuration="23.133827845s" totalImagesPullingTime="19.840904066s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:39:25.109697636 +0000 UTC m=+23.587443253" watchObservedRunningTime="2026-07-16 12:39:25.133827845 +0000 UTC m=+23.611573466" Jul 16 12:39:25.158448 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:25.158407 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5s6sz" podStartSLOduration=3.280681165 podStartE2EDuration="23.158375049s" totalImagesPullingTime="19.877693884s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:39:25.158161455 +0000 UTC m=+23.635907072" watchObservedRunningTime="2026-07-16 12:39:25.158375049 +0000 UTC m=+23.636120669" Jul 16 12:39:25.180205 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:25.180154 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4ns96" podStartSLOduration=3.362948795 podStartE2EDuration="23.18014197s" totalImagesPullingTime="19.817193175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:39:25.179743174 +0000 UTC m=+23.657488792" watchObservedRunningTime="2026-07-16 12:39:25.18014197 +0000 UTC m=+23.657887586" Jul 16 12:39:25.794748 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:25.794717 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4ns96" Jul 16 12:39:25.795289 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:25.795259 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4ns96" Jul 16 12:39:25.812568 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:25.812507 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dc8zj" podStartSLOduration=3.990544319 podStartE2EDuration="23.812487956s" totalImagesPullingTime="19.821943637s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:39:25.196980508 +0000 UTC m=+23.674726136" watchObservedRunningTime="2026-07-16 12:39:25.812487956 +0000 UTC m=+24.290233623" Jul 16 12:39:25.893851 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:25.893831 2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 16 12:39:25.900804 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:25.900702 2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T12:39:25.893849718Z","UUID":"b0acb419-1762-4917-b6ff-091727f08186","Handler":null,"Name":"","Endpoint":""} Jul 16 12:39:25.902101 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:25.902084 2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 16 12:39:25.902184 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:25.902120 2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 16 12:39:25.971503 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:25.971476 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:39:25.971668 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:25.971602 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd992" podUID="b9a976d1-7bb2-47d6-b285-788a6ff40cef" Jul 16 12:39:26.068470 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:26.068380 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gdt" event={"ID":"95be9185-aa73-4b64-9301-04e5ca0c661e","Type":"ContainerStarted","Data":"fd0250b6a993596cf943d2e0d89888dce0c7cd449ebcab364944e3a768a70738"} Jul 16 12:39:26.069987 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:26.069951 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mv2w2" event={"ID":"a30e0419-799d-45fb-a586-f06018bc1ee4","Type":"ContainerStarted","Data":"d9eeef643033237156d7e26b7b4019010c308564858f577492753cace7bfb8d3"} Jul 16 12:39:26.071795 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:26.071765 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal" event={"ID":"eec32a5cfb1937ffec65aa300c58f36d","Type":"ContainerStarted","Data":"eb25e18103e51ebf83e0b75be8b575ce35c8c80eb79332c668c14dab8f913bf8"} Jul 16 12:39:26.085293 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:26.085243 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mv2w2" podStartSLOduration=4.201632808 podStartE2EDuration="24.085228849s" totalImagesPullingTime="19.883596041s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:39:26.085071917 +0000 UTC m=+24.562817536" watchObservedRunningTime="2026-07-16 12:39:26.085228849 +0000 UTC m=+24.562974467" Jul 16 12:39:26.099703 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:26.099666 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-130.ec2.internal" podStartSLOduration=23.099648568 podStartE2EDuration="23.099648568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:39:26.099345159 +0000 UTC m=+24.577090774" watchObservedRunningTime="2026-07-16 12:39:26.099648568 +0000 UTC m=+24.577394186" Jul 16 12:39:26.972135 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:26.972107 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgvpv" Jul 16 12:39:26.972286 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:26.972107 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5cp8" Jul 16 12:39:26.972286 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:26.972249 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgvpv" podUID="b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa" Jul 16 12:39:26.972462 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:26.972314 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5cp8" podUID="fdd6ebe0-1750-4a12-a4c7-023cfadcbabc" Jul 16 12:39:27.079231 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:27.079202 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-986g4" event={"ID":"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d","Type":"ContainerStarted","Data":"3762c75e5b5d7ccfa87d799c7787ac27e8462e15f6bad51e24b48c1b550f6a91"} Jul 16 12:39:27.079677 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:27.079466 2582 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 16 12:39:27.249413 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:27.249371 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4ns96" Jul 16 12:39:27.250073 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:27.250049 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4ns96" Jul 16 12:39:27.971723 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:27.971686 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:39:27.971909 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:27.971815 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd992" podUID="b9a976d1-7bb2-47d6-b285-788a6ff40cef" Jul 16 12:39:28.971694 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:28.971533 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgvpv" Jul 16 12:39:28.972111 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:28.971594 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5cp8" Jul 16 12:39:28.972111 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:28.971779 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgvpv" podUID="b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa" Jul 16 12:39:28.972111 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:28.971871 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5cp8" podUID="fdd6ebe0-1750-4a12-a4c7-023cfadcbabc" Jul 16 12:39:29.972143 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:29.971971 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:39:29.972767 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:29.972212 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd992" podUID="b9a976d1-7bb2-47d6-b285-788a6ff40cef" Jul 16 12:39:30.087128 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:30.087096 2582 generic.go:356] "Generic (PLEG): container finished" podID="04c215b4-7f84-4763-9ad8-3db823a6c2c6" containerID="64e9a425d72bbb53e8fb0ce1c2dc51e2ba8aa9fac2024002d137b15b3db42b2e" exitCode=0 Jul 16 12:39:30.087308 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:30.087183 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m8pdn" event={"ID":"04c215b4-7f84-4763-9ad8-3db823a6c2c6","Type":"ContainerDied","Data":"64e9a425d72bbb53e8fb0ce1c2dc51e2ba8aa9fac2024002d137b15b3db42b2e"} Jul 16 12:39:30.090657 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:30.090635 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-986g4" event={"ID":"ffe9c110-1bbe-42a5-bd97-b2f3c70e680d","Type":"ContainerStarted","Data":"0866a341d040d80ed4bf34eef7f8f2d69b72142856f80fbb0bc3fc874e57dc81"} Jul 16 12:39:30.090890 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:30.090873 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:30.090989 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:30.090931 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:30.090989 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:30.090979 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:30.106476 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:30.106452 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:30.106588 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:30.106559 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:39:30.146235 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:30.145953 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-986g4" podStartSLOduration=8.217828612 podStartE2EDuration="28.14593359s" totalImagesPullingTime="19.928104978s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:39:30.144680045 +0000 UTC m=+28.622425686" watchObservedRunningTime="2026-07-16 12:39:30.14593359 +0000 UTC m=+28.623679208" Jul 16 12:39:30.971481 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:30.971444 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgvpv" Jul 16 12:39:30.971688 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:30.971452 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5cp8" Jul 16 12:39:30.971688 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:30.971594 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgvpv" podUID="b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa" Jul 16 12:39:30.971688 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:30.971656 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5cp8" podUID="fdd6ebe0-1750-4a12-a4c7-023cfadcbabc" Jul 16 12:39:31.095568 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:31.095525 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gdt" event={"ID":"95be9185-aa73-4b64-9301-04e5ca0c661e","Type":"ContainerStarted","Data":"12c809e9f96e64ea6cd3dcc92f4154d7560bd6371e59dba29235f97fed860a8f"} Jul 16 12:39:31.114867 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:31.114817 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r6gdt" podStartSLOduration=3.8298420650000002 podStartE2EDuration="29.114800349s" totalImagesPullingTime="25.284958284s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:39:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:39:31.114209488 +0000 UTC m=+29.591955104" watchObservedRunningTime="2026-07-16 12:39:31.114800349 +0000 UTC m=+29.592545965" Jul 16 12:39:31.519197 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:31.519120 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zd992"] Jul 16 12:39:31.519367 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:31.519262 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:39:31.519367 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:31.519353 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd992" podUID="b9a976d1-7bb2-47d6-b285-788a6ff40cef" Jul 16 12:39:31.529966 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:31.529938 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xgvpv"] Jul 16 12:39:31.530089 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:31.530034 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgvpv" Jul 16 12:39:31.530135 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:31.530117 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgvpv" podUID="b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa" Jul 16 12:39:31.532728 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:31.532705 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q5cp8"] Jul 16 12:39:31.532840 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:31.532787 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5cp8" Jul 16 12:39:31.532909 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:31.532882 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5cp8" podUID="fdd6ebe0-1750-4a12-a4c7-023cfadcbabc" Jul 16 12:39:32.102186 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:32.102147 2582 generic.go:356] "Generic (PLEG): container finished" podID="04c215b4-7f84-4763-9ad8-3db823a6c2c6" containerID="31de4fc2fb6ca2859860187e0d4858808ad93658ebe6db8788568da439f629a0" exitCode=0 Jul 16 12:39:32.102639 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:32.102226 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m8pdn" event={"ID":"04c215b4-7f84-4763-9ad8-3db823a6c2c6","Type":"ContainerDied","Data":"31de4fc2fb6ca2859860187e0d4858808ad93658ebe6db8788568da439f629a0"} Jul 16 12:39:32.972152 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:32.972115 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5cp8" Jul 16 12:39:32.972152 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:32.972155 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgvpv" Jul 16 12:39:32.972343 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:32.972245 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5cp8" podUID="fdd6ebe0-1750-4a12-a4c7-023cfadcbabc" Jul 16 12:39:32.972377 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:32.972348 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgvpv" podUID="b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa" Jul 16 12:39:33.971848 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:33.971812 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:39:33.972294 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:33.971913 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd992" podUID="b9a976d1-7bb2-47d6-b285-788a6ff40cef" Jul 16 12:39:34.109473 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:34.109436 2582 generic.go:356] "Generic (PLEG): container finished" podID="04c215b4-7f84-4763-9ad8-3db823a6c2c6" containerID="d7a512bf7cb1a8a3b25ea9a97ca17a7f30d8a745307f6a1ef760db88e9806423" exitCode=0 Jul 16 12:39:34.109636 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:34.109499 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m8pdn" event={"ID":"04c215b4-7f84-4763-9ad8-3db823a6c2c6","Type":"ContainerDied","Data":"d7a512bf7cb1a8a3b25ea9a97ca17a7f30d8a745307f6a1ef760db88e9806423"} Jul 16 12:39:34.971692 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:34.971513 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgvpv" Jul 16 12:39:34.971870 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:34.971556 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5cp8" Jul 16 12:39:34.971870 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:34.971785 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgvpv" podUID="b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa" Jul 16 12:39:34.971870 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:34.971858 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5cp8" podUID="fdd6ebe0-1750-4a12-a4c7-023cfadcbabc" Jul 16 12:39:35.700618 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:35.700518 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9a976d1-7bb2-47d6-b285-788a6ff40cef-original-pull-secret\") pod \"global-pull-secret-syncer-zd992\" (UID: \"b9a976d1-7bb2-47d6-b285-788a6ff40cef\") " pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:39:35.700782 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:35.700646 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:39:35.700782 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:35.700702 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9a976d1-7bb2-47d6-b285-788a6ff40cef-original-pull-secret podName:b9a976d1-7bb2-47d6-b285-788a6ff40cef nodeName:}" failed. No retries permitted until 2026-07-16 12:40:07.700685716 +0000 UTC m=+66.178431329 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b9a976d1-7bb2-47d6-b285-788a6ff40cef-original-pull-secret") pod "global-pull-secret-syncer-zd992" (UID: "b9a976d1-7bb2-47d6-b285-788a6ff40cef") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:39:35.800871 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:35.800838 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd6ebe0-1750-4a12-a4c7-023cfadcbabc-metrics-certs\") pod \"network-metrics-daemon-q5cp8\" (UID: \"fdd6ebe0-1750-4a12-a4c7-023cfadcbabc\") " pod="openshift-multus/network-metrics-daemon-q5cp8" Jul 16 12:39:35.801096 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:35.800918 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzjjn\" (UniqueName: \"kubernetes.io/projected/b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa-kube-api-access-nzjjn\") pod \"network-check-target-xgvpv\" (UID: \"b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa\") " pod="openshift-network-diagnostics/network-check-target-xgvpv" Jul 16 12:39:35.801096 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:35.801080 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:39:35.801207 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:35.801150 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:39:35.801207 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:35.801165 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:39:35.801207 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:35.801174 2582 projected.go:196] Error preparing data for projected volume kube-api-access-nzjjn for pod openshift-network-diagnostics/network-check-target-xgvpv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:39:35.801207 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:35.801189 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd6ebe0-1750-4a12-a4c7-023cfadcbabc-metrics-certs podName:fdd6ebe0-1750-4a12-a4c7-023cfadcbabc nodeName:}" failed. No retries permitted until 2026-07-16 12:40:07.801165148 +0000 UTC m=+66.278910757 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdd6ebe0-1750-4a12-a4c7-023cfadcbabc-metrics-certs") pod "network-metrics-daemon-q5cp8" (UID: "fdd6ebe0-1750-4a12-a4c7-023cfadcbabc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:39:35.801439 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:35.801225 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa-kube-api-access-nzjjn podName:b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa nodeName:}" failed. No retries permitted until 2026-07-16 12:40:07.801210872 +0000 UTC m=+66.278956488 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nzjjn" (UniqueName: "kubernetes.io/projected/b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa-kube-api-access-nzjjn") pod "network-check-target-xgvpv" (UID: "b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:39:35.971582 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:35.971494 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:39:35.971742 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:35.971622 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd992" podUID="b9a976d1-7bb2-47d6-b285-788a6ff40cef" Jul 16 12:39:36.871154 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:36.871076 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-130.ec2.internal" event="NodeReady" Jul 16 12:39:36.871635 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:36.871242 2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 16 12:39:36.917973 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:36.917942 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-cb5b8dfd-qvxd2"] Jul 16 12:39:36.920577 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:36.920557 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:39:36.923089 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:36.923065 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zz4g5\"" Jul 16 12:39:36.923240 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:36.923099 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 16 12:39:36.923240 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:36.923066 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 16 12:39:36.923548 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:36.923531 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 16 12:39:36.930678 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:36.928576 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qsszd"] Jul 16 12:39:36.930678 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:36.929039 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 16 12:39:36.931442 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:36.931145 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qsszd" Jul 16 12:39:36.933312 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:36.933289 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-r5ftr"] Jul 16 12:39:36.934163 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:36.933969 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4kt9r\"" Jul 16 12:39:36.934163 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:36.933990 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 16 12:39:36.934163 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:36.934158 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 16 12:39:36.936307 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:36.936288 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cb5b8dfd-qvxd2"] Jul 16 12:39:36.936422 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:36.936408 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r5ftr" Jul 16 12:39:36.938932 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:36.938915 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 16 12:39:36.939350 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:36.939326 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qr4zr\"" Jul 16 12:39:36.939590 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:36.939573 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 16 12:39:36.939843 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:36.939816 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 16 12:39:36.940566 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:36.940536 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qsszd"] Jul 16 12:39:36.946379 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:36.946359 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r5ftr"] Jul 16 12:39:36.971843 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:36.971813 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5cp8" Jul 16 12:39:36.971975 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:36.971958 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgvpv" Jul 16 12:39:36.974434 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:36.974406 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ff9kh\"" Jul 16 12:39:36.974537 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:36.974468 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8df2z\"" Jul 16 12:39:36.974537 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:36.974487 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 12:39:36.974955 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:36.974734 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 12:39:36.974955 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:36.974740 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 12:39:37.009556 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.009524 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b9a1a7fc-4864-48d9-8b45-477fedb54b17-registry-certificates\") pod \"image-registry-cb5b8dfd-qvxd2\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:39:37.009704 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.009587 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b9a1a7fc-4864-48d9-8b45-477fedb54b17-image-registry-private-configuration\") pod \"image-registry-cb5b8dfd-qvxd2\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:39:37.009704 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.009658 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b9a1a7fc-4864-48d9-8b45-477fedb54b17-ca-trust-extracted\") pod \"image-registry-cb5b8dfd-qvxd2\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:39:37.009704 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.009693 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-registry-tls\") pod \"image-registry-cb5b8dfd-qvxd2\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:39:37.009846 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.009718 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtslp\" (UniqueName: \"kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-kube-api-access-wtslp\") pod \"image-registry-cb5b8dfd-qvxd2\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:39:37.009846 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.009743 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b9a1a7fc-4864-48d9-8b45-477fedb54b17-installation-pull-secrets\") pod \"image-registry-cb5b8dfd-qvxd2\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:39:37.009846 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.009778 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9a1a7fc-4864-48d9-8b45-477fedb54b17-trusted-ca\") pod \"image-registry-cb5b8dfd-qvxd2\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:39:37.009962 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.009838 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-bound-sa-token\") pod \"image-registry-cb5b8dfd-qvxd2\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:39:37.090878 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.090845 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:39:37.110898 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.110870 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b9a1a7fc-4864-48d9-8b45-477fedb54b17-image-registry-private-configuration\") pod \"image-registry-cb5b8dfd-qvxd2\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:39:37.112447 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.110912 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8-metrics-tls\") pod \"dns-default-qsszd\" (UID: \"8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8\") " pod="openshift-dns/dns-default-qsszd" Jul 16 12:39:37.112447 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.110939 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7769852e-c7d8-4f0e-b59e-8b5dd8035f4d-cert\") pod \"ingress-canary-r5ftr\" (UID: \"7769852e-c7d8-4f0e-b59e-8b5dd8035f4d\") " pod="openshift-ingress-canary/ingress-canary-r5ftr" Jul 16 12:39:37.112447 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.110994 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b9a1a7fc-4864-48d9-8b45-477fedb54b17-ca-trust-extracted\") pod \"image-registry-cb5b8dfd-qvxd2\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:39:37.112447 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.111022 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-registry-tls\") pod \"image-registry-cb5b8dfd-qvxd2\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:39:37.112447 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.111048 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtslp\" (UniqueName: \"kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-kube-api-access-wtslp\") pod \"image-registry-cb5b8dfd-qvxd2\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:39:37.112447 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.111075 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b9a1a7fc-4864-48d9-8b45-477fedb54b17-installation-pull-secrets\") pod \"image-registry-cb5b8dfd-qvxd2\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:39:37.112447 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.111101 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkkhg\" (UniqueName: \"kubernetes.io/projected/8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8-kube-api-access-kkkhg\") pod \"dns-default-qsszd\" (UID: \"8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8\") " pod="openshift-dns/dns-default-qsszd" Jul 16 12:39:37.112447 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.111152 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9a1a7fc-4864-48d9-8b45-477fedb54b17-trusted-ca\") pod \"image-registry-cb5b8dfd-qvxd2\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:39:37.112447 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.111173 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-bound-sa-token\") pod \"image-registry-cb5b8dfd-qvxd2\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:39:37.112447 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.111193 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8-config-volume\") pod \"dns-default-qsszd\" (UID: \"8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8\") " pod="openshift-dns/dns-default-qsszd" Jul 16 12:39:37.112447 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:37.111196 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 12:39:37.112447 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:37.111216 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cb5b8dfd-qvxd2: secret "image-registry-tls" not found Jul 16 12:39:37.112447 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.111235 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8-tmp-dir\") pod \"dns-default-qsszd\" (UID: \"8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8\") " pod="openshift-dns/dns-default-qsszd" Jul 16 12:39:37.112447 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:37.111275 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-registry-tls podName:b9a1a7fc-4864-48d9-8b45-477fedb54b17 nodeName:}" failed. No retries permitted until 2026-07-16 12:39:37.611256592 +0000 UTC m=+36.089002207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-registry-tls") pod "image-registry-cb5b8dfd-qvxd2" (UID: "b9a1a7fc-4864-48d9-8b45-477fedb54b17") : secret "image-registry-tls" not found Jul 16 12:39:37.112447 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.111295 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b9a1a7fc-4864-48d9-8b45-477fedb54b17-registry-certificates\") pod \"image-registry-cb5b8dfd-qvxd2\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:39:37.112447 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.111326 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpb5z\" (UniqueName: \"kubernetes.io/projected/7769852e-c7d8-4f0e-b59e-8b5dd8035f4d-kube-api-access-xpb5z\") pod \"ingress-canary-r5ftr\" (UID: \"7769852e-c7d8-4f0e-b59e-8b5dd8035f4d\") " pod="openshift-ingress-canary/ingress-canary-r5ftr" Jul 16 12:39:37.112447 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.111482 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b9a1a7fc-4864-48d9-8b45-477fedb54b17-ca-trust-extracted\") pod \"image-registry-cb5b8dfd-qvxd2\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:39:37.113202 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.112121 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b9a1a7fc-4864-48d9-8b45-477fedb54b17-registry-certificates\") pod \"image-registry-cb5b8dfd-qvxd2\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:39:37.113202 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.112289 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9a1a7fc-4864-48d9-8b45-477fedb54b17-trusted-ca\") pod \"image-registry-cb5b8dfd-qvxd2\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:39:37.115632 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.115614 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b9a1a7fc-4864-48d9-8b45-477fedb54b17-image-registry-private-configuration\") pod \"image-registry-cb5b8dfd-qvxd2\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:39:37.115745 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.115697 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b9a1a7fc-4864-48d9-8b45-477fedb54b17-installation-pull-secrets\") pod \"image-registry-cb5b8dfd-qvxd2\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:39:37.119653 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.119632 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-bound-sa-token\") pod \"image-registry-cb5b8dfd-qvxd2\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:39:37.120018 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.119988 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtslp\" (UniqueName: \"kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-kube-api-access-wtslp\") pod \"image-registry-cb5b8dfd-qvxd2\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:39:37.204822 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.204781 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:39:37.211887 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.211856 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkkhg\" (UniqueName: \"kubernetes.io/projected/8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8-kube-api-access-kkkhg\") pod \"dns-default-qsszd\" (UID: \"8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8\") " pod="openshift-dns/dns-default-qsszd" Jul 16 12:39:37.212042 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.211912 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8-config-volume\") pod \"dns-default-qsszd\" (UID: \"8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8\") " pod="openshift-dns/dns-default-qsszd" Jul 16 12:39:37.212042 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.212019 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8-tmp-dir\") pod \"dns-default-qsszd\" (UID: \"8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8\") " pod="openshift-dns/dns-default-qsszd" Jul 16 12:39:37.212125 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.212046 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpb5z\" (UniqueName: \"kubernetes.io/projected/7769852e-c7d8-4f0e-b59e-8b5dd8035f4d-kube-api-access-xpb5z\") pod \"ingress-canary-r5ftr\" (UID: \"7769852e-c7d8-4f0e-b59e-8b5dd8035f4d\") " pod="openshift-ingress-canary/ingress-canary-r5ftr" Jul 16 12:39:37.212125 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.212076 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8-metrics-tls\") pod \"dns-default-qsszd\" (UID: \"8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8\") " pod="openshift-dns/dns-default-qsszd" Jul 16 12:39:37.212125 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.212093 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7769852e-c7d8-4f0e-b59e-8b5dd8035f4d-cert\") pod \"ingress-canary-r5ftr\" (UID: \"7769852e-c7d8-4f0e-b59e-8b5dd8035f4d\") " pod="openshift-ingress-canary/ingress-canary-r5ftr" Jul 16 12:39:37.212235 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:37.212184 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:39:37.212268 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:37.212246 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8-metrics-tls podName:8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8 nodeName:}" failed. No retries permitted until 2026-07-16 12:39:37.712227829 +0000 UTC m=+36.189973437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8-metrics-tls") pod "dns-default-qsszd" (UID: "8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8") : secret "dns-default-metrics-tls" not found Jul 16 12:39:37.212315 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.212293 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8-tmp-dir\") pod \"dns-default-qsszd\" (UID: \"8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8\") " pod="openshift-dns/dns-default-qsszd" Jul 16 12:39:37.212586 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.212534 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8-config-volume\") pod \"dns-default-qsszd\" (UID: \"8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8\") " pod="openshift-dns/dns-default-qsszd" Jul 16 12:39:37.214996 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.214973 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7769852e-c7d8-4f0e-b59e-8b5dd8035f4d-cert\") pod \"ingress-canary-r5ftr\" (UID: \"7769852e-c7d8-4f0e-b59e-8b5dd8035f4d\") " pod="openshift-ingress-canary/ingress-canary-r5ftr" Jul 16 12:39:37.226949 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.226928 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkkhg\" (UniqueName: \"kubernetes.io/projected/8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8-kube-api-access-kkkhg\") pod \"dns-default-qsszd\" (UID: \"8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8\") " pod="openshift-dns/dns-default-qsszd" Jul 16 12:39:37.227063 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.226980 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpb5z\" (UniqueName: \"kubernetes.io/projected/7769852e-c7d8-4f0e-b59e-8b5dd8035f4d-kube-api-access-xpb5z\") pod \"ingress-canary-r5ftr\" (UID: \"7769852e-c7d8-4f0e-b59e-8b5dd8035f4d\") " pod="openshift-ingress-canary/ingress-canary-r5ftr" Jul 16 12:39:37.254269 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.254235 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r5ftr" Jul 16 12:39:37.282920 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.282859 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:39:37.616156 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.616075 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-registry-tls\") pod \"image-registry-cb5b8dfd-qvxd2\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:39:37.616292 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:37.616247 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 12:39:37.616292 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:37.616266 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cb5b8dfd-qvxd2: secret "image-registry-tls" not found Jul 16 12:39:37.616364 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:37.616326 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-registry-tls podName:b9a1a7fc-4864-48d9-8b45-477fedb54b17 nodeName:}" failed. No retries permitted until 2026-07-16 12:39:38.616311055 +0000 UTC m=+37.094056672 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-registry-tls") pod "image-registry-cb5b8dfd-qvxd2" (UID: "b9a1a7fc-4864-48d9-8b45-477fedb54b17") : secret "image-registry-tls" not found Jul 16 12:39:37.716734 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.716694 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8-metrics-tls\") pod \"dns-default-qsszd\" (UID: \"8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8\") " pod="openshift-dns/dns-default-qsszd" Jul 16 12:39:37.716907 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:37.716871 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:39:37.716965 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:37.716945 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8-metrics-tls podName:8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8 nodeName:}" failed. No retries permitted until 2026-07-16 12:39:38.716924367 +0000 UTC m=+37.194669965 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8-metrics-tls") pod "dns-default-qsszd" (UID: "8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8") : secret "dns-default-metrics-tls" not found Jul 16 12:39:37.975491 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.975450 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:39:37.978100 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:37.978072 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 12:39:38.139621 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:38.139574 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r5ftr"] Jul 16 12:39:38.143502 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:38.143477 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7769852e_c7d8_4f0e_b59e_8b5dd8035f4d.slice/crio-1c80135704f2e5634b5e62a1a9f7becbc378432daafdef677cec3916dd0dee20 WatchSource:0}: Error finding container 1c80135704f2e5634b5e62a1a9f7becbc378432daafdef677cec3916dd0dee20: Status 404 returned error can't find the container with id 1c80135704f2e5634b5e62a1a9f7becbc378432daafdef677cec3916dd0dee20 Jul 16 12:39:38.625219 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:38.625192 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-registry-tls\") pod \"image-registry-cb5b8dfd-qvxd2\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:39:38.625404 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:38.625368 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 12:39:38.625404 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:38.625402 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cb5b8dfd-qvxd2: secret "image-registry-tls" not found Jul 16 12:39:38.625478 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:38.625460 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-registry-tls podName:b9a1a7fc-4864-48d9-8b45-477fedb54b17 nodeName:}" failed. No retries permitted until 2026-07-16 12:39:40.625444628 +0000 UTC m=+39.103190223 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-registry-tls") pod "image-registry-cb5b8dfd-qvxd2" (UID: "b9a1a7fc-4864-48d9-8b45-477fedb54b17") : secret "image-registry-tls" not found Jul 16 12:39:38.726658 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:38.726619 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8-metrics-tls\") pod \"dns-default-qsszd\" (UID: \"8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8\") " pod="openshift-dns/dns-default-qsszd" Jul 16 12:39:38.726842 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:38.726800 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:39:38.726912 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:38.726876 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8-metrics-tls podName:8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8 nodeName:}" failed. No retries permitted until 2026-07-16 12:39:40.726853393 +0000 UTC m=+39.204598998 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8-metrics-tls") pod "dns-default-qsszd" (UID: "8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8") : secret "dns-default-metrics-tls" not found Jul 16 12:39:39.128570 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:39.128454 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r5ftr" event={"ID":"7769852e-c7d8-4f0e-b59e-8b5dd8035f4d","Type":"ContainerStarted","Data":"1c80135704f2e5634b5e62a1a9f7becbc378432daafdef677cec3916dd0dee20"} Jul 16 12:39:39.131328 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:39.131300 2582 generic.go:356] "Generic (PLEG): container finished" podID="04c215b4-7f84-4763-9ad8-3db823a6c2c6" containerID="a7ba63ccb90838caf05881d11cab9a21fb157029c97865a183756454e21b1eed" exitCode=0 Jul 16 12:39:39.131475 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:39.131339 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m8pdn" event={"ID":"04c215b4-7f84-4763-9ad8-3db823a6c2c6","Type":"ContainerDied","Data":"a7ba63ccb90838caf05881d11cab9a21fb157029c97865a183756454e21b1eed"} Jul 16 12:39:39.175720 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:39.175689 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:39:39.231866 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:39.231816 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:39:40.137270 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:40.137239 2582 generic.go:356] "Generic (PLEG): container finished" podID="04c215b4-7f84-4763-9ad8-3db823a6c2c6" containerID="5f8e73d03b5e7f9260124550dfe044c68d2147719c5263f62ed4ac7889d43f46" exitCode=0 Jul 16 12:39:40.137717 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:40.137275 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m8pdn" event={"ID":"04c215b4-7f84-4763-9ad8-3db823a6c2c6","Type":"ContainerDied","Data":"5f8e73d03b5e7f9260124550dfe044c68d2147719c5263f62ed4ac7889d43f46"} Jul 16 12:39:40.640072 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:40.640048 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-registry-tls\") pod \"image-registry-cb5b8dfd-qvxd2\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:39:40.640238 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:40.640216 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 12:39:40.640279 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:40.640242 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cb5b8dfd-qvxd2: secret "image-registry-tls" not found Jul 16 12:39:40.640324 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:40.640305 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-registry-tls podName:b9a1a7fc-4864-48d9-8b45-477fedb54b17 nodeName:}" failed. No retries permitted until 2026-07-16 12:39:44.640286717 +0000 UTC m=+43.118032325 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-registry-tls") pod "image-registry-cb5b8dfd-qvxd2" (UID: "b9a1a7fc-4864-48d9-8b45-477fedb54b17") : secret "image-registry-tls" not found Jul 16 12:39:40.741044 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:40.741004 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8-metrics-tls\") pod \"dns-default-qsszd\" (UID: \"8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8\") " pod="openshift-dns/dns-default-qsszd" Jul 16 12:39:40.741198 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:40.741182 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:39:40.741286 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:40.741279 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8-metrics-tls podName:8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8 nodeName:}" failed. No retries permitted until 2026-07-16 12:39:44.741235867 +0000 UTC m=+43.218981477 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8-metrics-tls") pod "dns-default-qsszd" (UID: "8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8") : secret "dns-default-metrics-tls" not found Jul 16 12:39:41.145816 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:41.145777 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m8pdn" event={"ID":"04c215b4-7f84-4763-9ad8-3db823a6c2c6","Type":"ContainerStarted","Data":"d49567d4eb7570c16450336e740f8cc4a2820c7f919f34e9cda5c3f1f16d1761"} Jul 16 12:39:41.147174 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:41.147149 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r5ftr" event={"ID":"7769852e-c7d8-4f0e-b59e-8b5dd8035f4d","Type":"ContainerStarted","Data":"deb2cac997f6b3aaab09d67853c6d9d9ca0fda13ce4aae3ee737c6be0aaa2d18"} Jul 16 12:39:41.169719 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:41.169630 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-m8pdn" podStartSLOduration=8.988907051 podStartE2EDuration="39.169614511s" totalImagesPullingTime="30.18070746s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:39:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:39:41.168580726 +0000 UTC m=+39.646326352" watchObservedRunningTime="2026-07-16 12:39:41.169614511 +0000 UTC m=+39.647360187" Jul 16 12:39:41.184852 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:41.184815 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-r5ftr" podStartSLOduration=2.520193721 podStartE2EDuration="5.184803659s" totalImagesPullingTime="2.664609938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:39:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:39:41.18449417 +0000 UTC m=+39.662239809" watchObservedRunningTime="2026-07-16 12:39:41.184803659 +0000 UTC m=+39.662549274" Jul 16 12:39:41.300298 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:41.300142 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:39:41.400559 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:41.400530 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:39:42.158383 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:42.158354 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:39:42.259444 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:42.259377 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:39:44.671230 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:44.671191 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-registry-tls\") pod \"image-registry-cb5b8dfd-qvxd2\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:39:44.671614 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:44.671337 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 12:39:44.671614 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:44.671354 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cb5b8dfd-qvxd2: secret "image-registry-tls" not found Jul 16 12:39:44.671614 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:44.671432 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-registry-tls podName:b9a1a7fc-4864-48d9-8b45-477fedb54b17 nodeName:}" failed. No retries permitted until 2026-07-16 12:39:52.671414268 +0000 UTC m=+51.149159883 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-registry-tls") pod "image-registry-cb5b8dfd-qvxd2" (UID: "b9a1a7fc-4864-48d9-8b45-477fedb54b17") : secret "image-registry-tls" not found Jul 16 12:39:44.772537 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:44.772502 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8-metrics-tls\") pod \"dns-default-qsszd\" (UID: \"8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8\") " pod="openshift-dns/dns-default-qsszd" Jul 16 12:39:44.772679 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:44.772648 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:39:44.772717 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:44.772706 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8-metrics-tls podName:8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8 nodeName:}" failed. No retries permitted until 2026-07-16 12:39:52.772689288 +0000 UTC m=+51.250434902 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8-metrics-tls") pod "dns-default-qsszd" (UID: "8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8") : secret "dns-default-metrics-tls" not found Jul 16 12:39:46.057298 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:46.057265 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:39:46.282691 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:46.282663 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:39:46.904462 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:46.904417 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:39:47.004535 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:47.004506 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:39:49.349003 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:49.348974 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:39:49.409900 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:49.409863 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:39:51.979084 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:51.979057 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:39:51.991400 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:51.991364 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-44js4"] Jul 16 12:39:52.014155 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.014127 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-2fzwz"] Jul 16 12:39:52.014270 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.014243 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-44js4" Jul 16 12:39:52.016575 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.016557 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 16 12:39:52.016575 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.016594 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-hbp27\"" Jul 16 12:39:52.017552 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.017535 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 16 12:39:52.027375 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.027314 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56vw7\" (UniqueName: \"kubernetes.io/projected/f2085320-522b-4e21-ba87-b92e42ed1f78-kube-api-access-56vw7\") pod \"volume-data-source-validator-78495597f6-44js4\" (UID: \"f2085320-522b-4e21-ba87-b92e42ed1f78\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-44js4" Jul 16 12:39:52.038405 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.038196 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-44js4"] Jul 16 12:39:52.038405 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.038322 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-2fzwz" Jul 16 12:39:52.040924 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.040796 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-2fzwz"] Jul 16 12:39:52.040924 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.040813 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 16 12:39:52.040924 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.040892 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dgl9h\"" Jul 16 12:39:52.040924 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.040900 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 16 12:39:52.040924 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.040895 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 16 12:39:52.041264 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.041030 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 16 12:39:52.058269 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.058249 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 16 12:39:52.097274 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.097247 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:39:52.128465 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.128436 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5dd68711-4227-4f4c-b6b9-2b4eb11b7478-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-2fzwz\" (UID: \"5dd68711-4227-4f4c-b6b9-2b4eb11b7478\") " pod="openshift-insights/insights-operator-68d9dd94cf-2fzwz" Jul 16 12:39:52.128579 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.128495 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5dd68711-4227-4f4c-b6b9-2b4eb11b7478-serving-cert\") pod \"insights-operator-68d9dd94cf-2fzwz\" (UID: \"5dd68711-4227-4f4c-b6b9-2b4eb11b7478\") " pod="openshift-insights/insights-operator-68d9dd94cf-2fzwz" Jul 16 12:39:52.128579 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.128539 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56vw7\" (UniqueName: \"kubernetes.io/projected/f2085320-522b-4e21-ba87-b92e42ed1f78-kube-api-access-56vw7\") pod \"volume-data-source-validator-78495597f6-44js4\" (UID: \"f2085320-522b-4e21-ba87-b92e42ed1f78\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-44js4" Jul 16 12:39:52.128579 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.128564 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5dd68711-4227-4f4c-b6b9-2b4eb11b7478-tmp\") pod \"insights-operator-68d9dd94cf-2fzwz\" (UID: \"5dd68711-4227-4f4c-b6b9-2b4eb11b7478\") " pod="openshift-insights/insights-operator-68d9dd94cf-2fzwz" Jul 16 12:39:52.128737 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.128662 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5dd68711-4227-4f4c-b6b9-2b4eb11b7478-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-2fzwz\" (UID: \"5dd68711-4227-4f4c-b6b9-2b4eb11b7478\") " pod="openshift-insights/insights-operator-68d9dd94cf-2fzwz" Jul 16 12:39:52.128737 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.128689 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9bnn\" (UniqueName: \"kubernetes.io/projected/5dd68711-4227-4f4c-b6b9-2b4eb11b7478-kube-api-access-t9bnn\") pod \"insights-operator-68d9dd94cf-2fzwz\" (UID: \"5dd68711-4227-4f4c-b6b9-2b4eb11b7478\") " pod="openshift-insights/insights-operator-68d9dd94cf-2fzwz" Jul 16 12:39:52.128825 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.128742 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5dd68711-4227-4f4c-b6b9-2b4eb11b7478-snapshots\") pod \"insights-operator-68d9dd94cf-2fzwz\" (UID: \"5dd68711-4227-4f4c-b6b9-2b4eb11b7478\") " pod="openshift-insights/insights-operator-68d9dd94cf-2fzwz" Jul 16 12:39:52.140916 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.140895 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56vw7\" (UniqueName: \"kubernetes.io/projected/f2085320-522b-4e21-ba87-b92e42ed1f78-kube-api-access-56vw7\") pod \"volume-data-source-validator-78495597f6-44js4\" (UID: \"f2085320-522b-4e21-ba87-b92e42ed1f78\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-44js4" Jul 16 12:39:52.229491 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.229430 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5dd68711-4227-4f4c-b6b9-2b4eb11b7478-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-2fzwz\" (UID: \"5dd68711-4227-4f4c-b6b9-2b4eb11b7478\") " pod="openshift-insights/insights-operator-68d9dd94cf-2fzwz" Jul 16 12:39:52.229491 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.229460 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9bnn\" (UniqueName: \"kubernetes.io/projected/5dd68711-4227-4f4c-b6b9-2b4eb11b7478-kube-api-access-t9bnn\") pod \"insights-operator-68d9dd94cf-2fzwz\" (UID: \"5dd68711-4227-4f4c-b6b9-2b4eb11b7478\") " pod="openshift-insights/insights-operator-68d9dd94cf-2fzwz" Jul 16 12:39:52.229639 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.229495 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5dd68711-4227-4f4c-b6b9-2b4eb11b7478-snapshots\") pod \"insights-operator-68d9dd94cf-2fzwz\" (UID: \"5dd68711-4227-4f4c-b6b9-2b4eb11b7478\") " pod="openshift-insights/insights-operator-68d9dd94cf-2fzwz" Jul 16 12:39:52.229639 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.229571 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5dd68711-4227-4f4c-b6b9-2b4eb11b7478-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-2fzwz\" (UID: \"5dd68711-4227-4f4c-b6b9-2b4eb11b7478\") " pod="openshift-insights/insights-operator-68d9dd94cf-2fzwz" Jul 16 12:39:52.229639 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.229615 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5dd68711-4227-4f4c-b6b9-2b4eb11b7478-serving-cert\") pod \"insights-operator-68d9dd94cf-2fzwz\" (UID: \"5dd68711-4227-4f4c-b6b9-2b4eb11b7478\") " pod="openshift-insights/insights-operator-68d9dd94cf-2fzwz" Jul 16 12:39:52.229759 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.229742 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5dd68711-4227-4f4c-b6b9-2b4eb11b7478-tmp\") pod \"insights-operator-68d9dd94cf-2fzwz\" (UID: \"5dd68711-4227-4f4c-b6b9-2b4eb11b7478\") " pod="openshift-insights/insights-operator-68d9dd94cf-2fzwz" Jul 16 12:39:52.230110 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.230082 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5dd68711-4227-4f4c-b6b9-2b4eb11b7478-tmp\") pod \"insights-operator-68d9dd94cf-2fzwz\" (UID: \"5dd68711-4227-4f4c-b6b9-2b4eb11b7478\") " pod="openshift-insights/insights-operator-68d9dd94cf-2fzwz" Jul 16 12:39:52.230205 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.230091 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5dd68711-4227-4f4c-b6b9-2b4eb11b7478-snapshots\") pod \"insights-operator-68d9dd94cf-2fzwz\" (UID: \"5dd68711-4227-4f4c-b6b9-2b4eb11b7478\") " pod="openshift-insights/insights-operator-68d9dd94cf-2fzwz" Jul 16 12:39:52.230205 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.230162 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5dd68711-4227-4f4c-b6b9-2b4eb11b7478-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-2fzwz\" (UID: \"5dd68711-4227-4f4c-b6b9-2b4eb11b7478\") " pod="openshift-insights/insights-operator-68d9dd94cf-2fzwz" Jul 16 12:39:52.230315 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.230300 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5dd68711-4227-4f4c-b6b9-2b4eb11b7478-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-2fzwz\" (UID: \"5dd68711-4227-4f4c-b6b9-2b4eb11b7478\") " pod="openshift-insights/insights-operator-68d9dd94cf-2fzwz" Jul 16 12:39:52.231861 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.231841 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5dd68711-4227-4f4c-b6b9-2b4eb11b7478-serving-cert\") pod \"insights-operator-68d9dd94cf-2fzwz\" (UID: \"5dd68711-4227-4f4c-b6b9-2b4eb11b7478\") " pod="openshift-insights/insights-operator-68d9dd94cf-2fzwz" Jul 16 12:39:52.237121 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.237099 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9bnn\" (UniqueName: \"kubernetes.io/projected/5dd68711-4227-4f4c-b6b9-2b4eb11b7478-kube-api-access-t9bnn\") pod \"insights-operator-68d9dd94cf-2fzwz\" (UID: \"5dd68711-4227-4f4c-b6b9-2b4eb11b7478\") " pod="openshift-insights/insights-operator-68d9dd94cf-2fzwz" Jul 16 12:39:52.322690 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.322663 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-44js4" Jul 16 12:39:52.349046 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.349025 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-2fzwz" Jul 16 12:39:52.481259 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.481194 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-44js4"] Jul 16 12:39:52.501704 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.501679 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-2fzwz"] Jul 16 12:39:52.504339 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:52.504314 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd68711_4227_4f4c_b6b9_2b4eb11b7478.slice/crio-4065b86ba5df7b1728bcd90c06c7e1ec3ce66471e46f957baf25b912dc62b507 WatchSource:0}: Error finding container 4065b86ba5df7b1728bcd90c06c7e1ec3ce66471e46f957baf25b912dc62b507: Status 404 returned error can't find the container with id 4065b86ba5df7b1728bcd90c06c7e1ec3ce66471e46f957baf25b912dc62b507 Jul 16 12:39:52.733145 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.733063 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-registry-tls\") pod \"image-registry-cb5b8dfd-qvxd2\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:39:52.733288 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:52.733227 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 12:39:52.733288 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:52.733243 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cb5b8dfd-qvxd2: secret "image-registry-tls" not found Jul 16 12:39:52.733357 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:52.733299 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-registry-tls podName:b9a1a7fc-4864-48d9-8b45-477fedb54b17 nodeName:}" failed. No retries permitted until 2026-07-16 12:40:08.733281711 +0000 UTC m=+67.211027320 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-registry-tls") pod "image-registry-cb5b8dfd-qvxd2" (UID: "b9a1a7fc-4864-48d9-8b45-477fedb54b17") : secret "image-registry-tls" not found Jul 16 12:39:52.834290 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:52.834252 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8-metrics-tls\") pod \"dns-default-qsszd\" (UID: \"8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8\") " pod="openshift-dns/dns-default-qsszd" Jul 16 12:39:52.834468 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:52.834408 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:39:52.834468 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:52.834468 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8-metrics-tls podName:8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8 nodeName:}" failed. No retries permitted until 2026-07-16 12:40:08.834453378 +0000 UTC m=+67.312198971 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8-metrics-tls") pod "dns-default-qsszd" (UID: "8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8") : secret "dns-default-metrics-tls" not found Jul 16 12:39:53.175773 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:53.175701 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-2fzwz" event={"ID":"5dd68711-4227-4f4c-b6b9-2b4eb11b7478","Type":"ContainerStarted","Data":"4065b86ba5df7b1728bcd90c06c7e1ec3ce66471e46f957baf25b912dc62b507"} Jul 16 12:39:53.176974 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:53.176934 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-44js4" event={"ID":"f2085320-522b-4e21-ba87-b92e42ed1f78","Type":"ContainerStarted","Data":"f1561c1eeaf6344681b499cd009698415d4a777136958983d05375cf08c8dfff"} Jul 16 12:39:55.182088 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:55.182060 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-44js4" event={"ID":"f2085320-522b-4e21-ba87-b92e42ed1f78","Type":"ContainerStarted","Data":"19023b88f315d6d1b74b9517c9ae056d5e249eacb0bdab3819db67b34ee0a0a2"} Jul 16 12:39:55.201532 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:55.201492 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-44js4" podStartSLOduration=2.560430375 podStartE2EDuration="4.201477728s" totalImagesPullingTime="1.641047353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:39:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:39:55.201187552 +0000 UTC m=+53.678933158" watchObservedRunningTime="2026-07-16 12:39:55.201477728 +0000 UTC m=+53.679223343" Jul 16 12:39:55.794812 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:55.794778 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-h7g92"] Jul 16 12:39:55.809586 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:55.809560 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-h7g92"] Jul 16 12:39:55.809707 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:55.809672 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-h7g92" Jul 16 12:39:55.812477 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:55.812448 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 16 12:39:55.812477 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:55.812460 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 16 12:39:55.813447 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:55.813424 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-7vbgk\"" Jul 16 12:39:55.813554 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:55.813464 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 16 12:39:55.813554 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:55.813465 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 16 12:39:55.860206 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:55.860180 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8fd6e2f8-4515-47e3-94e9-32bc0f1a9e75-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-h7g92\" (UID: \"8fd6e2f8-4515-47e3-94e9-32bc0f1a9e75\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-h7g92" Jul 16 12:39:55.860350 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:55.860246 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8fd6e2f8-4515-47e3-94e9-32bc0f1a9e75-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-h7g92\" (UID: \"8fd6e2f8-4515-47e3-94e9-32bc0f1a9e75\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-h7g92" Jul 16 12:39:55.860350 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:55.860265 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwnmg\" (UniqueName: \"kubernetes.io/projected/8fd6e2f8-4515-47e3-94e9-32bc0f1a9e75-kube-api-access-nwnmg\") pod \"kube-storage-version-migrator-operator-799cb7f958-h7g92\" (UID: \"8fd6e2f8-4515-47e3-94e9-32bc0f1a9e75\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-h7g92" Jul 16 12:39:55.961051 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:55.961016 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8fd6e2f8-4515-47e3-94e9-32bc0f1a9e75-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-h7g92\" (UID: \"8fd6e2f8-4515-47e3-94e9-32bc0f1a9e75\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-h7g92" Jul 16 12:39:55.961179 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:55.961084 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8fd6e2f8-4515-47e3-94e9-32bc0f1a9e75-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-h7g92\" (UID: \"8fd6e2f8-4515-47e3-94e9-32bc0f1a9e75\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-h7g92" Jul 16 12:39:55.961179 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:55.961106 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwnmg\" (UniqueName: \"kubernetes.io/projected/8fd6e2f8-4515-47e3-94e9-32bc0f1a9e75-kube-api-access-nwnmg\") pod \"kube-storage-version-migrator-operator-799cb7f958-h7g92\" (UID: \"8fd6e2f8-4515-47e3-94e9-32bc0f1a9e75\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-h7g92" Jul 16 12:39:55.961570 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:55.961549 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8fd6e2f8-4515-47e3-94e9-32bc0f1a9e75-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-h7g92\" (UID: \"8fd6e2f8-4515-47e3-94e9-32bc0f1a9e75\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-h7g92" Jul 16 12:39:55.963489 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:55.963457 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8fd6e2f8-4515-47e3-94e9-32bc0f1a9e75-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-h7g92\" (UID: \"8fd6e2f8-4515-47e3-94e9-32bc0f1a9e75\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-h7g92" Jul 16 12:39:55.968980 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:55.968952 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwnmg\" (UniqueName: \"kubernetes.io/projected/8fd6e2f8-4515-47e3-94e9-32bc0f1a9e75-kube-api-access-nwnmg\") pod \"kube-storage-version-migrator-operator-799cb7f958-h7g92\" (UID: \"8fd6e2f8-4515-47e3-94e9-32bc0f1a9e75\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-h7g92" Jul 16 12:39:56.118632 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:56.118560 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-h7g92" Jul 16 12:39:56.186760 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:56.186723 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-2fzwz" event={"ID":"5dd68711-4227-4f4c-b6b9-2b4eb11b7478","Type":"ContainerStarted","Data":"88d4313d1081b6be46aefcdac5a9641966ca1d97ac3c924a3691ba79afaf2e18"} Jul 16 12:39:56.216486 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:56.216214 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-2fzwz" podStartSLOduration=2.595606107 podStartE2EDuration="5.216195353s" totalImagesPullingTime="2.620589246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:39:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:39:56.215817065 +0000 UTC m=+54.693562705" watchObservedRunningTime="2026-07-16 12:39:56.216195353 +0000 UTC m=+54.693940966" Jul 16 12:39:56.286570 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:56.286543 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-h7g92"] Jul 16 12:39:56.289280 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:56.289256 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fd6e2f8_4515_47e3_94e9_32bc0f1a9e75.slice/crio-63ed20b0a1cfe45fae9a6ead9d36833eb846b3667f603603ba304816f7b417e5 WatchSource:0}: Error finding container 63ed20b0a1cfe45fae9a6ead9d36833eb846b3667f603603ba304816f7b417e5: Status 404 returned error can't find the container with id 63ed20b0a1cfe45fae9a6ead9d36833eb846b3667f603603ba304816f7b417e5 Jul 16 12:39:56.439070 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:56.439035 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:39:56.528555 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:56.528526 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:39:57.070469 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:57.070433 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fdthr"] Jul 16 12:39:57.072416 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:57.072381 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fdthr" Jul 16 12:39:57.074778 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:57.074759 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 16 12:39:57.074872 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:57.074842 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 16 12:39:57.075869 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:57.075848 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9wswp\"" Jul 16 12:39:57.099853 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:57.099828 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fdthr"] Jul 16 12:39:57.153016 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:57.152984 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:39:57.171356 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:57.171304 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bda475ac-da25-423d-a7ca-ebc9c29b1720-data-volume\") pod \"insights-runtime-extractor-fdthr\" (UID: \"bda475ac-da25-423d-a7ca-ebc9c29b1720\") " pod="openshift-insights/insights-runtime-extractor-fdthr" Jul 16 12:39:57.171534 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:57.171375 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bda475ac-da25-423d-a7ca-ebc9c29b1720-crio-socket\") pod \"insights-runtime-extractor-fdthr\" (UID: \"bda475ac-da25-423d-a7ca-ebc9c29b1720\") " pod="openshift-insights/insights-runtime-extractor-fdthr" Jul 16 12:39:57.171534 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:57.171432 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bda475ac-da25-423d-a7ca-ebc9c29b1720-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fdthr\" (UID: \"bda475ac-da25-423d-a7ca-ebc9c29b1720\") " pod="openshift-insights/insights-runtime-extractor-fdthr" Jul 16 12:39:57.171534 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:57.171465 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bda475ac-da25-423d-a7ca-ebc9c29b1720-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fdthr\" (UID: \"bda475ac-da25-423d-a7ca-ebc9c29b1720\") " pod="openshift-insights/insights-runtime-extractor-fdthr" Jul 16 12:39:57.171656 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:57.171619 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9cfr\" (UniqueName: \"kubernetes.io/projected/bda475ac-da25-423d-a7ca-ebc9c29b1720-kube-api-access-f9cfr\") pod \"insights-runtime-extractor-fdthr\" (UID: \"bda475ac-da25-423d-a7ca-ebc9c29b1720\") " pod="openshift-insights/insights-runtime-extractor-fdthr" Jul 16 12:39:57.190736 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:57.190698 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-h7g92" event={"ID":"8fd6e2f8-4515-47e3-94e9-32bc0f1a9e75","Type":"ContainerStarted","Data":"63ed20b0a1cfe45fae9a6ead9d36833eb846b3667f603603ba304816f7b417e5"} Jul 16 12:39:57.229813 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:57.229782 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:39:57.272511 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:57.272479 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bda475ac-da25-423d-a7ca-ebc9c29b1720-data-volume\") pod \"insights-runtime-extractor-fdthr\" (UID: \"bda475ac-da25-423d-a7ca-ebc9c29b1720\") " pod="openshift-insights/insights-runtime-extractor-fdthr" Jul 16 12:39:57.272726 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:57.272514 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bda475ac-da25-423d-a7ca-ebc9c29b1720-crio-socket\") pod \"insights-runtime-extractor-fdthr\" (UID: \"bda475ac-da25-423d-a7ca-ebc9c29b1720\") " pod="openshift-insights/insights-runtime-extractor-fdthr" Jul 16 12:39:57.272726 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:57.272544 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bda475ac-da25-423d-a7ca-ebc9c29b1720-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fdthr\" (UID: \"bda475ac-da25-423d-a7ca-ebc9c29b1720\") " pod="openshift-insights/insights-runtime-extractor-fdthr" Jul 16 12:39:57.272726 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:57.272570 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bda475ac-da25-423d-a7ca-ebc9c29b1720-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fdthr\" (UID: \"bda475ac-da25-423d-a7ca-ebc9c29b1720\") " pod="openshift-insights/insights-runtime-extractor-fdthr" Jul 16 12:39:57.272726 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:57.272625 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bda475ac-da25-423d-a7ca-ebc9c29b1720-crio-socket\") pod \"insights-runtime-extractor-fdthr\" (UID: \"bda475ac-da25-423d-a7ca-ebc9c29b1720\") " pod="openshift-insights/insights-runtime-extractor-fdthr" Jul 16 12:39:57.272726 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:57.272648 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 12:39:57.272726 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:57.272706 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 12:39:57.272726 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:57.272722 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bda475ac-da25-423d-a7ca-ebc9c29b1720-insights-runtime-extractor-tls podName:bda475ac-da25-423d-a7ca-ebc9c29b1720 nodeName:}" failed. No retries permitted until 2026-07-16 12:39:57.772703023 +0000 UTC m=+56.250448621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bda475ac-da25-423d-a7ca-ebc9c29b1720-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fdthr" (UID: "bda475ac-da25-423d-a7ca-ebc9c29b1720") : secret "insights-runtime-extractor-tls" not found Jul 16 12:39:57.273128 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:57.272783 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bda475ac-da25-423d-a7ca-ebc9c29b1720-kube-rbac-proxy-cm podName:bda475ac-da25-423d-a7ca-ebc9c29b1720 nodeName:}" failed. No retries permitted until 2026-07-16 12:39:57.772764997 +0000 UTC m=+56.250510629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bda475ac-da25-423d-a7ca-ebc9c29b1720-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fdthr" (UID: "bda475ac-da25-423d-a7ca-ebc9c29b1720") : configmap "kube-rbac-proxy" not found Jul 16 12:39:57.273128 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:57.272772 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9cfr\" (UniqueName: \"kubernetes.io/projected/bda475ac-da25-423d-a7ca-ebc9c29b1720-kube-api-access-f9cfr\") pod \"insights-runtime-extractor-fdthr\" (UID: \"bda475ac-da25-423d-a7ca-ebc9c29b1720\") " pod="openshift-insights/insights-runtime-extractor-fdthr" Jul 16 12:39:57.273128 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:57.272845 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bda475ac-da25-423d-a7ca-ebc9c29b1720-data-volume\") pod \"insights-runtime-extractor-fdthr\" (UID: \"bda475ac-da25-423d-a7ca-ebc9c29b1720\") " pod="openshift-insights/insights-runtime-extractor-fdthr" Jul 16 12:39:57.284279 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:57.284252 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9cfr\" (UniqueName: \"kubernetes.io/projected/bda475ac-da25-423d-a7ca-ebc9c29b1720-kube-api-access-f9cfr\") pod \"insights-runtime-extractor-fdthr\" (UID: \"bda475ac-da25-423d-a7ca-ebc9c29b1720\") " pod="openshift-insights/insights-runtime-extractor-fdthr" Jul 16 12:39:57.776860 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:57.776809 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bda475ac-da25-423d-a7ca-ebc9c29b1720-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fdthr\" (UID: \"bda475ac-da25-423d-a7ca-ebc9c29b1720\") " pod="openshift-insights/insights-runtime-extractor-fdthr" Jul 16 12:39:57.776860 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:57.776854 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bda475ac-da25-423d-a7ca-ebc9c29b1720-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fdthr\" (UID: \"bda475ac-da25-423d-a7ca-ebc9c29b1720\") " pod="openshift-insights/insights-runtime-extractor-fdthr" Jul 16 12:39:57.777152 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:57.776929 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 12:39:57.777152 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:57.776979 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 12:39:57.777152 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:57.777067 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bda475ac-da25-423d-a7ca-ebc9c29b1720-insights-runtime-extractor-tls podName:bda475ac-da25-423d-a7ca-ebc9c29b1720 nodeName:}" failed. No retries permitted until 2026-07-16 12:39:58.777028231 +0000 UTC m=+57.254773843 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bda475ac-da25-423d-a7ca-ebc9c29b1720-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fdthr" (UID: "bda475ac-da25-423d-a7ca-ebc9c29b1720") : secret "insights-runtime-extractor-tls" not found Jul 16 12:39:57.777571 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:57.777554 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bda475ac-da25-423d-a7ca-ebc9c29b1720-kube-rbac-proxy-cm podName:bda475ac-da25-423d-a7ca-ebc9c29b1720 nodeName:}" failed. No retries permitted until 2026-07-16 12:39:58.777077773 +0000 UTC m=+57.254823373 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bda475ac-da25-423d-a7ca-ebc9c29b1720-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fdthr" (UID: "bda475ac-da25-423d-a7ca-ebc9c29b1720") : configmap "kube-rbac-proxy" not found Jul 16 12:39:58.675469 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:58.675433 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-dlwd4"] Jul 16 12:39:58.677758 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:58.677738 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-dlwd4" Jul 16 12:39:58.680228 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:58.680210 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-stgn5\"" Jul 16 12:39:58.695500 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:58.695459 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-dlwd4"] Jul 16 12:39:58.784424 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:58.784346 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bda475ac-da25-423d-a7ca-ebc9c29b1720-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fdthr\" (UID: \"bda475ac-da25-423d-a7ca-ebc9c29b1720\") " pod="openshift-insights/insights-runtime-extractor-fdthr" Jul 16 12:39:58.784424 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:58.784378 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bda475ac-da25-423d-a7ca-ebc9c29b1720-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fdthr\" (UID: \"bda475ac-da25-423d-a7ca-ebc9c29b1720\") " pod="openshift-insights/insights-runtime-extractor-fdthr" Jul 16 12:39:58.784578 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:58.784443 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmwz8\" (UniqueName: \"kubernetes.io/projected/32b71e49-e327-42c8-a37f-6e0db3090bc7-kube-api-access-fmwz8\") pod \"network-check-source-869dcfcdcd-dlwd4\" (UID: \"32b71e49-e327-42c8-a37f-6e0db3090bc7\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-dlwd4" Jul 16 12:39:58.784578 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:58.784520 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 12:39:58.784578 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:58.784546 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 12:39:58.784683 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:58.784601 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bda475ac-da25-423d-a7ca-ebc9c29b1720-kube-rbac-proxy-cm podName:bda475ac-da25-423d-a7ca-ebc9c29b1720 nodeName:}" failed. No retries permitted until 2026-07-16 12:40:00.784581828 +0000 UTC m=+59.262327444 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bda475ac-da25-423d-a7ca-ebc9c29b1720-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fdthr" (UID: "bda475ac-da25-423d-a7ca-ebc9c29b1720") : configmap "kube-rbac-proxy" not found Jul 16 12:39:58.784683 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:39:58.784618 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bda475ac-da25-423d-a7ca-ebc9c29b1720-insights-runtime-extractor-tls podName:bda475ac-da25-423d-a7ca-ebc9c29b1720 nodeName:}" failed. No retries permitted until 2026-07-16 12:40:00.784609335 +0000 UTC m=+59.262354929 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bda475ac-da25-423d-a7ca-ebc9c29b1720-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fdthr" (UID: "bda475ac-da25-423d-a7ca-ebc9c29b1720") : secret "insights-runtime-extractor-tls" not found Jul 16 12:39:58.786480 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:58.786453 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-js9ms"] Jul 16 12:39:58.788216 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:58.788201 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-js9ms" Jul 16 12:39:58.790540 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:58.790519 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 16 12:39:58.790829 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:58.790807 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 16 12:39:58.790915 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:58.790852 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 16 12:39:58.791079 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:58.791066 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 16 12:39:58.791123 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:58.791094 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-fcgqb\"" Jul 16 12:39:58.852613 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:58.852581 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-js9ms"] Jul 16 12:39:58.870157 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:58.870136 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lv8x6_7e332a21-d3fd-468d-a704-8a98f55f1596/dns-node-resolver/0.log" Jul 16 12:39:58.885634 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:58.885609 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmwz8\" (UniqueName: \"kubernetes.io/projected/32b71e49-e327-42c8-a37f-6e0db3090bc7-kube-api-access-fmwz8\") pod \"network-check-source-869dcfcdcd-dlwd4\" (UID: \"32b71e49-e327-42c8-a37f-6e0db3090bc7\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-dlwd4" Jul 16 12:39:58.885719 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:58.885668 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/04daa141-3460-4290-98db-eccefb64fe38-config\") pod \"service-ca-operator-7695fc6c8c-js9ms\" (UID: \"04daa141-3460-4290-98db-eccefb64fe38\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-js9ms" Jul 16 12:39:58.885719 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:58.885694 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gbqc\" (UniqueName: \"kubernetes.io/projected/04daa141-3460-4290-98db-eccefb64fe38-kube-api-access-5gbqc\") pod \"service-ca-operator-7695fc6c8c-js9ms\" (UID: \"04daa141-3460-4290-98db-eccefb64fe38\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-js9ms" Jul 16 12:39:58.885819 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:58.885744 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/04daa141-3460-4290-98db-eccefb64fe38-serving-cert\") pod \"service-ca-operator-7695fc6c8c-js9ms\" (UID: \"04daa141-3460-4290-98db-eccefb64fe38\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-js9ms" Jul 16 12:39:58.893676 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:58.893646 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmwz8\" (UniqueName: \"kubernetes.io/projected/32b71e49-e327-42c8-a37f-6e0db3090bc7-kube-api-access-fmwz8\") pod \"network-check-source-869dcfcdcd-dlwd4\" (UID: \"32b71e49-e327-42c8-a37f-6e0db3090bc7\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-dlwd4" Jul 16 12:39:58.986294 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:58.986269 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/04daa141-3460-4290-98db-eccefb64fe38-config\") pod \"service-ca-operator-7695fc6c8c-js9ms\" (UID: \"04daa141-3460-4290-98db-eccefb64fe38\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-js9ms" Jul 16 12:39:58.986469 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:58.986299 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-dlwd4" Jul 16 12:39:58.986469 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:58.986307 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gbqc\" (UniqueName: \"kubernetes.io/projected/04daa141-3460-4290-98db-eccefb64fe38-kube-api-access-5gbqc\") pod \"service-ca-operator-7695fc6c8c-js9ms\" (UID: \"04daa141-3460-4290-98db-eccefb64fe38\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-js9ms" Jul 16 12:39:58.986469 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:58.986362 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/04daa141-3460-4290-98db-eccefb64fe38-serving-cert\") pod \"service-ca-operator-7695fc6c8c-js9ms\" (UID: \"04daa141-3460-4290-98db-eccefb64fe38\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-js9ms" Jul 16 12:39:58.986818 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:58.986800 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/04daa141-3460-4290-98db-eccefb64fe38-config\") pod \"service-ca-operator-7695fc6c8c-js9ms\" (UID: \"04daa141-3460-4290-98db-eccefb64fe38\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-js9ms" Jul 16 12:39:58.988795 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:58.988775 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/04daa141-3460-4290-98db-eccefb64fe38-serving-cert\") pod \"service-ca-operator-7695fc6c8c-js9ms\" (UID: \"04daa141-3460-4290-98db-eccefb64fe38\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-js9ms" Jul 16 12:39:58.994252 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:58.994230 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gbqc\" (UniqueName: \"kubernetes.io/projected/04daa141-3460-4290-98db-eccefb64fe38-kube-api-access-5gbqc\") pod \"service-ca-operator-7695fc6c8c-js9ms\" (UID: \"04daa141-3460-4290-98db-eccefb64fe38\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-js9ms" Jul 16 12:39:59.096953 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:59.096912 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-js9ms" Jul 16 12:39:59.109866 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:59.109840 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-dlwd4"] Jul 16 12:39:59.112443 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:59.112416 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32b71e49_e327_42c8_a37f_6e0db3090bc7.slice/crio-a2bd029ce6039813a5887a23f05691ad5d1db4a557c1017c494bc9672f73ef9b WatchSource:0}: Error finding container a2bd029ce6039813a5887a23f05691ad5d1db4a557c1017c494bc9672f73ef9b: Status 404 returned error can't find the container with id a2bd029ce6039813a5887a23f05691ad5d1db4a557c1017c494bc9672f73ef9b Jul 16 12:39:59.198311 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:59.198280 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-h7g92" event={"ID":"8fd6e2f8-4515-47e3-94e9-32bc0f1a9e75","Type":"ContainerStarted","Data":"8b479232cd84482720db65b957c5fd1fe400014d03b60eb0996fefe7112d3429"} Jul 16 12:39:59.199979 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:59.199950 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-dlwd4" event={"ID":"32b71e49-e327-42c8-a37f-6e0db3090bc7","Type":"ContainerStarted","Data":"a2bd029ce6039813a5887a23f05691ad5d1db4a557c1017c494bc9672f73ef9b"} Jul 16 12:39:59.217811 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:59.217766 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-h7g92" podStartSLOduration=2.026956663 podStartE2EDuration="4.217752839s" totalImagesPullingTime="2.190796176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:39:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:39:59.216828614 +0000 UTC m=+57.694574231" watchObservedRunningTime="2026-07-16 12:39:59.217752839 +0000 UTC m=+57.695498433" Jul 16 12:39:59.224967 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:59.224942 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-js9ms"] Jul 16 12:39:59.228485 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:39:59.228453 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04daa141_3460_4290_98db_eccefb64fe38.slice/crio-ca66fdbc562104d3b9d8cc9378b2d8c226c42acc4379d8faf17db021116a031b WatchSource:0}: Error finding container ca66fdbc562104d3b9d8cc9378b2d8c226c42acc4379d8faf17db021116a031b: Status 404 returned error can't find the container with id ca66fdbc562104d3b9d8cc9378b2d8c226c42acc4379d8faf17db021116a031b Jul 16 12:39:59.470609 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:59.470584 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-76lb5_b95cb0aa-837a-41a4-913c-7e3b15f04fdf/node-ca/0.log" Jul 16 12:39:59.803498 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:59.803362 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:39:59.903640 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:39:59.903582 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:00.115490 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:00.115352 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:00.206187 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:00.205668 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-js9ms" event={"ID":"04daa141-3460-4290-98db-eccefb64fe38","Type":"ContainerStarted","Data":"ca66fdbc562104d3b9d8cc9378b2d8c226c42acc4379d8faf17db021116a031b"} Jul 16 12:40:00.206760 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:00.206714 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:00.265481 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:00.265382 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:00.270321 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:00.270297 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r5ftr_7769852e-c7d8-4f0e-b59e-8b5dd8035f4d/serve-healthcheck-canary/0.log" Jul 16 12:40:00.801994 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:00.801958 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bda475ac-da25-423d-a7ca-ebc9c29b1720-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fdthr\" (UID: \"bda475ac-da25-423d-a7ca-ebc9c29b1720\") " pod="openshift-insights/insights-runtime-extractor-fdthr" Jul 16 12:40:00.802166 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:00.802003 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bda475ac-da25-423d-a7ca-ebc9c29b1720-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fdthr\" (UID: \"bda475ac-da25-423d-a7ca-ebc9c29b1720\") " pod="openshift-insights/insights-runtime-extractor-fdthr" Jul 16 12:40:00.802166 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:40:00.802130 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 12:40:00.802268 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:40:00.802190 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bda475ac-da25-423d-a7ca-ebc9c29b1720-kube-rbac-proxy-cm podName:bda475ac-da25-423d-a7ca-ebc9c29b1720 nodeName:}" failed. No retries permitted until 2026-07-16 12:40:04.802174507 +0000 UTC m=+63.279920114 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bda475ac-da25-423d-a7ca-ebc9c29b1720-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fdthr" (UID: "bda475ac-da25-423d-a7ca-ebc9c29b1720") : configmap "kube-rbac-proxy" not found Jul 16 12:40:00.802268 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:40:00.802258 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 12:40:00.802356 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:40:00.802292 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bda475ac-da25-423d-a7ca-ebc9c29b1720-insights-runtime-extractor-tls podName:bda475ac-da25-423d-a7ca-ebc9c29b1720 nodeName:}" failed. No retries permitted until 2026-07-16 12:40:04.80228422 +0000 UTC m=+63.280029826 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bda475ac-da25-423d-a7ca-ebc9c29b1720-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fdthr" (UID: "bda475ac-da25-423d-a7ca-ebc9c29b1720") : secret "insights-runtime-extractor-tls" not found Jul 16 12:40:01.378261 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:01.378221 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:01.478422 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:01.478362 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:02.115407 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:02.115356 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-986g4" Jul 16 12:40:02.385977 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:02.385949 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:02.446353 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:02.446324 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:03.221834 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:03.221792 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-dlwd4" event={"ID":"32b71e49-e327-42c8-a37f-6e0db3090bc7","Type":"ContainerStarted","Data":"89a1f76125720215795526685a91832d78051e0bc8001b0b6889f4543a0b96da"} Jul 16 12:40:03.223269 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:03.223240 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-js9ms" event={"ID":"04daa141-3460-4290-98db-eccefb64fe38","Type":"ContainerStarted","Data":"7c1f855eaa555e0fbf75e53d5b9aba95d7040141b2c641b814344a773125f824"} Jul 16 12:40:03.237621 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:03.237577 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-dlwd4" podStartSLOduration=1.602757496 podStartE2EDuration="5.23756125s" totalImagesPullingTime="3.634803754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:39:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:40:03.236680995 +0000 UTC m=+61.714426627" watchObservedRunningTime="2026-07-16 12:40:03.23756125 +0000 UTC m=+61.715306867" Jul 16 12:40:03.251718 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:03.251678 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-js9ms" podStartSLOduration=1.746063514 podStartE2EDuration="5.251661626s" totalImagesPullingTime="3.505598112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:39:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:40:03.251460233 +0000 UTC m=+61.729205868" watchObservedRunningTime="2026-07-16 12:40:03.251661626 +0000 UTC m=+61.729407243" Jul 16 12:40:04.021849 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:04.021816 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:04.084182 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:04.084145 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:04.361894 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:04.361816 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:04.839141 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:04.839104 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bda475ac-da25-423d-a7ca-ebc9c29b1720-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fdthr\" (UID: \"bda475ac-da25-423d-a7ca-ebc9c29b1720\") " pod="openshift-insights/insights-runtime-extractor-fdthr" Jul 16 12:40:04.839141 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:04.839142 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bda475ac-da25-423d-a7ca-ebc9c29b1720-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fdthr\" (UID: \"bda475ac-da25-423d-a7ca-ebc9c29b1720\") " pod="openshift-insights/insights-runtime-extractor-fdthr" Jul 16 12:40:04.839361 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:40:04.839219 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 12:40:04.839361 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:40:04.839252 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 12:40:04.839361 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:40:04.839266 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bda475ac-da25-423d-a7ca-ebc9c29b1720-kube-rbac-proxy-cm podName:bda475ac-da25-423d-a7ca-ebc9c29b1720 nodeName:}" failed. No retries permitted until 2026-07-16 12:40:12.839252765 +0000 UTC m=+71.316998358 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bda475ac-da25-423d-a7ca-ebc9c29b1720-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fdthr" (UID: "bda475ac-da25-423d-a7ca-ebc9c29b1720") : configmap "kube-rbac-proxy" not found Jul 16 12:40:04.839361 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:40:04.839305 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bda475ac-da25-423d-a7ca-ebc9c29b1720-insights-runtime-extractor-tls podName:bda475ac-da25-423d-a7ca-ebc9c29b1720 nodeName:}" failed. No retries permitted until 2026-07-16 12:40:12.83929137 +0000 UTC m=+71.317036964 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bda475ac-da25-423d-a7ca-ebc9c29b1720-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fdthr" (UID: "bda475ac-da25-423d-a7ca-ebc9c29b1720") : secret "insights-runtime-extractor-tls" not found Jul 16 12:40:05.912948 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:05.912910 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:05.980981 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:05.980953 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:06.353848 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:06.353776 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:06.425314 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:06.425284 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:07.424940 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:07.424907 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:07.761310 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:07.761227 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9a976d1-7bb2-47d6-b285-788a6ff40cef-original-pull-secret\") pod \"global-pull-secret-syncer-zd992\" (UID: \"b9a976d1-7bb2-47d6-b285-788a6ff40cef\") " pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:40:07.763910 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:07.763893 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 12:40:07.773902 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:07.773877 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b9a976d1-7bb2-47d6-b285-788a6ff40cef-original-pull-secret\") pod \"global-pull-secret-syncer-zd992\" (UID: \"b9a976d1-7bb2-47d6-b285-788a6ff40cef\") " pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:40:07.861833 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:07.861799 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzjjn\" (UniqueName: \"kubernetes.io/projected/b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa-kube-api-access-nzjjn\") pod \"network-check-target-xgvpv\" (UID: \"b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa\") " pod="openshift-network-diagnostics/network-check-target-xgvpv" Jul 16 12:40:07.861833 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:07.861836 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd6ebe0-1750-4a12-a4c7-023cfadcbabc-metrics-certs\") pod \"network-metrics-daemon-q5cp8\" (UID: \"fdd6ebe0-1750-4a12-a4c7-023cfadcbabc\") " pod="openshift-multus/network-metrics-daemon-q5cp8" Jul 16 12:40:07.864323 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:07.864303 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzjjn\" (UniqueName: \"kubernetes.io/projected/b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa-kube-api-access-nzjjn\") pod \"network-check-target-xgvpv\" (UID: \"b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa\") " pod="openshift-network-diagnostics/network-check-target-xgvpv" Jul 16 12:40:07.864466 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:07.864453 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 12:40:07.872381 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:40:07.872363 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 16 12:40:07.872477 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:40:07.872432 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd6ebe0-1750-4a12-a4c7-023cfadcbabc-metrics-certs podName:fdd6ebe0-1750-4a12-a4c7-023cfadcbabc nodeName:}" failed. No retries permitted until 2026-07-16 12:41:11.872416766 +0000 UTC m=+130.350162364 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdd6ebe0-1750-4a12-a4c7-023cfadcbabc-metrics-certs") pod "network-metrics-daemon-q5cp8" (UID: "fdd6ebe0-1750-4a12-a4c7-023cfadcbabc") : secret "metrics-daemon-secret" not found Jul 16 12:40:07.891889 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:07.891867 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ff9kh\"" Jul 16 12:40:07.900092 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:07.900076 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgvpv" Jul 16 12:40:07.986265 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:07.986216 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd992" Jul 16 12:40:08.020449 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:08.020421 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xgvpv"] Jul 16 12:40:08.022604 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:40:08.022582 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31d47e2_f6dd_4b8d_8fa4_dc92bf29a9aa.slice/crio-80f2c3c2775ace6e95a39c18449168cdf4963d86eb8223c91431e39346e7b185 WatchSource:0}: Error finding container 80f2c3c2775ace6e95a39c18449168cdf4963d86eb8223c91431e39346e7b185: Status 404 returned error can't find the container with id 80f2c3c2775ace6e95a39c18449168cdf4963d86eb8223c91431e39346e7b185 Jul 16 12:40:08.114146 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:08.114113 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zd992"] Jul 16 12:40:08.118412 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:40:08.118371 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a976d1_7bb2_47d6_b285_788a6ff40cef.slice/crio-ecb1cce9271fa34fe7bbd8f0d468676cfa9f509b490f182d94db93f12c222a3d WatchSource:0}: Error finding container ecb1cce9271fa34fe7bbd8f0d468676cfa9f509b490f182d94db93f12c222a3d: Status 404 returned error can't find the container with id ecb1cce9271fa34fe7bbd8f0d468676cfa9f509b490f182d94db93f12c222a3d Jul 16 12:40:08.240284 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:08.240246 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zd992" event={"ID":"b9a976d1-7bb2-47d6-b285-788a6ff40cef","Type":"ContainerStarted","Data":"ecb1cce9271fa34fe7bbd8f0d468676cfa9f509b490f182d94db93f12c222a3d"} Jul 16 12:40:08.241617 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:08.241597 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xgvpv" event={"ID":"b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa","Type":"ContainerStarted","Data":"337c350bd5f83cc3dfb272739112f6887035f853ad46a152f9c47739fb60d243"} Jul 16 12:40:08.241689 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:08.241633 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xgvpv" event={"ID":"b31d47e2-f6dd-4b8d-8fa4-dc92bf29a9aa","Type":"ContainerStarted","Data":"80f2c3c2775ace6e95a39c18449168cdf4963d86eb8223c91431e39346e7b185"} Jul 16 12:40:08.241740 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:08.241730 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xgvpv" Jul 16 12:40:08.769865 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:08.769820 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-registry-tls\") pod \"image-registry-cb5b8dfd-qvxd2\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:40:08.770258 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:40:08.769953 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 12:40:08.770258 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:40:08.769976 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cb5b8dfd-qvxd2: secret "image-registry-tls" not found Jul 16 12:40:08.770258 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:40:08.770045 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-registry-tls podName:b9a1a7fc-4864-48d9-8b45-477fedb54b17 nodeName:}" failed. No retries permitted until 2026-07-16 12:40:40.770027835 +0000 UTC m=+99.247773430 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-registry-tls") pod "image-registry-cb5b8dfd-qvxd2" (UID: "b9a1a7fc-4864-48d9-8b45-477fedb54b17") : secret "image-registry-tls" not found Jul 16 12:40:08.871055 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:08.871016 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8-metrics-tls\") pod \"dns-default-qsszd\" (UID: \"8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8\") " pod="openshift-dns/dns-default-qsszd" Jul 16 12:40:08.871242 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:40:08.871192 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:40:08.871303 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:40:08.871271 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8-metrics-tls podName:8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8 nodeName:}" failed. No retries permitted until 2026-07-16 12:40:40.87125038 +0000 UTC m=+99.348995988 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8-metrics-tls") pod "dns-default-qsszd" (UID: "8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8") : secret "dns-default-metrics-tls" not found Jul 16 12:40:10.287893 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:10.287862 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:10.716150 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:10.716113 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:10.905765 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:10.905723 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:11.007492 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:11.007365 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:11.107701 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:11.107663 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:11.225915 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:11.225871 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:11.579441 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:11.579408 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:11.657451 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:11.657414 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:11.841672 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:11.841590 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:12.408493 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:12.408451 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:12.604990 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:12.604905 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:12.806454 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:12.806416 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:12.908644 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:12.908616 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bda475ac-da25-423d-a7ca-ebc9c29b1720-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fdthr\" (UID: \"bda475ac-da25-423d-a7ca-ebc9c29b1720\") " pod="openshift-insights/insights-runtime-extractor-fdthr" Jul 16 12:40:12.908850 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:12.908651 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bda475ac-da25-423d-a7ca-ebc9c29b1720-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fdthr\" (UID: \"bda475ac-da25-423d-a7ca-ebc9c29b1720\") " pod="openshift-insights/insights-runtime-extractor-fdthr" Jul 16 12:40:12.908850 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:40:12.908770 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 12:40:12.908850 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:40:12.908834 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bda475ac-da25-423d-a7ca-ebc9c29b1720-insights-runtime-extractor-tls podName:bda475ac-da25-423d-a7ca-ebc9c29b1720 nodeName:}" failed. No retries permitted until 2026-07-16 12:40:28.908818268 +0000 UTC m=+87.386563864 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bda475ac-da25-423d-a7ca-ebc9c29b1720-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fdthr" (UID: "bda475ac-da25-423d-a7ca-ebc9c29b1720") : secret "insights-runtime-extractor-tls" not found Jul 16 12:40:12.909183 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:12.909164 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bda475ac-da25-423d-a7ca-ebc9c29b1720-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fdthr\" (UID: \"bda475ac-da25-423d-a7ca-ebc9c29b1720\") " pod="openshift-insights/insights-runtime-extractor-fdthr" Jul 16 12:40:13.204909 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:13.204829 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:13.259326 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:13.259282 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zd992" event={"ID":"b9a976d1-7bb2-47d6-b285-788a6ff40cef","Type":"ContainerStarted","Data":"f9fb749f8fa0f435a821de10214031cac37ab622b17ab9f5b7f8b27a7fdb019e"} Jul 16 12:40:13.276626 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:13.276577 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zd992" podStartSLOduration=67.097960346 podStartE2EDuration="1m11.2765624s" totalImagesPullingTime="4.178602054s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:40:13.27628544 +0000 UTC m=+71.754031080" watchObservedRunningTime="2026-07-16 12:40:13.2765624 +0000 UTC m=+71.754308050" Jul 16 12:40:13.276780 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:13.276650 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xgvpv" podStartSLOduration=71.276644027 podStartE2EDuration="1m11.276644027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:40:08.256779368 +0000 UTC m=+66.734524995" watchObservedRunningTime="2026-07-16 12:40:13.276644027 +0000 UTC m=+71.754389643" Jul 16 12:40:13.417538 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:13.417506 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:13.803965 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:13.803932 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:14.198838 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:14.198807 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:14.399356 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:14.399324 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:14.598204 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:14.598124 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:14.698879 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:14.698847 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:15.005680 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:15.005649 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:15.210605 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:15.210573 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:15.269597 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:15.269524 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:15.614445 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:15.614338 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:16.406332 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:16.406300 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:16.799146 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:16.799055 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:17.001154 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:17.001120 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:17.080121 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:17.080050 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:17.200041 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:17.200009 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:17.401382 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:17.401354 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:18.401798 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:18.401756 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:18.808683 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:18.808605 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:19.000217 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:19.000187 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:19.081697 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:19.081632 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:19.198597 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:19.198563 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:19.350071 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:19.349984 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:19.451623 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:19.451591 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:20.405073 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:20.405039 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:20.807733 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:20.807636 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:21.010943 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:21.010914 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:21.210258 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:21.210222 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:21.413487 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:21.413451 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:26.613650 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:26.613533 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:26.714095 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:26.714049 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:26.815511 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:26.815475 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:27.158988 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:27.158953 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:28.941163 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:28.941126 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bda475ac-da25-423d-a7ca-ebc9c29b1720-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fdthr\" (UID: \"bda475ac-da25-423d-a7ca-ebc9c29b1720\") " pod="openshift-insights/insights-runtime-extractor-fdthr" Jul 16 12:40:28.943503 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:28.943479 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bda475ac-da25-423d-a7ca-ebc9c29b1720-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fdthr\" (UID: \"bda475ac-da25-423d-a7ca-ebc9c29b1720\") " pod="openshift-insights/insights-runtime-extractor-fdthr" Jul 16 12:40:29.184665 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:29.184631 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9wswp\"" Jul 16 12:40:29.192589 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:29.192521 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fdthr" Jul 16 12:40:29.315736 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:29.315709 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fdthr"] Jul 16 12:40:29.318938 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:40:29.318912 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda475ac_da25_423d_a7ca_ebc9c29b1720.slice/crio-c0f7214743197162d706cb4065bd1ced42356edea366cfbf117016418f1c44f3 WatchSource:0}: Error finding container c0f7214743197162d706cb4065bd1ced42356edea366cfbf117016418f1c44f3: Status 404 returned error can't find the container with id c0f7214743197162d706cb4065bd1ced42356edea366cfbf117016418f1c44f3 Jul 16 12:40:30.309733 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:30.309697 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fdthr" event={"ID":"bda475ac-da25-423d-a7ca-ebc9c29b1720","Type":"ContainerStarted","Data":"c0f7214743197162d706cb4065bd1ced42356edea366cfbf117016418f1c44f3"} Jul 16 12:40:30.503679 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:30.503648 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69fd8d4c85-5lkj5"] Jul 16 12:40:30.505886 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:30.505864 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69fd8d4c85-5lkj5" Jul 16 12:40:30.508630 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:30.508606 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 16 12:40:30.508630 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:30.508621 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 16 12:40:30.508825 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:30.508704 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 16 12:40:30.508825 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:30.508611 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 16 12:40:30.509647 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:30.509626 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 16 12:40:30.509770 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:30.509704 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 16 12:40:30.516025 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:30.515842 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 16 12:40:30.525980 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:30.525926 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69fd8d4c85-5lkj5"] Jul 16 12:40:30.619615 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:30.619530 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:30.655439 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:30.655404 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3b30b2cc-4802-4c6d-bc27-d46805cf66d8-hub\") pod \"cluster-proxy-proxy-agent-69fd8d4c85-5lkj5\" (UID: \"3b30b2cc-4802-4c6d-bc27-d46805cf66d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69fd8d4c85-5lkj5" Jul 16 12:40:30.655642 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:30.655451 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3b30b2cc-4802-4c6d-bc27-d46805cf66d8-ca\") pod \"cluster-proxy-proxy-agent-69fd8d4c85-5lkj5\" (UID: \"3b30b2cc-4802-4c6d-bc27-d46805cf66d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69fd8d4c85-5lkj5" Jul 16 12:40:30.655642 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:30.655477 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3b30b2cc-4802-4c6d-bc27-d46805cf66d8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-69fd8d4c85-5lkj5\" (UID: \"3b30b2cc-4802-4c6d-bc27-d46805cf66d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69fd8d4c85-5lkj5" Jul 16 12:40:30.655757 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:30.655651 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3b30b2cc-4802-4c6d-bc27-d46805cf66d8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-69fd8d4c85-5lkj5\" (UID: \"3b30b2cc-4802-4c6d-bc27-d46805cf66d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69fd8d4c85-5lkj5" Jul 16 12:40:30.655757 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:30.655747 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfffw\" (UniqueName: \"kubernetes.io/projected/3b30b2cc-4802-4c6d-bc27-d46805cf66d8-kube-api-access-bfffw\") pod \"cluster-proxy-proxy-agent-69fd8d4c85-5lkj5\" (UID: \"3b30b2cc-4802-4c6d-bc27-d46805cf66d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69fd8d4c85-5lkj5" Jul 16 12:40:30.655859 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:30.655788 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b30b2cc-4802-4c6d-bc27-d46805cf66d8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-69fd8d4c85-5lkj5\" (UID: \"3b30b2cc-4802-4c6d-bc27-d46805cf66d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69fd8d4c85-5lkj5" Jul 16 12:40:30.710625 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:30.710593 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:30.757557 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:30.756897 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfffw\" (UniqueName: \"kubernetes.io/projected/3b30b2cc-4802-4c6d-bc27-d46805cf66d8-kube-api-access-bfffw\") pod \"cluster-proxy-proxy-agent-69fd8d4c85-5lkj5\" (UID: \"3b30b2cc-4802-4c6d-bc27-d46805cf66d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69fd8d4c85-5lkj5" Jul 16 12:40:30.757557 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:30.756959 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b30b2cc-4802-4c6d-bc27-d46805cf66d8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-69fd8d4c85-5lkj5\" (UID: \"3b30b2cc-4802-4c6d-bc27-d46805cf66d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69fd8d4c85-5lkj5" Jul 16 12:40:30.757557 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:30.756996 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3b30b2cc-4802-4c6d-bc27-d46805cf66d8-hub\") pod \"cluster-proxy-proxy-agent-69fd8d4c85-5lkj5\" (UID: \"3b30b2cc-4802-4c6d-bc27-d46805cf66d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69fd8d4c85-5lkj5" Jul 16 12:40:30.757557 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:30.757026 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3b30b2cc-4802-4c6d-bc27-d46805cf66d8-ca\") pod \"cluster-proxy-proxy-agent-69fd8d4c85-5lkj5\" (UID: \"3b30b2cc-4802-4c6d-bc27-d46805cf66d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69fd8d4c85-5lkj5" Jul 16 12:40:30.757557 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:30.757057 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3b30b2cc-4802-4c6d-bc27-d46805cf66d8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-69fd8d4c85-5lkj5\" (UID: \"3b30b2cc-4802-4c6d-bc27-d46805cf66d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69fd8d4c85-5lkj5" Jul 16 12:40:30.757557 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:30.757086 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3b30b2cc-4802-4c6d-bc27-d46805cf66d8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-69fd8d4c85-5lkj5\" (UID: \"3b30b2cc-4802-4c6d-bc27-d46805cf66d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69fd8d4c85-5lkj5" Jul 16 12:40:30.758273 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:30.758242 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b30b2cc-4802-4c6d-bc27-d46805cf66d8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-69fd8d4c85-5lkj5\" (UID: \"3b30b2cc-4802-4c6d-bc27-d46805cf66d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69fd8d4c85-5lkj5" Jul 16 12:40:30.760496 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:30.760466 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3b30b2cc-4802-4c6d-bc27-d46805cf66d8-hub\") pod \"cluster-proxy-proxy-agent-69fd8d4c85-5lkj5\" (UID: \"3b30b2cc-4802-4c6d-bc27-d46805cf66d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69fd8d4c85-5lkj5" Jul 16 12:40:30.760641 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:30.760626 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3b30b2cc-4802-4c6d-bc27-d46805cf66d8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-69fd8d4c85-5lkj5\" (UID: \"3b30b2cc-4802-4c6d-bc27-d46805cf66d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69fd8d4c85-5lkj5" Jul 16 12:40:30.760732 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:30.760710 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3b30b2cc-4802-4c6d-bc27-d46805cf66d8-ca\") pod \"cluster-proxy-proxy-agent-69fd8d4c85-5lkj5\" (UID: \"3b30b2cc-4802-4c6d-bc27-d46805cf66d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69fd8d4c85-5lkj5" Jul 16 12:40:30.761025 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:30.761004 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3b30b2cc-4802-4c6d-bc27-d46805cf66d8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-69fd8d4c85-5lkj5\" (UID: \"3b30b2cc-4802-4c6d-bc27-d46805cf66d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69fd8d4c85-5lkj5" Jul 16 12:40:30.765164 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:30.765139 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfffw\" (UniqueName: \"kubernetes.io/projected/3b30b2cc-4802-4c6d-bc27-d46805cf66d8-kube-api-access-bfffw\") pod \"cluster-proxy-proxy-agent-69fd8d4c85-5lkj5\" (UID: \"3b30b2cc-4802-4c6d-bc27-d46805cf66d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69fd8d4c85-5lkj5" Jul 16 12:40:30.809923 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:30.809892 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:30.823057 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:30.823030 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69fd8d4c85-5lkj5" Jul 16 12:40:30.912304 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:30.912260 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:31.004122 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:31.003989 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69fd8d4c85-5lkj5"] Jul 16 12:40:31.007033 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:40:31.007001 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b30b2cc_4802_4c6d_bc27_d46805cf66d8.slice/crio-34fbc1f413bcf75a8aa831ec45d17304e93fbefc7740c36a69cd52dfadb19b52 WatchSource:0}: Error finding container 34fbc1f413bcf75a8aa831ec45d17304e93fbefc7740c36a69cd52dfadb19b52: Status 404 returned error can't find the container with id 34fbc1f413bcf75a8aa831ec45d17304e93fbefc7740c36a69cd52dfadb19b52 Jul 16 12:40:31.314643 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:31.314554 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69fd8d4c85-5lkj5" event={"ID":"3b30b2cc-4802-4c6d-bc27-d46805cf66d8","Type":"ContainerStarted","Data":"34fbc1f413bcf75a8aa831ec45d17304e93fbefc7740c36a69cd52dfadb19b52"} Jul 16 12:40:34.770631 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:34.770552 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:34.865744 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:34.865707 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:35.152625 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:35.152588 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:35.333342 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:35.333266 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fdthr" event={"ID":"bda475ac-da25-423d-a7ca-ebc9c29b1720","Type":"ContainerStarted","Data":"c1b4e7dce030ae486d39a9788f5859aa9b3e86f8077fbc8300b3110a8398355b"} Jul 16 12:40:38.347117 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:38.347039 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fdthr" event={"ID":"bda475ac-da25-423d-a7ca-ebc9c29b1720","Type":"ContainerStarted","Data":"f585169540daa56fe83ae62850f860f09e649cfe2d0b372a9aaaacaebbafdf27"} Jul 16 12:40:38.517405 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:38.517343 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:38.618465 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:38.618373 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:38.861468 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:38.861428 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:39.100717 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:39.100664 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:39.184945 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:39.184894 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:39.247363 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:39.247333 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xgvpv" Jul 16 12:40:39.537873 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:39.537833 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:39.643352 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:39.643316 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:39.984453 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:39.984418 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:40.054927 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:40.054894 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:40.588029 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:40.587996 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:40.829914 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:40.829881 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:40.846150 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:40.846077 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-registry-tls\") pod \"image-registry-cb5b8dfd-qvxd2\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:40:40.848871 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:40.848847 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-registry-tls\") pod \"image-registry-cb5b8dfd-qvxd2\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:40:40.921299 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:40.921257 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:40.946687 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:40.946648 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8-metrics-tls\") pod \"dns-default-qsszd\" (UID: \"8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8\") " pod="openshift-dns/dns-default-qsszd" Jul 16 12:40:40.949376 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:40.949351 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8-metrics-tls\") pod \"dns-default-qsszd\" (UID: \"8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8\") " pod="openshift-dns/dns-default-qsszd" Jul 16 12:40:41.033593 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.033556 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:41.132788 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.132754 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:41.141714 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.141687 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zz4g5\"" Jul 16 12:40:41.149741 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.149710 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:40:41.150858 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.150833 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4kt9r\"" Jul 16 12:40:41.159666 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.159632 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qsszd" Jul 16 12:40:41.283201 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.283168 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jwm54"] Jul 16 12:40:41.285359 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.285323 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jwm54" Jul 16 12:40:41.287950 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.287782 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 16 12:40:41.287950 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.287786 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 16 12:40:41.287950 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.287861 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 16 12:40:41.288196 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.288080 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 16 12:40:41.288878 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.288814 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 16 12:40:41.288878 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.288814 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 16 12:40:41.289140 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.288917 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jsphr\"" Jul 16 12:40:41.350464 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.350437 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0ab69574-a013-488c-aec8-71640bb7eddf-node-exporter-accelerators-collector-config\") pod \"node-exporter-jwm54\" (UID: \"0ab69574-a013-488c-aec8-71640bb7eddf\") " pod="openshift-monitoring/node-exporter-jwm54" Jul 16 12:40:41.350599 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.350481 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0ab69574-a013-488c-aec8-71640bb7eddf-node-exporter-textfile\") pod \"node-exporter-jwm54\" (UID: \"0ab69574-a013-488c-aec8-71640bb7eddf\") " pod="openshift-monitoring/node-exporter-jwm54" Jul 16 12:40:41.350599 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.350503 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ab69574-a013-488c-aec8-71640bb7eddf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jwm54\" (UID: \"0ab69574-a013-488c-aec8-71640bb7eddf\") " pod="openshift-monitoring/node-exporter-jwm54" Jul 16 12:40:41.350674 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.350609 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0ab69574-a013-488c-aec8-71640bb7eddf-node-exporter-tls\") pod \"node-exporter-jwm54\" (UID: \"0ab69574-a013-488c-aec8-71640bb7eddf\") " pod="openshift-monitoring/node-exporter-jwm54" Jul 16 12:40:41.350674 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.350655 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p25g4\" (UniqueName: \"kubernetes.io/projected/0ab69574-a013-488c-aec8-71640bb7eddf-kube-api-access-p25g4\") pod \"node-exporter-jwm54\" (UID: \"0ab69574-a013-488c-aec8-71640bb7eddf\") " pod="openshift-monitoring/node-exporter-jwm54" Jul 16 12:40:41.350743 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.350681 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ab69574-a013-488c-aec8-71640bb7eddf-metrics-client-ca\") pod \"node-exporter-jwm54\" (UID: \"0ab69574-a013-488c-aec8-71640bb7eddf\") " pod="openshift-monitoring/node-exporter-jwm54" Jul 16 12:40:41.350743 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.350730 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0ab69574-a013-488c-aec8-71640bb7eddf-node-exporter-wtmp\") pod \"node-exporter-jwm54\" (UID: \"0ab69574-a013-488c-aec8-71640bb7eddf\") " pod="openshift-monitoring/node-exporter-jwm54" Jul 16 12:40:41.350804 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.350756 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ab69574-a013-488c-aec8-71640bb7eddf-sys\") pod \"node-exporter-jwm54\" (UID: \"0ab69574-a013-488c-aec8-71640bb7eddf\") " pod="openshift-monitoring/node-exporter-jwm54" Jul 16 12:40:41.350804 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.350787 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0ab69574-a013-488c-aec8-71640bb7eddf-root\") pod \"node-exporter-jwm54\" (UID: \"0ab69574-a013-488c-aec8-71640bb7eddf\") " pod="openshift-monitoring/node-exporter-jwm54" Jul 16 12:40:41.383051 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.382988 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:41.408086 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.408057 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qsszd"] Jul 16 12:40:41.410902 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:40:41.410874 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa4a2f1_20ee_4202_b1f6_452b7a6a78a8.slice/crio-80f25427cb8b412fe4a5680e781e40773adc3ce0dea35345f52ca7d9623ab25a WatchSource:0}: Error finding container 80f25427cb8b412fe4a5680e781e40773adc3ce0dea35345f52ca7d9623ab25a: Status 404 returned error can't find the container with id 80f25427cb8b412fe4a5680e781e40773adc3ce0dea35345f52ca7d9623ab25a Jul 16 12:40:41.424814 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.424755 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cb5b8dfd-qvxd2"] Jul 16 12:40:41.427137 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:40:41.427116 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a1a7fc_4864_48d9_8b45_477fedb54b17.slice/crio-ad89fa1ffbdd139639a3007b002f691e939101134c78a8498a48fa6a8cd804e3 WatchSource:0}: Error finding container ad89fa1ffbdd139639a3007b002f691e939101134c78a8498a48fa6a8cd804e3: Status 404 returned error can't find the container with id ad89fa1ffbdd139639a3007b002f691e939101134c78a8498a48fa6a8cd804e3 Jul 16 12:40:41.451189 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.451164 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0ab69574-a013-488c-aec8-71640bb7eddf-node-exporter-tls\") pod \"node-exporter-jwm54\" (UID: \"0ab69574-a013-488c-aec8-71640bb7eddf\") " pod="openshift-monitoring/node-exporter-jwm54" Jul 16 12:40:41.451281 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.451208 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p25g4\" (UniqueName: \"kubernetes.io/projected/0ab69574-a013-488c-aec8-71640bb7eddf-kube-api-access-p25g4\") pod \"node-exporter-jwm54\" (UID: \"0ab69574-a013-488c-aec8-71640bb7eddf\") " pod="openshift-monitoring/node-exporter-jwm54" Jul 16 12:40:41.451281 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.451225 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ab69574-a013-488c-aec8-71640bb7eddf-metrics-client-ca\") pod \"node-exporter-jwm54\" (UID: \"0ab69574-a013-488c-aec8-71640bb7eddf\") " pod="openshift-monitoring/node-exporter-jwm54" Jul 16 12:40:41.451281 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.451264 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0ab69574-a013-488c-aec8-71640bb7eddf-node-exporter-wtmp\") pod \"node-exporter-jwm54\" (UID: \"0ab69574-a013-488c-aec8-71640bb7eddf\") " pod="openshift-monitoring/node-exporter-jwm54" Jul 16 12:40:41.451452 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.451282 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ab69574-a013-488c-aec8-71640bb7eddf-sys\") pod \"node-exporter-jwm54\" (UID: \"0ab69574-a013-488c-aec8-71640bb7eddf\") " pod="openshift-monitoring/node-exporter-jwm54" Jul 16 12:40:41.451452 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.451305 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0ab69574-a013-488c-aec8-71640bb7eddf-root\") pod \"node-exporter-jwm54\" (UID: \"0ab69574-a013-488c-aec8-71640bb7eddf\") " pod="openshift-monitoring/node-exporter-jwm54" Jul 16 12:40:41.451452 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.451346 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0ab69574-a013-488c-aec8-71640bb7eddf-node-exporter-accelerators-collector-config\") pod \"node-exporter-jwm54\" (UID: \"0ab69574-a013-488c-aec8-71640bb7eddf\") " pod="openshift-monitoring/node-exporter-jwm54" Jul 16 12:40:41.451628 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.451453 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0ab69574-a013-488c-aec8-71640bb7eddf-node-exporter-wtmp\") pod \"node-exporter-jwm54\" (UID: \"0ab69574-a013-488c-aec8-71640bb7eddf\") " pod="openshift-monitoring/node-exporter-jwm54" Jul 16 12:40:41.451628 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.451469 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0ab69574-a013-488c-aec8-71640bb7eddf-root\") pod \"node-exporter-jwm54\" (UID: \"0ab69574-a013-488c-aec8-71640bb7eddf\") " pod="openshift-monitoring/node-exporter-jwm54" Jul 16 12:40:41.451628 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.451504 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ab69574-a013-488c-aec8-71640bb7eddf-sys\") pod \"node-exporter-jwm54\" (UID: \"0ab69574-a013-488c-aec8-71640bb7eddf\") " pod="openshift-monitoring/node-exporter-jwm54" Jul 16 12:40:41.451628 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.451381 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0ab69574-a013-488c-aec8-71640bb7eddf-node-exporter-textfile\") pod \"node-exporter-jwm54\" (UID: \"0ab69574-a013-488c-aec8-71640bb7eddf\") " pod="openshift-monitoring/node-exporter-jwm54" Jul 16 12:40:41.451628 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.451589 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ab69574-a013-488c-aec8-71640bb7eddf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jwm54\" (UID: \"0ab69574-a013-488c-aec8-71640bb7eddf\") " pod="openshift-monitoring/node-exporter-jwm54" Jul 16 12:40:41.451869 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.451849 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0ab69574-a013-488c-aec8-71640bb7eddf-node-exporter-textfile\") pod \"node-exporter-jwm54\" (UID: \"0ab69574-a013-488c-aec8-71640bb7eddf\") " pod="openshift-monitoring/node-exporter-jwm54" Jul 16 12:40:41.451999 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.451969 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ab69574-a013-488c-aec8-71640bb7eddf-metrics-client-ca\") pod \"node-exporter-jwm54\" (UID: \"0ab69574-a013-488c-aec8-71640bb7eddf\") " pod="openshift-monitoring/node-exporter-jwm54" Jul 16 12:40:41.452220 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.452200 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0ab69574-a013-488c-aec8-71640bb7eddf-node-exporter-accelerators-collector-config\") pod \"node-exporter-jwm54\" (UID: \"0ab69574-a013-488c-aec8-71640bb7eddf\") " pod="openshift-monitoring/node-exporter-jwm54" Jul 16 12:40:41.453669 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.453647 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0ab69574-a013-488c-aec8-71640bb7eddf-node-exporter-tls\") pod \"node-exporter-jwm54\" (UID: \"0ab69574-a013-488c-aec8-71640bb7eddf\") " pod="openshift-monitoring/node-exporter-jwm54" Jul 16 12:40:41.453807 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.453789 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ab69574-a013-488c-aec8-71640bb7eddf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jwm54\" (UID: \"0ab69574-a013-488c-aec8-71640bb7eddf\") " pod="openshift-monitoring/node-exporter-jwm54" Jul 16 12:40:41.459044 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.459024 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p25g4\" (UniqueName: \"kubernetes.io/projected/0ab69574-a013-488c-aec8-71640bb7eddf-kube-api-access-p25g4\") pod \"node-exporter-jwm54\" (UID: \"0ab69574-a013-488c-aec8-71640bb7eddf\") " pod="openshift-monitoring/node-exporter-jwm54" Jul 16 12:40:41.477122 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.477093 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:41.594199 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.594156 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:41.598111 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.598091 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jwm54" Jul 16 12:40:41.606886 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:40:41.606860 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab69574_a013_488c_aec8_71640bb7eddf.slice/crio-6426e5e6d3f7ecc4fd7947b633b3dc75b028611d98602c3f67e3ecee33726c91 WatchSource:0}: Error finding container 6426e5e6d3f7ecc4fd7947b633b3dc75b028611d98602c3f67e3ecee33726c91: Status 404 returned error can't find the container with id 6426e5e6d3f7ecc4fd7947b633b3dc75b028611d98602c3f67e3ecee33726c91 Jul 16 12:40:41.921542 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:41.921493 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:42.024023 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:42.023989 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:42.225174 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:42.225101 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:42.304282 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:42.304249 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:42.362578 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:42.362539 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fdthr" event={"ID":"bda475ac-da25-423d-a7ca-ebc9c29b1720","Type":"ContainerStarted","Data":"cd55972cc565bbacb8c4525428cba5ac1ea09b89a8c9d456a2d9ce6e6f82f07d"} Jul 16 12:40:42.364070 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:42.364040 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jwm54" event={"ID":"0ab69574-a013-488c-aec8-71640bb7eddf","Type":"ContainerStarted","Data":"6426e5e6d3f7ecc4fd7947b633b3dc75b028611d98602c3f67e3ecee33726c91"} Jul 16 12:40:42.365698 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:42.365660 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" event={"ID":"b9a1a7fc-4864-48d9-8b45-477fedb54b17","Type":"ContainerStarted","Data":"bfa98e368054b37fcbedb9c791dbcb56cec93f18f10660b2224a295286211e44"} Jul 16 12:40:42.365820 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:42.365718 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" event={"ID":"b9a1a7fc-4864-48d9-8b45-477fedb54b17","Type":"ContainerStarted","Data":"ad89fa1ffbdd139639a3007b002f691e939101134c78a8498a48fa6a8cd804e3"} Jul 16 12:40:42.365820 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:42.365744 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:40:42.366898 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:42.366875 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qsszd" event={"ID":"8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8","Type":"ContainerStarted","Data":"80f25427cb8b412fe4a5680e781e40773adc3ce0dea35345f52ca7d9623ab25a"} Jul 16 12:40:42.379809 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:42.379765 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-fdthr" podStartSLOduration=33.517838902 podStartE2EDuration="45.379745402s" totalImagesPullingTime="11.8619065s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:39:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:40:42.379678196 +0000 UTC m=+100.857423838" watchObservedRunningTime="2026-07-16 12:40:42.379745402 +0000 UTC m=+100.857491023" Jul 16 12:40:42.399723 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:42.399681 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" podStartSLOduration=100.399662277 podStartE2EDuration="1m40.399662277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:40:42.398617335 +0000 UTC m=+100.876362977" watchObservedRunningTime="2026-07-16 12:40:42.399662277 +0000 UTC m=+100.877407895" Jul 16 12:40:42.639497 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:42.639456 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:42.806285 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:42.806250 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:44.378540 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:44.378500 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qsszd" event={"ID":"8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8","Type":"ContainerStarted","Data":"6df11afd8051bfabf1fc1bd84bf59821499b8f8141df2a2f1ae2027e4449934a"} Jul 16 12:40:44.378971 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:44.378576 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qsszd" event={"ID":"8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8","Type":"ContainerStarted","Data":"a5c398dc498d56a751776f68584fcb1817712d75d5191ab438f1ae9681be068a"} Jul 16 12:40:44.378971 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:44.378612 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qsszd" Jul 16 12:40:44.381334 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:44.381208 2582 generic.go:356] "Generic (PLEG): container finished" podID="0ab69574-a013-488c-aec8-71640bb7eddf" containerID="7d24cd8093bb1be299044cd901ddd7971d2c312d2d8f9a1f68621d1c64c0197d" exitCode=0 Jul 16 12:40:44.381334 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:44.381296 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jwm54" event={"ID":"0ab69574-a013-488c-aec8-71640bb7eddf","Type":"ContainerDied","Data":"7d24cd8093bb1be299044cd901ddd7971d2c312d2d8f9a1f68621d1c64c0197d"} Jul 16 12:40:44.397133 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:44.397089 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qsszd" podStartSLOduration=66.125374832 podStartE2EDuration="1m8.397067828s" totalImagesPullingTime="2.271692996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:39:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:40:44.395882695 +0000 UTC m=+102.873628312" watchObservedRunningTime="2026-07-16 12:40:44.397067828 +0000 UTC m=+102.874813445" Jul 16 12:40:44.725516 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:44.725439 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:44.834780 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:44.834747 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:45.389147 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:45.389106 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jwm54" event={"ID":"0ab69574-a013-488c-aec8-71640bb7eddf","Type":"ContainerStarted","Data":"6d98e197c935a5a7d23b74208f4e87ef70fd0bad412a465b7e2df328f5900a09"} Jul 16 12:40:45.389594 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:45.389176 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jwm54" event={"ID":"0ab69574-a013-488c-aec8-71640bb7eddf","Type":"ContainerStarted","Data":"d7925cef6bacc97732dd54a49598f5f90ee334ef9b0511da2e717cfbcf7858ff"} Jul 16 12:40:45.411167 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:45.411120 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jwm54" podStartSLOduration=2.2480175 podStartE2EDuration="4.411103831s" totalImagesPullingTime="2.072112115s" totalInitContainerRuntime="90.974216ms" isStatefulPod=true podCreationTimestamp="2026-07-16 12:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:40:45.409153634 +0000 UTC m=+103.886899289" watchObservedRunningTime="2026-07-16 12:40:45.411103831 +0000 UTC m=+103.888849449" Jul 16 12:40:46.014555 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:46.014514 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:46.092851 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:46.092810 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:46.330486 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:46.330402 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:47.399208 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:47.399168 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69fd8d4c85-5lkj5" event={"ID":"3b30b2cc-4802-4c6d-bc27-d46805cf66d8","Type":"ContainerStarted","Data":"b152df3955bfbbe8532be06872338f943ab229de4f2dde07b4ca30e10bb59e18"} Jul 16 12:40:47.399615 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:47.399223 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69fd8d4c85-5lkj5" event={"ID":"3b30b2cc-4802-4c6d-bc27-d46805cf66d8","Type":"ContainerStarted","Data":"979636a90653aa5458c1e54d85487ca71f845c2dd6736e87fdcc5dcf62705ff4"} Jul 16 12:40:47.399615 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:47.399241 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69fd8d4c85-5lkj5" event={"ID":"3b30b2cc-4802-4c6d-bc27-d46805cf66d8","Type":"ContainerStarted","Data":"37fa47f94cb6ecdfba28c3af4ee877fd7128b8aef5d1f0de1812ed9568289f0a"} Jul 16 12:40:47.420685 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:47.420636 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69fd8d4c85-5lkj5" podStartSLOduration=1.744481603 podStartE2EDuration="17.420620826s" totalImagesPullingTime="15.676139223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:40:47.419962308 +0000 UTC m=+105.897707924" watchObservedRunningTime="2026-07-16 12:40:47.420620826 +0000 UTC m=+105.898366506" Jul 16 12:40:47.483328 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:47.483278 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:47.579106 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:47.579072 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:47.679472 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:47.679379 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:49.658168 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:49.658134 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:49.758578 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:49.758546 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:50.200427 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:50.200364 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:50.255605 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:50.255579 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:53.078497 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:53.078464 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cb5b8dfd-qvxd2"] Jul 16 12:40:53.219167 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:53.219133 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:53.319245 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:53.319209 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:54.391836 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:54.391800 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qsszd" Jul 16 12:40:54.810150 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:54.810071 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:54.908026 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:54.907998 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:55.169065 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:55.169034 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:55.271538 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:55.271505 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:56.607034 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.607007 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77f77d9d9b-4vlks"] Jul 16 12:40:56.609033 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.609018 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77f77d9d9b-4vlks" Jul 16 12:40:56.611447 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.611421 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 16 12:40:56.611583 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.611509 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 16 12:40:56.612569 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.612552 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 16 12:40:56.612683 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.612575 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 16 12:40:56.612683 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.612587 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 16 12:40:56.612877 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.612862 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kv8xc\"" Jul 16 12:40:56.612947 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.612874 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 16 12:40:56.612947 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.612878 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 16 12:40:56.616686 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.616669 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 16 12:40:56.619274 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.619253 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77f77d9d9b-4vlks"] Jul 16 12:40:56.622092 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.622065 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:56.677764 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.677744 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-console-serving-cert\") pod \"console-77f77d9d9b-4vlks\" (UID: \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\") " pod="openshift-console/console-77f77d9d9b-4vlks" Jul 16 12:40:56.677842 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.677774 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-service-ca\") pod \"console-77f77d9d9b-4vlks\" (UID: \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\") " pod="openshift-console/console-77f77d9d9b-4vlks" Jul 16 12:40:56.677842 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.677807 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-oauth-serving-cert\") pod \"console-77f77d9d9b-4vlks\" (UID: \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\") " pod="openshift-console/console-77f77d9d9b-4vlks" Jul 16 12:40:56.677842 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.677828 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-console-oauth-config\") pod \"console-77f77d9d9b-4vlks\" (UID: \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\") " pod="openshift-console/console-77f77d9d9b-4vlks" Jul 16 12:40:56.677946 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.677844 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-console-config\") pod \"console-77f77d9d9b-4vlks\" (UID: \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\") " pod="openshift-console/console-77f77d9d9b-4vlks" Jul 16 12:40:56.677946 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.677871 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxqng\" (UniqueName: \"kubernetes.io/projected/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-kube-api-access-zxqng\") pod \"console-77f77d9d9b-4vlks\" (UID: \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\") " pod="openshift-console/console-77f77d9d9b-4vlks" Jul 16 12:40:56.677946 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.677890 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-trusted-ca-bundle\") pod \"console-77f77d9d9b-4vlks\" (UID: \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\") " pod="openshift-console/console-77f77d9d9b-4vlks" Jul 16 12:40:56.691637 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.691615 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:56.778523 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.778498 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-service-ca\") pod \"console-77f77d9d9b-4vlks\" (UID: \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\") " pod="openshift-console/console-77f77d9d9b-4vlks" Jul 16 12:40:56.778644 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.778550 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-oauth-serving-cert\") pod \"console-77f77d9d9b-4vlks\" (UID: \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\") " pod="openshift-console/console-77f77d9d9b-4vlks" Jul 16 12:40:56.778644 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.778586 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-console-oauth-config\") pod \"console-77f77d9d9b-4vlks\" (UID: \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\") " pod="openshift-console/console-77f77d9d9b-4vlks" Jul 16 12:40:56.778644 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.778609 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-console-config\") pod \"console-77f77d9d9b-4vlks\" (UID: \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\") " pod="openshift-console/console-77f77d9d9b-4vlks" Jul 16 12:40:56.778808 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.778659 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxqng\" (UniqueName: \"kubernetes.io/projected/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-kube-api-access-zxqng\") pod \"console-77f77d9d9b-4vlks\" (UID: \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\") " pod="openshift-console/console-77f77d9d9b-4vlks" Jul 16 12:40:56.778808 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.778687 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-trusted-ca-bundle\") pod \"console-77f77d9d9b-4vlks\" (UID: \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\") " pod="openshift-console/console-77f77d9d9b-4vlks" Jul 16 12:40:56.778808 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.778736 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-console-serving-cert\") pod \"console-77f77d9d9b-4vlks\" (UID: \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\") " pod="openshift-console/console-77f77d9d9b-4vlks" Jul 16 12:40:56.779533 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.779288 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-service-ca\") pod \"console-77f77d9d9b-4vlks\" (UID: \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\") " pod="openshift-console/console-77f77d9d9b-4vlks" Jul 16 12:40:56.779533 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.779353 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-oauth-serving-cert\") pod \"console-77f77d9d9b-4vlks\" (UID: \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\") " pod="openshift-console/console-77f77d9d9b-4vlks" Jul 16 12:40:56.779533 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.779370 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-console-config\") pod \"console-77f77d9d9b-4vlks\" (UID: \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\") " pod="openshift-console/console-77f77d9d9b-4vlks" Jul 16 12:40:56.779846 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.779812 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-trusted-ca-bundle\") pod \"console-77f77d9d9b-4vlks\" (UID: \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\") " pod="openshift-console/console-77f77d9d9b-4vlks" Jul 16 12:40:56.781229 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.781201 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-console-serving-cert\") pod \"console-77f77d9d9b-4vlks\" (UID: \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\") " pod="openshift-console/console-77f77d9d9b-4vlks" Jul 16 12:40:56.781305 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.781261 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-console-oauth-config\") pod \"console-77f77d9d9b-4vlks\" (UID: \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\") " pod="openshift-console/console-77f77d9d9b-4vlks" Jul 16 12:40:56.789353 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.789334 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxqng\" (UniqueName: \"kubernetes.io/projected/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-kube-api-access-zxqng\") pod \"console-77f77d9d9b-4vlks\" (UID: \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\") " pod="openshift-console/console-77f77d9d9b-4vlks" Jul 16 12:40:56.918755 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:56.918725 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77f77d9d9b-4vlks" Jul 16 12:40:57.045576 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:57.045539 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77f77d9d9b-4vlks"] Jul 16 12:40:57.048231 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:40:57.048210 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce0f4c40_d236_4a0d_a84f_b9abe985c4d4.slice/crio-fa2599bf25c4f7a5bb1a5cfee4aabf4120cb821a4158d406f63e2e71aa2b9880 WatchSource:0}: Error finding container fa2599bf25c4f7a5bb1a5cfee4aabf4120cb821a4158d406f63e2e71aa2b9880: Status 404 returned error can't find the container with id fa2599bf25c4f7a5bb1a5cfee4aabf4120cb821a4158d406f63e2e71aa2b9880 Jul 16 12:40:57.435870 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:57.435836 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77f77d9d9b-4vlks" event={"ID":"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4","Type":"ContainerStarted","Data":"fa2599bf25c4f7a5bb1a5cfee4aabf4120cb821a4158d406f63e2e71aa2b9880"} Jul 16 12:40:58.731294 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:58.731252 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:40:58.803993 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:40:58.803960 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:41:00.449097 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:00.449060 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77f77d9d9b-4vlks" event={"ID":"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4","Type":"ContainerStarted","Data":"c1883b58b54b68d6c2c1f3e66791d713e9f6304217d43e4be7c6f7474de4f935"} Jul 16 12:41:00.467003 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:00.466959 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77f77d9d9b-4vlks" podStartSLOduration=1.595556338 podStartE2EDuration="4.466945038s" totalImagesPullingTime="2.8713887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:40:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:41:00.466185344 +0000 UTC m=+118.943930961" watchObservedRunningTime="2026-07-16 12:41:00.466945038 +0000 UTC m=+118.944690652" Jul 16 12:41:03.084762 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:03.084728 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:41:03.535720 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:03.535686 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:41:03.636435 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:03.636376 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:41:03.737524 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:03.737489 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:41:04.362535 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:04.362503 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:41:04.484116 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:04.484081 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:41:06.192878 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:06.192842 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77f77d9d9b-4vlks"] Jul 16 12:41:06.919191 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:06.919162 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77f77d9d9b-4vlks" Jul 16 12:41:07.004903 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:07.004871 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:41:07.879477 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:07.879442 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:41:11.903288 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:11.903255 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd6ebe0-1750-4a12-a4c7-023cfadcbabc-metrics-certs\") pod \"network-metrics-daemon-q5cp8\" (UID: \"fdd6ebe0-1750-4a12-a4c7-023cfadcbabc\") " pod="openshift-multus/network-metrics-daemon-q5cp8" Jul 16 12:41:11.905668 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:11.905638 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd6ebe0-1750-4a12-a4c7-023cfadcbabc-metrics-certs\") pod \"network-metrics-daemon-q5cp8\" (UID: \"fdd6ebe0-1750-4a12-a4c7-023cfadcbabc\") " pod="openshift-multus/network-metrics-daemon-q5cp8" Jul 16 12:41:12.066008 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:12.065960 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:41:12.086178 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:12.086154 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8df2z\"" Jul 16 12:41:12.094902 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:12.094885 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5cp8" Jul 16 12:41:12.166930 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:12.166839 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:41:12.220816 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:12.220785 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q5cp8"] Jul 16 12:41:12.223559 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:41:12.223531 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd6ebe0_1750_4a12_a4c7_023cfadcbabc.slice/crio-a26afe0d4da9f72e63f1939a9b3c9a99d47b30ff0bc49ff73b3f5b584cf1c66e WatchSource:0}: Error finding container a26afe0d4da9f72e63f1939a9b3c9a99d47b30ff0bc49ff73b3f5b584cf1c66e: Status 404 returned error can't find the container with id a26afe0d4da9f72e63f1939a9b3c9a99d47b30ff0bc49ff73b3f5b584cf1c66e Jul 16 12:41:12.268148 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:12.268118 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:41:12.489748 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:12.489663 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q5cp8" event={"ID":"fdd6ebe0-1750-4a12-a4c7-023cfadcbabc","Type":"ContainerStarted","Data":"a26afe0d4da9f72e63f1939a9b3c9a99d47b30ff0bc49ff73b3f5b584cf1c66e"} Jul 16 12:41:12.491039 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:12.491015 2582 generic.go:356] "Generic (PLEG): container finished" podID="5dd68711-4227-4f4c-b6b9-2b4eb11b7478" containerID="88d4313d1081b6be46aefcdac5a9641966ca1d97ac3c924a3691ba79afaf2e18" exitCode=0 Jul 16 12:41:12.491139 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:12.491089 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-2fzwz" event={"ID":"5dd68711-4227-4f4c-b6b9-2b4eb11b7478","Type":"ContainerDied","Data":"88d4313d1081b6be46aefcdac5a9641966ca1d97ac3c924a3691ba79afaf2e18"} Jul 16 12:41:12.491400 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:12.491372 2582 scope.go:117] "RemoveContainer" containerID="88d4313d1081b6be46aefcdac5a9641966ca1d97ac3c924a3691ba79afaf2e18" Jul 16 12:41:13.496729 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:13.496698 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-2fzwz" event={"ID":"5dd68711-4227-4f4c-b6b9-2b4eb11b7478","Type":"ContainerStarted","Data":"3d03907b5b9d027eafb242900d337c144c268448460ff1d38d4ad1b666665e0c"} Jul 16 12:41:15.505436 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:15.505374 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q5cp8" event={"ID":"fdd6ebe0-1750-4a12-a4c7-023cfadcbabc","Type":"ContainerStarted","Data":"a509924bb26d6ef4dba7fd58382100213b25a035cbc11b470486326e9812f4f5"} Jul 16 12:41:15.505883 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:15.505457 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q5cp8" event={"ID":"fdd6ebe0-1750-4a12-a4c7-023cfadcbabc","Type":"ContainerStarted","Data":"1ee2a59199c5f3426dfff7f83d3a801cfa6359bad551415207719133b04da803"} Jul 16 12:41:15.527285 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:15.527229 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-q5cp8" podStartSLOduration=130.673698095 podStartE2EDuration="2m13.527214385s" totalImagesPullingTime="2.85351629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:41:15.525982814 +0000 UTC m=+134.003728430" watchObservedRunningTime="2026-07-16 12:41:15.527214385 +0000 UTC m=+134.004960000" Jul 16 12:41:15.628784 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:15.628755 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:41:15.731100 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:15.731071 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:41:15.834000 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:15.833968 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:41:17.362350 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:17.362314 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:41:17.428553 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:17.428522 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:41:17.555457 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:17.555426 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:41:17.623466 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:17.623378 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:41:18.098030 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.097974 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" podUID="b9a1a7fc-4864-48d9-8b45-477fedb54b17" containerName="registry" containerID="cri-o://bfa98e368054b37fcbedb9c791dbcb56cec93f18f10660b2224a295286211e44" gracePeriod=30 Jul 16 12:41:18.368478 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.368421 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:41:18.516935 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.516899 2582 generic.go:356] "Generic (PLEG): container finished" podID="b9a1a7fc-4864-48d9-8b45-477fedb54b17" containerID="bfa98e368054b37fcbedb9c791dbcb56cec93f18f10660b2224a295286211e44" exitCode=0 Jul 16 12:41:18.517109 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.516952 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" Jul 16 12:41:18.517109 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.516955 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" event={"ID":"b9a1a7fc-4864-48d9-8b45-477fedb54b17","Type":"ContainerDied","Data":"bfa98e368054b37fcbedb9c791dbcb56cec93f18f10660b2224a295286211e44"} Jul 16 12:41:18.517109 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.517079 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cb5b8dfd-qvxd2" event={"ID":"b9a1a7fc-4864-48d9-8b45-477fedb54b17","Type":"ContainerDied","Data":"ad89fa1ffbdd139639a3007b002f691e939101134c78a8498a48fa6a8cd804e3"} Jul 16 12:41:18.517109 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.517105 2582 scope.go:117] "RemoveContainer" containerID="bfa98e368054b37fcbedb9c791dbcb56cec93f18f10660b2224a295286211e44" Jul 16 12:41:18.527606 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.527591 2582 scope.go:117] "RemoveContainer" containerID="bfa98e368054b37fcbedb9c791dbcb56cec93f18f10660b2224a295286211e44" Jul 16 12:41:18.527880 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:41:18.527858 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfa98e368054b37fcbedb9c791dbcb56cec93f18f10660b2224a295286211e44\": container with ID starting with bfa98e368054b37fcbedb9c791dbcb56cec93f18f10660b2224a295286211e44 not found: ID does not exist" containerID="bfa98e368054b37fcbedb9c791dbcb56cec93f18f10660b2224a295286211e44" Jul 16 12:41:18.527936 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.527889 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfa98e368054b37fcbedb9c791dbcb56cec93f18f10660b2224a295286211e44"} err="failed to get container status \"bfa98e368054b37fcbedb9c791dbcb56cec93f18f10660b2224a295286211e44\": rpc error: code = NotFound desc = could not find container \"bfa98e368054b37fcbedb9c791dbcb56cec93f18f10660b2224a295286211e44\": container with ID starting with bfa98e368054b37fcbedb9c791dbcb56cec93f18f10660b2224a295286211e44 not found: ID does not exist" Jul 16 12:41:18.558210 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.558184 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b9a1a7fc-4864-48d9-8b45-477fedb54b17-image-registry-private-configuration\") pod \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " Jul 16 12:41:18.558296 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.558232 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtslp\" (UniqueName: \"kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-kube-api-access-wtslp\") pod \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " Jul 16 12:41:18.558296 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.558268 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b9a1a7fc-4864-48d9-8b45-477fedb54b17-ca-trust-extracted\") pod \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " Jul 16 12:41:18.558433 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.558415 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-bound-sa-token\") pod \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " Jul 16 12:41:18.558501 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.558467 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-registry-tls\") pod \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " Jul 16 12:41:18.558556 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.558531 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9a1a7fc-4864-48d9-8b45-477fedb54b17-trusted-ca\") pod \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " Jul 16 12:41:18.558605 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.558560 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b9a1a7fc-4864-48d9-8b45-477fedb54b17-registry-certificates\") pod \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " Jul 16 12:41:18.558605 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.558584 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b9a1a7fc-4864-48d9-8b45-477fedb54b17-installation-pull-secrets\") pod \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\" (UID: \"b9a1a7fc-4864-48d9-8b45-477fedb54b17\") " Jul 16 12:41:18.559336 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.559279 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9a1a7fc-4864-48d9-8b45-477fedb54b17-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b9a1a7fc-4864-48d9-8b45-477fedb54b17" (UID: "b9a1a7fc-4864-48d9-8b45-477fedb54b17"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:41:18.559548 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.559518 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9a1a7fc-4864-48d9-8b45-477fedb54b17-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b9a1a7fc-4864-48d9-8b45-477fedb54b17" (UID: "b9a1a7fc-4864-48d9-8b45-477fedb54b17"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:41:18.560889 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.560866 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9a1a7fc-4864-48d9-8b45-477fedb54b17-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b9a1a7fc-4864-48d9-8b45-477fedb54b17" (UID: "b9a1a7fc-4864-48d9-8b45-477fedb54b17"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:41:18.560889 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.560874 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-kube-api-access-wtslp" (OuterVolumeSpecName: "kube-api-access-wtslp") pod "b9a1a7fc-4864-48d9-8b45-477fedb54b17" (UID: "b9a1a7fc-4864-48d9-8b45-477fedb54b17"). InnerVolumeSpecName "kube-api-access-wtslp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:41:18.561032 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.560989 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b9a1a7fc-4864-48d9-8b45-477fedb54b17" (UID: "b9a1a7fc-4864-48d9-8b45-477fedb54b17"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:41:18.561295 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.561276 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b9a1a7fc-4864-48d9-8b45-477fedb54b17" (UID: "b9a1a7fc-4864-48d9-8b45-477fedb54b17"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:41:18.561356 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.561293 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9a1a7fc-4864-48d9-8b45-477fedb54b17-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b9a1a7fc-4864-48d9-8b45-477fedb54b17" (UID: "b9a1a7fc-4864-48d9-8b45-477fedb54b17"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:41:18.567553 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.567529 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a1a7fc-4864-48d9-8b45-477fedb54b17-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b9a1a7fc-4864-48d9-8b45-477fedb54b17" (UID: "b9a1a7fc-4864-48d9-8b45-477fedb54b17"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:41:18.659262 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.659233 2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-registry-tls\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:41:18.659262 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.659257 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9a1a7fc-4864-48d9-8b45-477fedb54b17-trusted-ca\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:41:18.659433 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.659267 2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b9a1a7fc-4864-48d9-8b45-477fedb54b17-registry-certificates\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:41:18.659433 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.659276 2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b9a1a7fc-4864-48d9-8b45-477fedb54b17-installation-pull-secrets\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:41:18.659433 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.659287 2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b9a1a7fc-4864-48d9-8b45-477fedb54b17-image-registry-private-configuration\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:41:18.659433 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.659296 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtslp\" (UniqueName: \"kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-kube-api-access-wtslp\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:41:18.659433 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.659304 2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b9a1a7fc-4864-48d9-8b45-477fedb54b17-ca-trust-extracted\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:41:18.659433 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.659312 2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9a1a7fc-4864-48d9-8b45-477fedb54b17-bound-sa-token\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:41:18.839171 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.839134 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cb5b8dfd-qvxd2"] Jul 16 12:41:18.842371 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:18.842347 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-cb5b8dfd-qvxd2"] Jul 16 12:41:19.362609 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:19.362583 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:41:19.450754 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:19.450722 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:41:19.976547 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:19.976514 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9a1a7fc-4864-48d9-8b45-477fedb54b17" path="/var/lib/kubelet/pods/b9a1a7fc-4864-48d9-8b45-477fedb54b17/volumes" Jul 16 12:41:21.319503 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:21.319467 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:41:21.420775 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:21.420743 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:41:24.538316 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:24.538281 2582 generic.go:356] "Generic (PLEG): container finished" podID="8fd6e2f8-4515-47e3-94e9-32bc0f1a9e75" containerID="8b479232cd84482720db65b957c5fd1fe400014d03b60eb0996fefe7112d3429" exitCode=0 Jul 16 12:41:24.538740 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:24.538358 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-h7g92" event={"ID":"8fd6e2f8-4515-47e3-94e9-32bc0f1a9e75","Type":"ContainerDied","Data":"8b479232cd84482720db65b957c5fd1fe400014d03b60eb0996fefe7112d3429"} Jul 16 12:41:24.538740 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:24.538706 2582 scope.go:117] "RemoveContainer" containerID="8b479232cd84482720db65b957c5fd1fe400014d03b60eb0996fefe7112d3429" Jul 16 12:41:25.546827 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:25.546793 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-h7g92" event={"ID":"8fd6e2f8-4515-47e3-94e9-32bc0f1a9e75","Type":"ContainerStarted","Data":"8f304977cfaa6c86fec2535f301b684fdaf01634aceb132a97b038cd3a864f20"} Jul 16 12:41:28.561376 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:28.561277 2582 generic.go:356] "Generic (PLEG): container finished" podID="04daa141-3460-4290-98db-eccefb64fe38" containerID="7c1f855eaa555e0fbf75e53d5b9aba95d7040141b2c641b814344a773125f824" exitCode=0 Jul 16 12:41:28.561376 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:28.561346 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-js9ms" event={"ID":"04daa141-3460-4290-98db-eccefb64fe38","Type":"ContainerDied","Data":"7c1f855eaa555e0fbf75e53d5b9aba95d7040141b2c641b814344a773125f824"} Jul 16 12:41:28.561883 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:28.561738 2582 scope.go:117] "RemoveContainer" containerID="7c1f855eaa555e0fbf75e53d5b9aba95d7040141b2c641b814344a773125f824" Jul 16 12:41:29.566652 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:29.566617 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-js9ms" event={"ID":"04daa141-3460-4290-98db-eccefb64fe38","Type":"ContainerStarted","Data":"f76b79e6a56a462741deecfa4d25611775b174b1fdab2b78f69ee80eeb455f98"} Jul 16 12:41:31.212913 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:31.212863 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-77f77d9d9b-4vlks" podUID="ce0f4c40-d236-4a0d-a84f-b9abe985c4d4" containerName="console" containerID="cri-o://c1883b58b54b68d6c2c1f3e66791d713e9f6304217d43e4be7c6f7474de4f935" gracePeriod=15 Jul 16 12:41:31.450674 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:31.450650 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77f77d9d9b-4vlks" Jul 16 12:41:31.551319 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:31.551248 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxqng\" (UniqueName: \"kubernetes.io/projected/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-kube-api-access-zxqng\") pod \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\" (UID: \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\") " Jul 16 12:41:31.551319 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:31.551281 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-trusted-ca-bundle\") pod \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\" (UID: \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\") " Jul 16 12:41:31.551319 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:31.551305 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-console-serving-cert\") pod \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\" (UID: \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\") " Jul 16 12:41:31.551606 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:31.551366 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-console-config\") pod \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\" (UID: \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\") " Jul 16 12:41:31.551606 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:31.551432 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-service-ca\") pod \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\" (UID: \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\") " Jul 16 12:41:31.551606 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:31.551465 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-oauth-serving-cert\") pod \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\" (UID: \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\") " Jul 16 12:41:31.551606 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:31.551490 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-console-oauth-config\") pod \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\" (UID: \"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4\") " Jul 16 12:41:31.551849 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:31.551817 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ce0f4c40-d236-4a0d-a84f-b9abe985c4d4" (UID: "ce0f4c40-d236-4a0d-a84f-b9abe985c4d4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:41:31.551915 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:31.551842 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-service-ca" (OuterVolumeSpecName: "service-ca") pod "ce0f4c40-d236-4a0d-a84f-b9abe985c4d4" (UID: "ce0f4c40-d236-4a0d-a84f-b9abe985c4d4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:41:31.551966 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:31.551886 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-console-config" (OuterVolumeSpecName: "console-config") pod "ce0f4c40-d236-4a0d-a84f-b9abe985c4d4" (UID: "ce0f4c40-d236-4a0d-a84f-b9abe985c4d4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:41:31.552246 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:31.552213 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ce0f4c40-d236-4a0d-a84f-b9abe985c4d4" (UID: "ce0f4c40-d236-4a0d-a84f-b9abe985c4d4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:41:31.553771 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:31.553744 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-kube-api-access-zxqng" (OuterVolumeSpecName: "kube-api-access-zxqng") pod "ce0f4c40-d236-4a0d-a84f-b9abe985c4d4" (UID: "ce0f4c40-d236-4a0d-a84f-b9abe985c4d4"). InnerVolumeSpecName "kube-api-access-zxqng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:41:31.553856 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:31.553750 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ce0f4c40-d236-4a0d-a84f-b9abe985c4d4" (UID: "ce0f4c40-d236-4a0d-a84f-b9abe985c4d4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:41:31.553856 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:31.553774 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ce0f4c40-d236-4a0d-a84f-b9abe985c4d4" (UID: "ce0f4c40-d236-4a0d-a84f-b9abe985c4d4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:41:31.574203 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:31.574172 2582 generic.go:356] "Generic (PLEG): container finished" podID="ce0f4c40-d236-4a0d-a84f-b9abe985c4d4" containerID="c1883b58b54b68d6c2c1f3e66791d713e9f6304217d43e4be7c6f7474de4f935" exitCode=0 Jul 16 12:41:31.574330 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:31.574220 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77f77d9d9b-4vlks" Jul 16 12:41:31.574330 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:31.574232 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77f77d9d9b-4vlks" event={"ID":"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4","Type":"ContainerDied","Data":"c1883b58b54b68d6c2c1f3e66791d713e9f6304217d43e4be7c6f7474de4f935"} Jul 16 12:41:31.574330 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:31.574280 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77f77d9d9b-4vlks" event={"ID":"ce0f4c40-d236-4a0d-a84f-b9abe985c4d4","Type":"ContainerDied","Data":"fa2599bf25c4f7a5bb1a5cfee4aabf4120cb821a4158d406f63e2e71aa2b9880"} Jul 16 12:41:31.574330 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:31.574306 2582 scope.go:117] "RemoveContainer" containerID="c1883b58b54b68d6c2c1f3e66791d713e9f6304217d43e4be7c6f7474de4f935" Jul 16 12:41:31.582289 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:31.582269 2582 scope.go:117] "RemoveContainer" containerID="c1883b58b54b68d6c2c1f3e66791d713e9f6304217d43e4be7c6f7474de4f935" Jul 16 12:41:31.582550 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:41:31.582533 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1883b58b54b68d6c2c1f3e66791d713e9f6304217d43e4be7c6f7474de4f935\": container with ID starting with c1883b58b54b68d6c2c1f3e66791d713e9f6304217d43e4be7c6f7474de4f935 not found: ID does not exist" containerID="c1883b58b54b68d6c2c1f3e66791d713e9f6304217d43e4be7c6f7474de4f935" Jul 16 12:41:31.582600 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:31.582558 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1883b58b54b68d6c2c1f3e66791d713e9f6304217d43e4be7c6f7474de4f935"} err="failed to get container status \"c1883b58b54b68d6c2c1f3e66791d713e9f6304217d43e4be7c6f7474de4f935\": rpc error: code = NotFound desc = could not find container \"c1883b58b54b68d6c2c1f3e66791d713e9f6304217d43e4be7c6f7474de4f935\": container with ID starting with c1883b58b54b68d6c2c1f3e66791d713e9f6304217d43e4be7c6f7474de4f935 not found: ID does not exist" Jul 16 12:41:31.592984 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:31.592954 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77f77d9d9b-4vlks"] Jul 16 12:41:31.595527 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:31.595509 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77f77d9d9b-4vlks"] Jul 16 12:41:31.652810 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:31.652794 2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-service-ca\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:41:31.652894 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:31.652812 2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-oauth-serving-cert\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:41:31.652894 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:31.652822 2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-console-oauth-config\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:41:31.652894 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:31.652832 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxqng\" (UniqueName: \"kubernetes.io/projected/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-kube-api-access-zxqng\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:41:31.652894 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:31.652840 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-trusted-ca-bundle\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:41:31.652894 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:31.652848 2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-console-serving-cert\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:41:31.652894 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:31.652856 2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4-console-config\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:41:31.975706 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:31.975663 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce0f4c40-d236-4a0d-a84f-b9abe985c4d4" path="/var/lib/kubelet/pods/ce0f4c40-d236-4a0d-a84f-b9abe985c4d4/volumes" Jul 16 12:41:49.276460 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:49.276420 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:41:57.823325 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:57.823250 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:41:57.905404 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:41:57.905355 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:42:09.312121 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:09.312092 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:42:09.413360 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:09.413327 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:42:12.593248 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.593217 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-645f7565bc-5c4pr"] Jul 16 12:42:12.593616 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.593470 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce0f4c40-d236-4a0d-a84f-b9abe985c4d4" containerName="console" Jul 16 12:42:12.593616 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.593481 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce0f4c40-d236-4a0d-a84f-b9abe985c4d4" containerName="console" Jul 16 12:42:12.593616 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.593490 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9a1a7fc-4864-48d9-8b45-477fedb54b17" containerName="registry" Jul 16 12:42:12.593616 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.593496 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9a1a7fc-4864-48d9-8b45-477fedb54b17" containerName="registry" Jul 16 12:42:12.593616 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.593541 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9a1a7fc-4864-48d9-8b45-477fedb54b17" containerName="registry" Jul 16 12:42:12.593616 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.593551 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce0f4c40-d236-4a0d-a84f-b9abe985c4d4" containerName="console" Jul 16 12:42:12.597676 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.597660 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-645f7565bc-5c4pr" Jul 16 12:42:12.600206 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.600183 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 16 12:42:12.600439 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.600424 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 16 12:42:12.601365 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.601348 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 16 12:42:12.601497 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.601474 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 16 12:42:12.601558 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.601536 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 16 12:42:12.601558 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.601477 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 16 12:42:12.601819 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.601799 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kv8xc\"" Jul 16 12:42:12.601931 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.601802 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 16 12:42:12.606592 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.606569 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 16 12:42:12.607823 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.607802 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-645f7565bc-5c4pr"] Jul 16 12:42:12.628111 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.628091 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef74c52a-35d9-4b5c-9186-a494580f4011-console-oauth-config\") pod \"console-645f7565bc-5c4pr\" (UID: \"ef74c52a-35d9-4b5c-9186-a494580f4011\") " pod="openshift-console/console-645f7565bc-5c4pr" Jul 16 12:42:12.628200 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.628116 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef74c52a-35d9-4b5c-9186-a494580f4011-service-ca\") pod \"console-645f7565bc-5c4pr\" (UID: \"ef74c52a-35d9-4b5c-9186-a494580f4011\") " pod="openshift-console/console-645f7565bc-5c4pr" Jul 16 12:42:12.628200 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.628138 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef74c52a-35d9-4b5c-9186-a494580f4011-oauth-serving-cert\") pod \"console-645f7565bc-5c4pr\" (UID: \"ef74c52a-35d9-4b5c-9186-a494580f4011\") " pod="openshift-console/console-645f7565bc-5c4pr" Jul 16 12:42:12.628200 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.628153 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmtqs\" (UniqueName: \"kubernetes.io/projected/ef74c52a-35d9-4b5c-9186-a494580f4011-kube-api-access-kmtqs\") pod \"console-645f7565bc-5c4pr\" (UID: \"ef74c52a-35d9-4b5c-9186-a494580f4011\") " pod="openshift-console/console-645f7565bc-5c4pr" Jul 16 12:42:12.628294 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.628217 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef74c52a-35d9-4b5c-9186-a494580f4011-console-config\") pod \"console-645f7565bc-5c4pr\" (UID: \"ef74c52a-35d9-4b5c-9186-a494580f4011\") " pod="openshift-console/console-645f7565bc-5c4pr" Jul 16 12:42:12.628294 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.628251 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef74c52a-35d9-4b5c-9186-a494580f4011-trusted-ca-bundle\") pod \"console-645f7565bc-5c4pr\" (UID: \"ef74c52a-35d9-4b5c-9186-a494580f4011\") " pod="openshift-console/console-645f7565bc-5c4pr" Jul 16 12:42:12.628358 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.628313 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef74c52a-35d9-4b5c-9186-a494580f4011-console-serving-cert\") pod \"console-645f7565bc-5c4pr\" (UID: \"ef74c52a-35d9-4b5c-9186-a494580f4011\") " pod="openshift-console/console-645f7565bc-5c4pr" Jul 16 12:42:12.729534 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.729503 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef74c52a-35d9-4b5c-9186-a494580f4011-console-oauth-config\") pod \"console-645f7565bc-5c4pr\" (UID: \"ef74c52a-35d9-4b5c-9186-a494580f4011\") " pod="openshift-console/console-645f7565bc-5c4pr" Jul 16 12:42:12.729534 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.729536 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef74c52a-35d9-4b5c-9186-a494580f4011-service-ca\") pod \"console-645f7565bc-5c4pr\" (UID: \"ef74c52a-35d9-4b5c-9186-a494580f4011\") " pod="openshift-console/console-645f7565bc-5c4pr" Jul 16 12:42:12.729748 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.729557 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef74c52a-35d9-4b5c-9186-a494580f4011-oauth-serving-cert\") pod \"console-645f7565bc-5c4pr\" (UID: \"ef74c52a-35d9-4b5c-9186-a494580f4011\") " pod="openshift-console/console-645f7565bc-5c4pr" Jul 16 12:42:12.729748 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.729572 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmtqs\" (UniqueName: \"kubernetes.io/projected/ef74c52a-35d9-4b5c-9186-a494580f4011-kube-api-access-kmtqs\") pod \"console-645f7565bc-5c4pr\" (UID: \"ef74c52a-35d9-4b5c-9186-a494580f4011\") " pod="openshift-console/console-645f7565bc-5c4pr" Jul 16 12:42:12.729748 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.729591 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef74c52a-35d9-4b5c-9186-a494580f4011-console-config\") pod \"console-645f7565bc-5c4pr\" (UID: \"ef74c52a-35d9-4b5c-9186-a494580f4011\") " pod="openshift-console/console-645f7565bc-5c4pr" Jul 16 12:42:12.729748 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.729612 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef74c52a-35d9-4b5c-9186-a494580f4011-trusted-ca-bundle\") pod \"console-645f7565bc-5c4pr\" (UID: \"ef74c52a-35d9-4b5c-9186-a494580f4011\") " pod="openshift-console/console-645f7565bc-5c4pr" Jul 16 12:42:12.729748 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.729650 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef74c52a-35d9-4b5c-9186-a494580f4011-console-serving-cert\") pod \"console-645f7565bc-5c4pr\" (UID: \"ef74c52a-35d9-4b5c-9186-a494580f4011\") " pod="openshift-console/console-645f7565bc-5c4pr" Jul 16 12:42:12.730460 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.730334 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef74c52a-35d9-4b5c-9186-a494580f4011-service-ca\") pod \"console-645f7565bc-5c4pr\" (UID: \"ef74c52a-35d9-4b5c-9186-a494580f4011\") " pod="openshift-console/console-645f7565bc-5c4pr" Jul 16 12:42:12.730617 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.730503 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef74c52a-35d9-4b5c-9186-a494580f4011-trusted-ca-bundle\") pod \"console-645f7565bc-5c4pr\" (UID: \"ef74c52a-35d9-4b5c-9186-a494580f4011\") " pod="openshift-console/console-645f7565bc-5c4pr" Jul 16 12:42:12.730617 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.730358 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef74c52a-35d9-4b5c-9186-a494580f4011-oauth-serving-cert\") pod \"console-645f7565bc-5c4pr\" (UID: \"ef74c52a-35d9-4b5c-9186-a494580f4011\") " pod="openshift-console/console-645f7565bc-5c4pr" Jul 16 12:42:12.730617 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.730358 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef74c52a-35d9-4b5c-9186-a494580f4011-console-config\") pod \"console-645f7565bc-5c4pr\" (UID: \"ef74c52a-35d9-4b5c-9186-a494580f4011\") " pod="openshift-console/console-645f7565bc-5c4pr" Jul 16 12:42:12.732182 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.732154 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef74c52a-35d9-4b5c-9186-a494580f4011-console-serving-cert\") pod \"console-645f7565bc-5c4pr\" (UID: \"ef74c52a-35d9-4b5c-9186-a494580f4011\") " pod="openshift-console/console-645f7565bc-5c4pr" Jul 16 12:42:12.732305 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.732283 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef74c52a-35d9-4b5c-9186-a494580f4011-console-oauth-config\") pod \"console-645f7565bc-5c4pr\" (UID: \"ef74c52a-35d9-4b5c-9186-a494580f4011\") " pod="openshift-console/console-645f7565bc-5c4pr" Jul 16 12:42:12.736677 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.736657 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmtqs\" (UniqueName: \"kubernetes.io/projected/ef74c52a-35d9-4b5c-9186-a494580f4011-kube-api-access-kmtqs\") pod \"console-645f7565bc-5c4pr\" (UID: \"ef74c52a-35d9-4b5c-9186-a494580f4011\") " pod="openshift-console/console-645f7565bc-5c4pr" Jul 16 12:42:12.907990 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:12.907962 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-645f7565bc-5c4pr" Jul 16 12:42:13.034868 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:13.034827 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-645f7565bc-5c4pr"] Jul 16 12:42:13.038525 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:42:13.038500 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef74c52a_35d9_4b5c_9186_a494580f4011.slice/crio-0674d56022d68b1ee471f6b3f6a9ad2eb87d0034ac860bb67620847dc0d004b6 WatchSource:0}: Error finding container 0674d56022d68b1ee471f6b3f6a9ad2eb87d0034ac860bb67620847dc0d004b6: Status 404 returned error can't find the container with id 0674d56022d68b1ee471f6b3f6a9ad2eb87d0034ac860bb67620847dc0d004b6 Jul 16 12:42:13.089811 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:13.089789 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:42:13.210356 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:13.210276 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:42:13.719534 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:13.719502 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-645f7565bc-5c4pr" event={"ID":"ef74c52a-35d9-4b5c-9186-a494580f4011","Type":"ContainerStarted","Data":"6c0156808f4520c2b8b411d7d5532f77f2b3d76f1a162cb3f8b51a4b5101704f"} Jul 16 12:42:13.719917 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:13.719550 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-645f7565bc-5c4pr" event={"ID":"ef74c52a-35d9-4b5c-9186-a494580f4011","Type":"ContainerStarted","Data":"0674d56022d68b1ee471f6b3f6a9ad2eb87d0034ac860bb67620847dc0d004b6"} Jul 16 12:42:13.738182 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:13.738136 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-645f7565bc-5c4pr" podStartSLOduration=1.7381216689999999 podStartE2EDuration="1.738121669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:42:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:42:13.73719219 +0000 UTC m=+192.214937806" watchObservedRunningTime="2026-07-16 12:42:13.738121669 +0000 UTC m=+192.215867284" Jul 16 12:42:19.353264 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:19.353224 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:42:19.940807 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:19.940765 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:42:22.908892 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:22.908856 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-645f7565bc-5c4pr" Jul 16 12:42:22.909498 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:22.908926 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-645f7565bc-5c4pr" Jul 16 12:42:22.913762 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:22.913737 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-645f7565bc-5c4pr" Jul 16 12:42:23.757268 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:23.757238 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-645f7565bc-5c4pr" Jul 16 12:42:23.912768 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:23.912738 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:42:24.014154 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:24.014075 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:42:24.488074 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:24.488045 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:42:39.096377 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:39.096341 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:42:39.189051 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:39.189026 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:42:39.743829 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:39.743795 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qdk8w"] Jul 16 12:42:39.747195 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:39.747178 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qdk8w" Jul 16 12:42:39.753106 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:39.752969 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 16 12:42:39.753106 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:39.752991 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 16 12:42:39.753335 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:39.753158 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mq68f\"" Jul 16 12:42:39.755264 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:39.755237 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qdk8w"] Jul 16 12:42:39.825278 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:39.825248 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49d9d2b1-2602-4dc6-b03c-fe148aec141a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qdk8w\" (UID: \"49d9d2b1-2602-4dc6-b03c-fe148aec141a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qdk8w" Jul 16 12:42:39.825465 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:39.825287 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49d9d2b1-2602-4dc6-b03c-fe148aec141a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qdk8w\" (UID: \"49d9d2b1-2602-4dc6-b03c-fe148aec141a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qdk8w" Jul 16 12:42:39.825465 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:39.825353 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfkjw\" (UniqueName: \"kubernetes.io/projected/49d9d2b1-2602-4dc6-b03c-fe148aec141a-kube-api-access-lfkjw\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qdk8w\" (UID: \"49d9d2b1-2602-4dc6-b03c-fe148aec141a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qdk8w" Jul 16 12:42:39.926628 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:39.926591 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfkjw\" (UniqueName: \"kubernetes.io/projected/49d9d2b1-2602-4dc6-b03c-fe148aec141a-kube-api-access-lfkjw\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qdk8w\" (UID: \"49d9d2b1-2602-4dc6-b03c-fe148aec141a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qdk8w" Jul 16 12:42:39.926815 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:39.926650 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49d9d2b1-2602-4dc6-b03c-fe148aec141a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qdk8w\" (UID: \"49d9d2b1-2602-4dc6-b03c-fe148aec141a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qdk8w" Jul 16 12:42:39.926815 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:39.926685 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49d9d2b1-2602-4dc6-b03c-fe148aec141a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qdk8w\" (UID: \"49d9d2b1-2602-4dc6-b03c-fe148aec141a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qdk8w" Jul 16 12:42:39.927093 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:39.927070 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49d9d2b1-2602-4dc6-b03c-fe148aec141a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qdk8w\" (UID: \"49d9d2b1-2602-4dc6-b03c-fe148aec141a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qdk8w" Jul 16 12:42:39.927093 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:39.927082 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49d9d2b1-2602-4dc6-b03c-fe148aec141a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qdk8w\" (UID: \"49d9d2b1-2602-4dc6-b03c-fe148aec141a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qdk8w" Jul 16 12:42:39.935679 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:39.935656 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfkjw\" (UniqueName: \"kubernetes.io/projected/49d9d2b1-2602-4dc6-b03c-fe148aec141a-kube-api-access-lfkjw\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qdk8w\" (UID: \"49d9d2b1-2602-4dc6-b03c-fe148aec141a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qdk8w" Jul 16 12:42:40.060661 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:40.060566 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qdk8w" Jul 16 12:42:40.186344 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:40.186314 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qdk8w"] Jul 16 12:42:40.189289 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:42:40.189261 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d9d2b1_2602_4dc6_b03c_fe148aec141a.slice/crio-7550e4335effe0c121f79770c56682b5cd7375ccd0aacc82d746c196d8261aad WatchSource:0}: Error finding container 7550e4335effe0c121f79770c56682b5cd7375ccd0aacc82d746c196d8261aad: Status 404 returned error can't find the container with id 7550e4335effe0c121f79770c56682b5cd7375ccd0aacc82d746c196d8261aad Jul 16 12:42:40.814606 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:40.814574 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qdk8w" event={"ID":"49d9d2b1-2602-4dc6-b03c-fe148aec141a","Type":"ContainerStarted","Data":"7550e4335effe0c121f79770c56682b5cd7375ccd0aacc82d746c196d8261aad"} Jul 16 12:42:43.716219 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:43.716187 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:42:43.816582 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:43.816548 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:42:46.839408 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:46.839292 2582 generic.go:356] "Generic (PLEG): container finished" podID="49d9d2b1-2602-4dc6-b03c-fe148aec141a" containerID="a48a731c4e14920731bc08b02cd913f2a97222c45ed3534956b80ba9bb91b57f" exitCode=0 Jul 16 12:42:46.839408 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:46.839381 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qdk8w" event={"ID":"49d9d2b1-2602-4dc6-b03c-fe148aec141a","Type":"ContainerDied","Data":"a48a731c4e14920731bc08b02cd913f2a97222c45ed3534956b80ba9bb91b57f"} Jul 16 12:42:49.852780 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:49.852744 2582 generic.go:356] "Generic (PLEG): container finished" podID="49d9d2b1-2602-4dc6-b03c-fe148aec141a" containerID="ddc4abf2c5066520fb8c71edc4cb46067bd5b69ac878c2d2feb72b3c715096dc" exitCode=0 Jul 16 12:42:49.853149 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:49.852818 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qdk8w" event={"ID":"49d9d2b1-2602-4dc6-b03c-fe148aec141a","Type":"ContainerDied","Data":"ddc4abf2c5066520fb8c71edc4cb46067bd5b69ac878c2d2feb72b3c715096dc"} Jul 16 12:42:56.880740 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:56.880700 2582 generic.go:356] "Generic (PLEG): container finished" podID="49d9d2b1-2602-4dc6-b03c-fe148aec141a" containerID="0222541cee17cbec368649ad9446384f3268bc01751d7d840313abfe3960572c" exitCode=0 Jul 16 12:42:56.881109 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:56.880786 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qdk8w" event={"ID":"49d9d2b1-2602-4dc6-b03c-fe148aec141a","Type":"ContainerDied","Data":"0222541cee17cbec368649ad9446384f3268bc01751d7d840313abfe3960572c"} Jul 16 12:42:58.011058 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:58.011035 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qdk8w" Jul 16 12:42:58.175651 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:58.175560 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49d9d2b1-2602-4dc6-b03c-fe148aec141a-util\") pod \"49d9d2b1-2602-4dc6-b03c-fe148aec141a\" (UID: \"49d9d2b1-2602-4dc6-b03c-fe148aec141a\") " Jul 16 12:42:58.175651 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:58.175651 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfkjw\" (UniqueName: \"kubernetes.io/projected/49d9d2b1-2602-4dc6-b03c-fe148aec141a-kube-api-access-lfkjw\") pod \"49d9d2b1-2602-4dc6-b03c-fe148aec141a\" (UID: \"49d9d2b1-2602-4dc6-b03c-fe148aec141a\") " Jul 16 12:42:58.175870 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:58.175675 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49d9d2b1-2602-4dc6-b03c-fe148aec141a-bundle\") pod \"49d9d2b1-2602-4dc6-b03c-fe148aec141a\" (UID: \"49d9d2b1-2602-4dc6-b03c-fe148aec141a\") " Jul 16 12:42:58.176303 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:58.176281 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d9d2b1-2602-4dc6-b03c-fe148aec141a-bundle" (OuterVolumeSpecName: "bundle") pod "49d9d2b1-2602-4dc6-b03c-fe148aec141a" (UID: "49d9d2b1-2602-4dc6-b03c-fe148aec141a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:42:58.178006 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:58.177984 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49d9d2b1-2602-4dc6-b03c-fe148aec141a-kube-api-access-lfkjw" (OuterVolumeSpecName: "kube-api-access-lfkjw") pod "49d9d2b1-2602-4dc6-b03c-fe148aec141a" (UID: "49d9d2b1-2602-4dc6-b03c-fe148aec141a"). InnerVolumeSpecName "kube-api-access-lfkjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:42:58.179866 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:58.179843 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d9d2b1-2602-4dc6-b03c-fe148aec141a-util" (OuterVolumeSpecName: "util") pod "49d9d2b1-2602-4dc6-b03c-fe148aec141a" (UID: "49d9d2b1-2602-4dc6-b03c-fe148aec141a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:42:58.276502 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:58.276466 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfkjw\" (UniqueName: \"kubernetes.io/projected/49d9d2b1-2602-4dc6-b03c-fe148aec141a-kube-api-access-lfkjw\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:42:58.276502 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:58.276494 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49d9d2b1-2602-4dc6-b03c-fe148aec141a-bundle\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:42:58.276502 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:58.276504 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49d9d2b1-2602-4dc6-b03c-fe148aec141a-util\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:42:58.890215 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:58.890188 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qdk8w" Jul 16 12:42:58.890343 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:58.890181 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qdk8w" event={"ID":"49d9d2b1-2602-4dc6-b03c-fe148aec141a","Type":"ContainerDied","Data":"7550e4335effe0c121f79770c56682b5cd7375ccd0aacc82d746c196d8261aad"} Jul 16 12:42:58.890343 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:42:58.890312 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7550e4335effe0c121f79770c56682b5cd7375ccd0aacc82d746c196d8261aad" Jul 16 12:43:00.712994 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:00.712965 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:01.646766 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:01.646728 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:04.330408 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:04.330359 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:04.435767 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:04.435735 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:06.714267 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:06.714232 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:06.810882 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:06.810851 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:07.028755 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:07.028679 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:07.127196 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:07.127165 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:07.180711 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:07.180686 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frcp5f"] Jul 16 12:43:07.180926 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:07.180914 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49d9d2b1-2602-4dc6-b03c-fe148aec141a" containerName="extract" Jul 16 12:43:07.180966 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:07.180927 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="49d9d2b1-2602-4dc6-b03c-fe148aec141a" containerName="extract" Jul 16 12:43:07.180966 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:07.180945 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49d9d2b1-2602-4dc6-b03c-fe148aec141a" containerName="util" Jul 16 12:43:07.180966 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:07.180950 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="49d9d2b1-2602-4dc6-b03c-fe148aec141a" containerName="util" Jul 16 12:43:07.180966 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:07.180957 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49d9d2b1-2602-4dc6-b03c-fe148aec141a" containerName="pull" Jul 16 12:43:07.180966 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:07.180962 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="49d9d2b1-2602-4dc6-b03c-fe148aec141a" containerName="pull" Jul 16 12:43:07.181103 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:07.181001 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="49d9d2b1-2602-4dc6-b03c-fe148aec141a" containerName="extract" Jul 16 12:43:07.184788 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:07.184772 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frcp5f" Jul 16 12:43:07.187231 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:07.187215 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 16 12:43:07.187331 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:07.187312 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mq68f\"" Jul 16 12:43:07.187424 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:07.187404 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 16 12:43:07.194139 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:07.194121 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frcp5f"] Jul 16 12:43:07.242851 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:07.242821 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2czq9\" (UniqueName: \"kubernetes.io/projected/5dbe1b50-5885-492b-a811-68b031e33f38-kube-api-access-2czq9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frcp5f\" (UID: \"5dbe1b50-5885-492b-a811-68b031e33f38\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frcp5f" Jul 16 12:43:07.242980 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:07.242865 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5dbe1b50-5885-492b-a811-68b031e33f38-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frcp5f\" (UID: \"5dbe1b50-5885-492b-a811-68b031e33f38\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frcp5f" Jul 16 12:43:07.242980 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:07.242883 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5dbe1b50-5885-492b-a811-68b031e33f38-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frcp5f\" (UID: \"5dbe1b50-5885-492b-a811-68b031e33f38\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frcp5f" Jul 16 12:43:07.292753 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:07.292690 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:07.343953 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:07.343926 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5dbe1b50-5885-492b-a811-68b031e33f38-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frcp5f\" (UID: \"5dbe1b50-5885-492b-a811-68b031e33f38\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frcp5f" Jul 16 12:43:07.344063 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:07.343956 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5dbe1b50-5885-492b-a811-68b031e33f38-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frcp5f\" (UID: \"5dbe1b50-5885-492b-a811-68b031e33f38\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frcp5f" Jul 16 12:43:07.344063 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:07.344009 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2czq9\" (UniqueName: \"kubernetes.io/projected/5dbe1b50-5885-492b-a811-68b031e33f38-kube-api-access-2czq9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frcp5f\" (UID: \"5dbe1b50-5885-492b-a811-68b031e33f38\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frcp5f" Jul 16 12:43:07.344302 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:07.344285 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5dbe1b50-5885-492b-a811-68b031e33f38-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frcp5f\" (UID: \"5dbe1b50-5885-492b-a811-68b031e33f38\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frcp5f" Jul 16 12:43:07.344337 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:07.344312 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5dbe1b50-5885-492b-a811-68b031e33f38-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frcp5f\" (UID: \"5dbe1b50-5885-492b-a811-68b031e33f38\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frcp5f" Jul 16 12:43:07.352066 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:07.352039 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2czq9\" (UniqueName: \"kubernetes.io/projected/5dbe1b50-5885-492b-a811-68b031e33f38-kube-api-access-2czq9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frcp5f\" (UID: \"5dbe1b50-5885-492b-a811-68b031e33f38\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frcp5f" Jul 16 12:43:07.494888 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:07.494848 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frcp5f" Jul 16 12:43:07.620362 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:07.620252 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frcp5f"] Jul 16 12:43:07.622820 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:43:07.622793 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dbe1b50_5885_492b_a811_68b031e33f38.slice/crio-de1d773c6b4802301d01a53874979092a47cb81eccae2319c123642155e5fcd5 WatchSource:0}: Error finding container de1d773c6b4802301d01a53874979092a47cb81eccae2319c123642155e5fcd5: Status 404 returned error can't find the container with id de1d773c6b4802301d01a53874979092a47cb81eccae2319c123642155e5fcd5 Jul 16 12:43:07.924118 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:07.924087 2582 generic.go:356] "Generic (PLEG): container finished" podID="5dbe1b50-5885-492b-a811-68b031e33f38" containerID="b2b65f33a8c0981d2fdfe28cb38713926155ee733cfe45b345bd0689577a0759" exitCode=0 Jul 16 12:43:07.924524 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:07.924175 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frcp5f" event={"ID":"5dbe1b50-5885-492b-a811-68b031e33f38","Type":"ContainerDied","Data":"b2b65f33a8c0981d2fdfe28cb38713926155ee733cfe45b345bd0689577a0759"} Jul 16 12:43:07.924524 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:07.924273 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frcp5f" event={"ID":"5dbe1b50-5885-492b-a811-68b031e33f38","Type":"ContainerStarted","Data":"de1d773c6b4802301d01a53874979092a47cb81eccae2319c123642155e5fcd5"} Jul 16 12:43:08.333832 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:08.333735 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:08.434586 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:08.434553 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:08.720671 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:08.720637 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:09.114852 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:09.112798 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:09.323468 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:09.323436 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-m5cpm"] Jul 16 12:43:09.328850 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:09.328823 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-m5cpm" Jul 16 12:43:09.331352 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:09.331325 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-x4vmt\"" Jul 16 12:43:09.332189 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:09.332157 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 16 12:43:09.332322 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:09.332263 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 16 12:43:09.334314 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:09.334292 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-m5cpm"] Jul 16 12:43:09.358608 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:09.358578 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8180ec52-de4d-4bd2-b2e8-6a9bb3cb42d6-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-m5cpm\" (UID: \"8180ec52-de4d-4bd2-b2e8-6a9bb3cb42d6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m5cpm" Jul 16 12:43:09.358738 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:09.358634 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz6qn\" (UniqueName: \"kubernetes.io/projected/8180ec52-de4d-4bd2-b2e8-6a9bb3cb42d6-kube-api-access-hz6qn\") pod \"cert-manager-cainjector-559cdd8c87-m5cpm\" (UID: \"8180ec52-de4d-4bd2-b2e8-6a9bb3cb42d6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m5cpm" Jul 16 12:43:09.459702 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:09.459660 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8180ec52-de4d-4bd2-b2e8-6a9bb3cb42d6-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-m5cpm\" (UID: \"8180ec52-de4d-4bd2-b2e8-6a9bb3cb42d6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m5cpm" Jul 16 12:43:09.459888 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:09.459723 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz6qn\" (UniqueName: \"kubernetes.io/projected/8180ec52-de4d-4bd2-b2e8-6a9bb3cb42d6-kube-api-access-hz6qn\") pod \"cert-manager-cainjector-559cdd8c87-m5cpm\" (UID: \"8180ec52-de4d-4bd2-b2e8-6a9bb3cb42d6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m5cpm" Jul 16 12:43:09.468405 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:09.468364 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8180ec52-de4d-4bd2-b2e8-6a9bb3cb42d6-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-m5cpm\" (UID: \"8180ec52-de4d-4bd2-b2e8-6a9bb3cb42d6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m5cpm" Jul 16 12:43:09.468998 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:09.468977 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz6qn\" (UniqueName: \"kubernetes.io/projected/8180ec52-de4d-4bd2-b2e8-6a9bb3cb42d6-kube-api-access-hz6qn\") pod \"cert-manager-cainjector-559cdd8c87-m5cpm\" (UID: \"8180ec52-de4d-4bd2-b2e8-6a9bb3cb42d6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m5cpm" Jul 16 12:43:09.640172 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:09.640137 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-m5cpm" Jul 16 12:43:09.928764 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:09.928732 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-m5cpm"] Jul 16 12:43:09.930626 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:43:09.930585 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8180ec52_de4d_4bd2_b2e8_6a9bb3cb42d6.slice/crio-3efa3b715ecf318b5a5babe5c3317978f1c804fe60b42ba96b3b005fc19f28f1 WatchSource:0}: Error finding container 3efa3b715ecf318b5a5babe5c3317978f1c804fe60b42ba96b3b005fc19f28f1: Status 404 returned error can't find the container with id 3efa3b715ecf318b5a5babe5c3317978f1c804fe60b42ba96b3b005fc19f28f1 Jul 16 12:43:09.935248 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:09.935219 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frcp5f" event={"ID":"5dbe1b50-5885-492b-a811-68b031e33f38","Type":"ContainerStarted","Data":"278040a63e246fcfd65d0f1be732aa6a53489f4aa8eab85fadc62419ee572663"} Jul 16 12:43:10.940157 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:10.940126 2582 generic.go:356] "Generic (PLEG): container finished" podID="5dbe1b50-5885-492b-a811-68b031e33f38" containerID="278040a63e246fcfd65d0f1be732aa6a53489f4aa8eab85fadc62419ee572663" exitCode=0 Jul 16 12:43:10.940579 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:10.940198 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frcp5f" event={"ID":"5dbe1b50-5885-492b-a811-68b031e33f38","Type":"ContainerDied","Data":"278040a63e246fcfd65d0f1be732aa6a53489f4aa8eab85fadc62419ee572663"} Jul 16 12:43:10.941500 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:10.941480 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-m5cpm" event={"ID":"8180ec52-de4d-4bd2-b2e8-6a9bb3cb42d6","Type":"ContainerStarted","Data":"3efa3b715ecf318b5a5babe5c3317978f1c804fe60b42ba96b3b005fc19f28f1"} Jul 16 12:43:11.391755 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:11.391719 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:11.448757 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:11.448727 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:11.946408 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:11.946352 2582 generic.go:356] "Generic (PLEG): container finished" podID="5dbe1b50-5885-492b-a811-68b031e33f38" containerID="92e29ea4dc7bd1852a28ae984f36ba484c8aa83058cb6c286dc54c793d8d3a17" exitCode=0 Jul 16 12:43:11.946969 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:11.946423 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frcp5f" event={"ID":"5dbe1b50-5885-492b-a811-68b031e33f38","Type":"ContainerDied","Data":"92e29ea4dc7bd1852a28ae984f36ba484c8aa83058cb6c286dc54c793d8d3a17"} Jul 16 12:43:13.092687 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:13.092663 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frcp5f" Jul 16 12:43:13.186967 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:13.186934 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2czq9\" (UniqueName: \"kubernetes.io/projected/5dbe1b50-5885-492b-a811-68b031e33f38-kube-api-access-2czq9\") pod \"5dbe1b50-5885-492b-a811-68b031e33f38\" (UID: \"5dbe1b50-5885-492b-a811-68b031e33f38\") " Jul 16 12:43:13.187143 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:13.186994 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5dbe1b50-5885-492b-a811-68b031e33f38-bundle\") pod \"5dbe1b50-5885-492b-a811-68b031e33f38\" (UID: \"5dbe1b50-5885-492b-a811-68b031e33f38\") " Jul 16 12:43:13.187473 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:13.187428 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbe1b50-5885-492b-a811-68b031e33f38-bundle" (OuterVolumeSpecName: "bundle") pod "5dbe1b50-5885-492b-a811-68b031e33f38" (UID: "5dbe1b50-5885-492b-a811-68b031e33f38"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:43:13.189355 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:13.189328 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dbe1b50-5885-492b-a811-68b031e33f38-kube-api-access-2czq9" (OuterVolumeSpecName: "kube-api-access-2czq9") pod "5dbe1b50-5885-492b-a811-68b031e33f38" (UID: "5dbe1b50-5885-492b-a811-68b031e33f38"). InnerVolumeSpecName "kube-api-access-2czq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:43:13.287968 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:13.287891 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5dbe1b50-5885-492b-a811-68b031e33f38-util\") pod \"5dbe1b50-5885-492b-a811-68b031e33f38\" (UID: \"5dbe1b50-5885-492b-a811-68b031e33f38\") " Jul 16 12:43:13.288101 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:13.288031 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2czq9\" (UniqueName: \"kubernetes.io/projected/5dbe1b50-5885-492b-a811-68b031e33f38-kube-api-access-2czq9\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:43:13.288101 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:13.288051 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5dbe1b50-5885-492b-a811-68b031e33f38-bundle\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:43:13.292908 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:13.292879 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbe1b50-5885-492b-a811-68b031e33f38-util" (OuterVolumeSpecName: "util") pod "5dbe1b50-5885-492b-a811-68b031e33f38" (UID: "5dbe1b50-5885-492b-a811-68b031e33f38"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:43:13.389053 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:13.389016 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5dbe1b50-5885-492b-a811-68b031e33f38-util\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:43:13.957725 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:13.957688 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frcp5f" event={"ID":"5dbe1b50-5885-492b-a811-68b031e33f38","Type":"ContainerDied","Data":"de1d773c6b4802301d01a53874979092a47cb81eccae2319c123642155e5fcd5"} Jul 16 12:43:13.957725 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:13.957715 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frcp5f" Jul 16 12:43:13.957926 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:13.957742 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de1d773c6b4802301d01a53874979092a47cb81eccae2319c123642155e5fcd5" Jul 16 12:43:14.963610 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:14.963572 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-m5cpm" event={"ID":"8180ec52-de4d-4bd2-b2e8-6a9bb3cb42d6","Type":"ContainerStarted","Data":"c5676a7602c3dd3b33f2c5cedd796cb14db4159d18d97f3662769036505e624c"} Jul 16 12:43:14.980077 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:14.980028 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-m5cpm" podStartSLOduration=1.3310185909999999 podStartE2EDuration="5.980013071s" totalImagesPullingTime="4.64899448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:43:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:43:14.979087775 +0000 UTC m=+253.456833392" watchObservedRunningTime="2026-07-16 12:43:14.980013071 +0000 UTC m=+253.457758687" Jul 16 12:43:15.361766 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:15.361692 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:15.442651 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:15.442607 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:16.900894 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:16.900847 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:17.931788 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:17.931757 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:17.989449 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:17.989418 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:19.747170 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:19.747137 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w75pc"] Jul 16 12:43:19.747543 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:19.747365 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dbe1b50-5885-492b-a811-68b031e33f38" containerName="util" Jul 16 12:43:19.747543 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:19.747375 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dbe1b50-5885-492b-a811-68b031e33f38" containerName="util" Jul 16 12:43:19.747543 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:19.747401 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dbe1b50-5885-492b-a811-68b031e33f38" containerName="pull" Jul 16 12:43:19.747543 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:19.747406 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dbe1b50-5885-492b-a811-68b031e33f38" containerName="pull" Jul 16 12:43:19.747543 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:19.747419 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dbe1b50-5885-492b-a811-68b031e33f38" containerName="extract" Jul 16 12:43:19.747543 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:19.747424 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dbe1b50-5885-492b-a811-68b031e33f38" containerName="extract" Jul 16 12:43:19.747543 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:19.747472 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dbe1b50-5885-492b-a811-68b031e33f38" containerName="extract" Jul 16 12:43:19.749064 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:19.749048 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w75pc" Jul 16 12:43:19.751403 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:19.751369 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 16 12:43:19.752429 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:19.752410 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 16 12:43:19.752550 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:19.752454 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-vsnqg\"" Jul 16 12:43:19.758016 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:19.757996 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w75pc"] Jul 16 12:43:19.837020 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:19.836987 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dj8n\" (UniqueName: \"kubernetes.io/projected/733e00d9-ce5a-4a74-8990-746cd68081f5-kube-api-access-4dj8n\") pod \"openshift-lws-operator-fd8ccff4c-w75pc\" (UID: \"733e00d9-ce5a-4a74-8990-746cd68081f5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w75pc" Jul 16 12:43:19.837163 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:19.837101 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/733e00d9-ce5a-4a74-8990-746cd68081f5-tmp\") pod \"openshift-lws-operator-fd8ccff4c-w75pc\" (UID: \"733e00d9-ce5a-4a74-8990-746cd68081f5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w75pc" Jul 16 12:43:19.937993 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:19.937965 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/733e00d9-ce5a-4a74-8990-746cd68081f5-tmp\") pod \"openshift-lws-operator-fd8ccff4c-w75pc\" (UID: \"733e00d9-ce5a-4a74-8990-746cd68081f5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w75pc" Jul 16 12:43:19.938130 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:19.937999 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dj8n\" (UniqueName: \"kubernetes.io/projected/733e00d9-ce5a-4a74-8990-746cd68081f5-kube-api-access-4dj8n\") pod \"openshift-lws-operator-fd8ccff4c-w75pc\" (UID: \"733e00d9-ce5a-4a74-8990-746cd68081f5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w75pc" Jul 16 12:43:19.938338 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:19.938318 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/733e00d9-ce5a-4a74-8990-746cd68081f5-tmp\") pod \"openshift-lws-operator-fd8ccff4c-w75pc\" (UID: \"733e00d9-ce5a-4a74-8990-746cd68081f5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w75pc" Jul 16 12:43:19.946005 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:19.945973 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dj8n\" (UniqueName: \"kubernetes.io/projected/733e00d9-ce5a-4a74-8990-746cd68081f5-kube-api-access-4dj8n\") pod \"openshift-lws-operator-fd8ccff4c-w75pc\" (UID: \"733e00d9-ce5a-4a74-8990-746cd68081f5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w75pc" Jul 16 12:43:20.057970 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:20.057894 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w75pc" Jul 16 12:43:20.207363 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:20.207337 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w75pc"] Jul 16 12:43:20.209845 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:43:20.209816 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod733e00d9_ce5a_4a74_8990_746cd68081f5.slice/crio-423ebf346bc2a69b25b7a808fd8a3989cf2592ef3c121b57b84031d78bbb9d9f WatchSource:0}: Error finding container 423ebf346bc2a69b25b7a808fd8a3989cf2592ef3c121b57b84031d78bbb9d9f: Status 404 returned error can't find the container with id 423ebf346bc2a69b25b7a808fd8a3989cf2592ef3c121b57b84031d78bbb9d9f Jul 16 12:43:20.990504 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:20.990463 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w75pc" event={"ID":"733e00d9-ce5a-4a74-8990-746cd68081f5","Type":"ContainerStarted","Data":"423ebf346bc2a69b25b7a808fd8a3989cf2592ef3c121b57b84031d78bbb9d9f"} Jul 16 12:43:23.000278 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:23.000238 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w75pc" event={"ID":"733e00d9-ce5a-4a74-8990-746cd68081f5","Type":"ContainerStarted","Data":"21874b74c93d908509dde8e579b9974a126b14de263aba2a563614f8d5c46911"} Jul 16 12:43:23.017183 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:23.017139 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w75pc" podStartSLOduration=1.876993477 podStartE2EDuration="4.017124435s" totalImagesPullingTime="2.140130958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:43:23.016058188 +0000 UTC m=+261.493803805" watchObservedRunningTime="2026-07-16 12:43:23.017124435 +0000 UTC m=+261.494870050" Jul 16 12:43:23.741535 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:23.741501 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:23.841820 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:23.841786 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:25.734184 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:25.734155 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:26.799844 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:26.799810 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrvnz5"] Jul 16 12:43:26.803133 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:26.803114 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrvnz5" Jul 16 12:43:26.805577 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:26.805556 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 16 12:43:26.805654 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:26.805563 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 16 12:43:26.805654 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:26.805649 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mq68f\"" Jul 16 12:43:26.810959 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:26.810932 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrvnz5"] Jul 16 12:43:26.890205 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:26.890173 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/917cb1f5-ea2e-45e2-9a9b-efcc585ec376-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrvnz5\" (UID: \"917cb1f5-ea2e-45e2-9a9b-efcc585ec376\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrvnz5" Jul 16 12:43:26.890360 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:26.890231 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/917cb1f5-ea2e-45e2-9a9b-efcc585ec376-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrvnz5\" (UID: \"917cb1f5-ea2e-45e2-9a9b-efcc585ec376\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrvnz5" Jul 16 12:43:26.890360 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:26.890268 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4rmd\" (UniqueName: \"kubernetes.io/projected/917cb1f5-ea2e-45e2-9a9b-efcc585ec376-kube-api-access-d4rmd\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrvnz5\" (UID: \"917cb1f5-ea2e-45e2-9a9b-efcc585ec376\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrvnz5" Jul 16 12:43:26.990758 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:26.990731 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/917cb1f5-ea2e-45e2-9a9b-efcc585ec376-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrvnz5\" (UID: \"917cb1f5-ea2e-45e2-9a9b-efcc585ec376\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrvnz5" Jul 16 12:43:26.990875 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:26.990790 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/917cb1f5-ea2e-45e2-9a9b-efcc585ec376-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrvnz5\" (UID: \"917cb1f5-ea2e-45e2-9a9b-efcc585ec376\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrvnz5" Jul 16 12:43:26.990923 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:26.990863 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4rmd\" (UniqueName: \"kubernetes.io/projected/917cb1f5-ea2e-45e2-9a9b-efcc585ec376-kube-api-access-d4rmd\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrvnz5\" (UID: \"917cb1f5-ea2e-45e2-9a9b-efcc585ec376\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrvnz5" Jul 16 12:43:26.991499 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:26.991443 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/917cb1f5-ea2e-45e2-9a9b-efcc585ec376-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrvnz5\" (UID: \"917cb1f5-ea2e-45e2-9a9b-efcc585ec376\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrvnz5" Jul 16 12:43:26.995654 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:26.991867 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/917cb1f5-ea2e-45e2-9a9b-efcc585ec376-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrvnz5\" (UID: \"917cb1f5-ea2e-45e2-9a9b-efcc585ec376\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrvnz5" Jul 16 12:43:26.999166 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:26.999148 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4rmd\" (UniqueName: \"kubernetes.io/projected/917cb1f5-ea2e-45e2-9a9b-efcc585ec376-kube-api-access-d4rmd\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrvnz5\" (UID: \"917cb1f5-ea2e-45e2-9a9b-efcc585ec376\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrvnz5" Jul 16 12:43:27.112587 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:27.112526 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrvnz5" Jul 16 12:43:27.243382 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:27.243339 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrvnz5"] Jul 16 12:43:27.246767 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:43:27.246731 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod917cb1f5_ea2e_45e2_9a9b_efcc585ec376.slice/crio-5f7ee2e0357ee3d5c57132563b829782b6dd7accefbb1aab639cf4a956a3c77b WatchSource:0}: Error finding container 5f7ee2e0357ee3d5c57132563b829782b6dd7accefbb1aab639cf4a956a3c77b: Status 404 returned error can't find the container with id 5f7ee2e0357ee3d5c57132563b829782b6dd7accefbb1aab639cf4a956a3c77b Jul 16 12:43:28.018594 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:28.018563 2582 generic.go:356] "Generic (PLEG): container finished" podID="917cb1f5-ea2e-45e2-9a9b-efcc585ec376" containerID="4f63a6621deac393e4343d374a5c244f39994ee8515329d5a4baf31a605ee954" exitCode=0 Jul 16 12:43:28.018899 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:28.018658 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrvnz5" event={"ID":"917cb1f5-ea2e-45e2-9a9b-efcc585ec376","Type":"ContainerDied","Data":"4f63a6621deac393e4343d374a5c244f39994ee8515329d5a4baf31a605ee954"} Jul 16 12:43:28.018899 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:28.018712 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrvnz5" event={"ID":"917cb1f5-ea2e-45e2-9a9b-efcc585ec376","Type":"ContainerStarted","Data":"5f7ee2e0357ee3d5c57132563b829782b6dd7accefbb1aab639cf4a956a3c77b"} Jul 16 12:43:30.027166 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:30.027129 2582 generic.go:356] "Generic (PLEG): container finished" podID="917cb1f5-ea2e-45e2-9a9b-efcc585ec376" containerID="b8b6780be05fddea8348bb049ffdd0e77056f449c303a5b0d16357c616b14d5e" exitCode=0 Jul 16 12:43:30.027166 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:30.027168 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrvnz5" event={"ID":"917cb1f5-ea2e-45e2-9a9b-efcc585ec376","Type":"ContainerDied","Data":"b8b6780be05fddea8348bb049ffdd0e77056f449c303a5b0d16357c616b14d5e"} Jul 16 12:43:31.032962 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:31.032924 2582 generic.go:356] "Generic (PLEG): container finished" podID="917cb1f5-ea2e-45e2-9a9b-efcc585ec376" containerID="6b252abb28bcd2c39560fa4b592d4d5a7b75e3c1734cbd78db6fb4982d24330f" exitCode=0 Jul 16 12:43:31.033338 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:31.033015 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrvnz5" event={"ID":"917cb1f5-ea2e-45e2-9a9b-efcc585ec376","Type":"ContainerDied","Data":"6b252abb28bcd2c39560fa4b592d4d5a7b75e3c1734cbd78db6fb4982d24330f"} Jul 16 12:43:32.159032 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:32.159005 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrvnz5" Jul 16 12:43:32.228923 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:32.228886 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4rmd\" (UniqueName: \"kubernetes.io/projected/917cb1f5-ea2e-45e2-9a9b-efcc585ec376-kube-api-access-d4rmd\") pod \"917cb1f5-ea2e-45e2-9a9b-efcc585ec376\" (UID: \"917cb1f5-ea2e-45e2-9a9b-efcc585ec376\") " Jul 16 12:43:32.229090 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:32.228938 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/917cb1f5-ea2e-45e2-9a9b-efcc585ec376-bundle\") pod \"917cb1f5-ea2e-45e2-9a9b-efcc585ec376\" (UID: \"917cb1f5-ea2e-45e2-9a9b-efcc585ec376\") " Jul 16 12:43:32.229090 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:32.229001 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/917cb1f5-ea2e-45e2-9a9b-efcc585ec376-util\") pod \"917cb1f5-ea2e-45e2-9a9b-efcc585ec376\" (UID: \"917cb1f5-ea2e-45e2-9a9b-efcc585ec376\") " Jul 16 12:43:32.229650 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:32.229616 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/917cb1f5-ea2e-45e2-9a9b-efcc585ec376-bundle" (OuterVolumeSpecName: "bundle") pod "917cb1f5-ea2e-45e2-9a9b-efcc585ec376" (UID: "917cb1f5-ea2e-45e2-9a9b-efcc585ec376"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:43:32.231134 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:32.231110 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/917cb1f5-ea2e-45e2-9a9b-efcc585ec376-kube-api-access-d4rmd" (OuterVolumeSpecName: "kube-api-access-d4rmd") pod "917cb1f5-ea2e-45e2-9a9b-efcc585ec376" (UID: "917cb1f5-ea2e-45e2-9a9b-efcc585ec376"). InnerVolumeSpecName "kube-api-access-d4rmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:43:32.234437 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:32.234416 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/917cb1f5-ea2e-45e2-9a9b-efcc585ec376-util" (OuterVolumeSpecName: "util") pod "917cb1f5-ea2e-45e2-9a9b-efcc585ec376" (UID: "917cb1f5-ea2e-45e2-9a9b-efcc585ec376"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:43:32.329983 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:32.329902 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/917cb1f5-ea2e-45e2-9a9b-efcc585ec376-util\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:43:32.329983 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:32.329933 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4rmd\" (UniqueName: \"kubernetes.io/projected/917cb1f5-ea2e-45e2-9a9b-efcc585ec376-kube-api-access-d4rmd\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:43:32.329983 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:32.329949 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/917cb1f5-ea2e-45e2-9a9b-efcc585ec376-bundle\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:43:33.042108 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:33.042075 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrvnz5" event={"ID":"917cb1f5-ea2e-45e2-9a9b-efcc585ec376","Type":"ContainerDied","Data":"5f7ee2e0357ee3d5c57132563b829782b6dd7accefbb1aab639cf4a956a3c77b"} Jul 16 12:43:33.042108 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:33.042091 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrvnz5" Jul 16 12:43:33.042307 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:33.042119 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f7ee2e0357ee3d5c57132563b829782b6dd7accefbb1aab639cf4a956a3c77b" Jul 16 12:43:43.499579 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:43.499534 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:43.597334 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:43.597293 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:44.109154 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.109121 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:44.330896 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.330846 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:44.403673 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.403641 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-65f8bdf75c-nsqgx"] Jul 16 12:43:44.403906 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.403895 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="917cb1f5-ea2e-45e2-9a9b-efcc585ec376" containerName="util" Jul 16 12:43:44.403947 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.403908 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="917cb1f5-ea2e-45e2-9a9b-efcc585ec376" containerName="util" Jul 16 12:43:44.403947 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.403921 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="917cb1f5-ea2e-45e2-9a9b-efcc585ec376" containerName="pull" Jul 16 12:43:44.403947 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.403926 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="917cb1f5-ea2e-45e2-9a9b-efcc585ec376" containerName="pull" Jul 16 12:43:44.403947 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.403932 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="917cb1f5-ea2e-45e2-9a9b-efcc585ec376" containerName="extract" Jul 16 12:43:44.403947 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.403937 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="917cb1f5-ea2e-45e2-9a9b-efcc585ec376" containerName="extract" Jul 16 12:43:44.404090 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.403982 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="917cb1f5-ea2e-45e2-9a9b-efcc585ec376" containerName="extract" Jul 16 12:43:44.411033 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.411009 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-65f8bdf75c-nsqgx" Jul 16 12:43:44.414359 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.414337 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 16 12:43:44.414668 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.414651 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 16 12:43:44.425273 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.425253 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 16 12:43:44.425367 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.425305 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 16 12:43:44.426441 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.426422 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-lqknb\"" Jul 16 12:43:44.445241 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.445219 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-65f8bdf75c-nsqgx"] Jul 16 12:43:44.446445 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.446423 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:44.518072 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.518041 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wxqd\" (UniqueName: \"kubernetes.io/projected/bacb49f8-640b-4c93-9958-a8a0eb492f11-kube-api-access-2wxqd\") pod \"opendatahub-operator-controller-manager-65f8bdf75c-nsqgx\" (UID: \"bacb49f8-640b-4c93-9958-a8a0eb492f11\") " pod="opendatahub/opendatahub-operator-controller-manager-65f8bdf75c-nsqgx" Jul 16 12:43:44.518428 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.518097 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bacb49f8-640b-4c93-9958-a8a0eb492f11-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65f8bdf75c-nsqgx\" (UID: \"bacb49f8-640b-4c93-9958-a8a0eb492f11\") " pod="opendatahub/opendatahub-operator-controller-manager-65f8bdf75c-nsqgx" Jul 16 12:43:44.518428 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.518117 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bacb49f8-640b-4c93-9958-a8a0eb492f11-tmp\") pod \"opendatahub-operator-controller-manager-65f8bdf75c-nsqgx\" (UID: \"bacb49f8-640b-4c93-9958-a8a0eb492f11\") " pod="opendatahub/opendatahub-operator-controller-manager-65f8bdf75c-nsqgx" Jul 16 12:43:44.518428 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.518140 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bacb49f8-640b-4c93-9958-a8a0eb492f11-webhook-cert\") pod \"opendatahub-operator-controller-manager-65f8bdf75c-nsqgx\" (UID: \"bacb49f8-640b-4c93-9958-a8a0eb492f11\") " pod="opendatahub/opendatahub-operator-controller-manager-65f8bdf75c-nsqgx" Jul 16 12:43:44.518428 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.518157 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bacb49f8-640b-4c93-9958-a8a0eb492f11-manifests\") pod \"opendatahub-operator-controller-manager-65f8bdf75c-nsqgx\" (UID: \"bacb49f8-640b-4c93-9958-a8a0eb492f11\") " pod="opendatahub/opendatahub-operator-controller-manager-65f8bdf75c-nsqgx" Jul 16 12:43:44.580252 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.580219 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:44.619474 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.619437 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bacb49f8-640b-4c93-9958-a8a0eb492f11-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65f8bdf75c-nsqgx\" (UID: \"bacb49f8-640b-4c93-9958-a8a0eb492f11\") " pod="opendatahub/opendatahub-operator-controller-manager-65f8bdf75c-nsqgx" Jul 16 12:43:44.619607 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.619479 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bacb49f8-640b-4c93-9958-a8a0eb492f11-tmp\") pod \"opendatahub-operator-controller-manager-65f8bdf75c-nsqgx\" (UID: \"bacb49f8-640b-4c93-9958-a8a0eb492f11\") " pod="opendatahub/opendatahub-operator-controller-manager-65f8bdf75c-nsqgx" Jul 16 12:43:44.619653 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.619597 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bacb49f8-640b-4c93-9958-a8a0eb492f11-webhook-cert\") pod \"opendatahub-operator-controller-manager-65f8bdf75c-nsqgx\" (UID: \"bacb49f8-640b-4c93-9958-a8a0eb492f11\") " pod="opendatahub/opendatahub-operator-controller-manager-65f8bdf75c-nsqgx" Jul 16 12:43:44.619653 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.619630 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bacb49f8-640b-4c93-9958-a8a0eb492f11-manifests\") pod \"opendatahub-operator-controller-manager-65f8bdf75c-nsqgx\" (UID: \"bacb49f8-640b-4c93-9958-a8a0eb492f11\") " pod="opendatahub/opendatahub-operator-controller-manager-65f8bdf75c-nsqgx" Jul 16 12:43:44.619724 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.619685 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wxqd\" (UniqueName: \"kubernetes.io/projected/bacb49f8-640b-4c93-9958-a8a0eb492f11-kube-api-access-2wxqd\") pod \"opendatahub-operator-controller-manager-65f8bdf75c-nsqgx\" (UID: \"bacb49f8-640b-4c93-9958-a8a0eb492f11\") " pod="opendatahub/opendatahub-operator-controller-manager-65f8bdf75c-nsqgx" Jul 16 12:43:44.620022 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.619992 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bacb49f8-640b-4c93-9958-a8a0eb492f11-manifests\") pod \"opendatahub-operator-controller-manager-65f8bdf75c-nsqgx\" (UID: \"bacb49f8-640b-4c93-9958-a8a0eb492f11\") " pod="opendatahub/opendatahub-operator-controller-manager-65f8bdf75c-nsqgx" Jul 16 12:43:44.621818 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.621796 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bacb49f8-640b-4c93-9958-a8a0eb492f11-tmp\") pod \"opendatahub-operator-controller-manager-65f8bdf75c-nsqgx\" (UID: \"bacb49f8-640b-4c93-9958-a8a0eb492f11\") " pod="opendatahub/opendatahub-operator-controller-manager-65f8bdf75c-nsqgx" Jul 16 12:43:44.622121 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.622100 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bacb49f8-640b-4c93-9958-a8a0eb492f11-webhook-cert\") pod \"opendatahub-operator-controller-manager-65f8bdf75c-nsqgx\" (UID: \"bacb49f8-640b-4c93-9958-a8a0eb492f11\") " pod="opendatahub/opendatahub-operator-controller-manager-65f8bdf75c-nsqgx" Jul 16 12:43:44.622195 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.622169 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bacb49f8-640b-4c93-9958-a8a0eb492f11-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65f8bdf75c-nsqgx\" (UID: \"bacb49f8-640b-4c93-9958-a8a0eb492f11\") " pod="opendatahub/opendatahub-operator-controller-manager-65f8bdf75c-nsqgx" Jul 16 12:43:44.634682 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.634656 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wxqd\" (UniqueName: \"kubernetes.io/projected/bacb49f8-640b-4c93-9958-a8a0eb492f11-kube-api-access-2wxqd\") pod \"opendatahub-operator-controller-manager-65f8bdf75c-nsqgx\" (UID: \"bacb49f8-640b-4c93-9958-a8a0eb492f11\") " pod="opendatahub/opendatahub-operator-controller-manager-65f8bdf75c-nsqgx" Jul 16 12:43:44.682408 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.682330 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:44.720495 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.720472 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-65f8bdf75c-nsqgx" Jul 16 12:43:44.784349 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.784265 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:44.862849 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:44.862821 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-65f8bdf75c-nsqgx"] Jul 16 12:43:44.865733 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:43:44.865699 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbacb49f8_640b_4c93_9958_a8a0eb492f11.slice/crio-c824a154d2954a98d9a0dbd070dc4682d5a4e3bd6f829e1d68167484ae751ad0 WatchSource:0}: Error finding container c824a154d2954a98d9a0dbd070dc4682d5a4e3bd6f829e1d68167484ae751ad0: Status 404 returned error can't find the container with id c824a154d2954a98d9a0dbd070dc4682d5a4e3bd6f829e1d68167484ae751ad0 Jul 16 12:43:45.051051 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:45.050976 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:45.091688 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:45.091662 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65f8bdf75c-nsqgx" event={"ID":"bacb49f8-640b-4c93-9958-a8a0eb492f11","Type":"ContainerStarted","Data":"c824a154d2954a98d9a0dbd070dc4682d5a4e3bd6f829e1d68167484ae751ad0"} Jul 16 12:43:45.150470 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:45.150437 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:49.109578 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:49.109537 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65f8bdf75c-nsqgx" event={"ID":"bacb49f8-640b-4c93-9958-a8a0eb492f11","Type":"ContainerStarted","Data":"49fec194238a3df51dd36e8a69934d17d0261bc7060fe48c6e427a3e63efb3d2"} Jul 16 12:43:49.953605 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:49.953568 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:50.051409 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:50.051360 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:43:50.114595 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:50.114564 2582 generic.go:356] "Generic (PLEG): container finished" podID="bacb49f8-640b-4c93-9958-a8a0eb492f11" containerID="49fec194238a3df51dd36e8a69934d17d0261bc7060fe48c6e427a3e63efb3d2" exitCode=0 Jul 16 12:43:50.115032 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:50.114615 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65f8bdf75c-nsqgx" event={"ID":"bacb49f8-640b-4c93-9958-a8a0eb492f11","Type":"ContainerDied","Data":"49fec194238a3df51dd36e8a69934d17d0261bc7060fe48c6e427a3e63efb3d2"} Jul 16 12:43:51.121630 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:51.121546 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65f8bdf75c-nsqgx" event={"ID":"bacb49f8-640b-4c93-9958-a8a0eb492f11","Type":"ContainerStarted","Data":"ac460e332f67a40d95956940c1e586b6da55cb5eb9e52e9e97fca8e9fd0dc0b1"} Jul 16 12:43:51.121989 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:51.121693 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-65f8bdf75c-nsqgx" Jul 16 12:43:51.162844 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:43:51.162791 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-65f8bdf75c-nsqgx" podStartSLOduration=2.261827009 podStartE2EDuration="7.16277482s" totalImagesPullingTime="3.844001484s" totalInitContainerRuntime="1.056946327s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:43:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:43:51.162339046 +0000 UTC m=+289.640084663" watchObservedRunningTime="2026-07-16 12:43:51.16277482 +0000 UTC m=+289.640520470" Jul 16 12:44:01.871846 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:01.871818 2582 kubelet.go:1648] "Image garbage collection succeeded" Jul 16 12:44:02.127939 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:02.127911 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-65f8bdf75c-nsqgx" Jul 16 12:44:03.986743 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:03.986709 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:44:04.092316 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:04.092280 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:44:04.688524 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:04.688488 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:44:04.834565 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:04.834531 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6678559b48-mz44r"] Jul 16 12:44:04.839451 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:04.839423 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6678559b48-mz44r" Jul 16 12:44:04.842473 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:04.842449 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xmp2w\"" Jul 16 12:44:04.842607 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:04.842457 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 16 12:44:04.843111 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:04.843094 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 16 12:44:04.843232 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:04.843214 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 16 12:44:04.843295 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:04.843231 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 16 12:44:04.852071 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:04.852047 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6678559b48-mz44r"] Jul 16 12:44:04.869727 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:04.869702 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-f6mzx"] Jul 16 12:44:04.872776 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:04.872760 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-f6mzx" Jul 16 12:44:04.874954 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:04.874937 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 16 12:44:04.875090 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:04.875074 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-7h2ss\"" Jul 16 12:44:04.892435 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:04.892416 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-f6mzx"] Jul 16 12:44:04.923578 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:04.923554 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:44:04.981503 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:04.981433 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-477sr\" (UniqueName: \"kubernetes.io/projected/e6b692ad-70c0-42bd-ae15-4caddafc8a79-kube-api-access-477sr\") pod \"kube-auth-proxy-6678559b48-mz44r\" (UID: \"e6b692ad-70c0-42bd-ae15-4caddafc8a79\") " pod="openshift-ingress/kube-auth-proxy-6678559b48-mz44r" Jul 16 12:44:04.981503 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:04.981479 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6b692ad-70c0-42bd-ae15-4caddafc8a79-tmp\") pod \"kube-auth-proxy-6678559b48-mz44r\" (UID: \"e6b692ad-70c0-42bd-ae15-4caddafc8a79\") " pod="openshift-ingress/kube-auth-proxy-6678559b48-mz44r" Jul 16 12:44:04.981653 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:04.981513 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6b692ad-70c0-42bd-ae15-4caddafc8a79-tls-certs\") pod \"kube-auth-proxy-6678559b48-mz44r\" (UID: \"e6b692ad-70c0-42bd-ae15-4caddafc8a79\") " pod="openshift-ingress/kube-auth-proxy-6678559b48-mz44r" Jul 16 12:44:04.981653 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:04.981530 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1432236c-bfc1-4616-b394-08d76a14e655-cert\") pod \"odh-model-controller-7998c67dc4-f6mzx\" (UID: \"1432236c-bfc1-4616-b394-08d76a14e655\") " pod="opendatahub/odh-model-controller-7998c67dc4-f6mzx" Jul 16 12:44:04.981653 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:04.981550 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2sn5\" (UniqueName: \"kubernetes.io/projected/1432236c-bfc1-4616-b394-08d76a14e655-kube-api-access-m2sn5\") pod \"odh-model-controller-7998c67dc4-f6mzx\" (UID: \"1432236c-bfc1-4616-b394-08d76a14e655\") " pod="opendatahub/odh-model-controller-7998c67dc4-f6mzx" Jul 16 12:44:05.082483 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:05.082452 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6b692ad-70c0-42bd-ae15-4caddafc8a79-tmp\") pod \"kube-auth-proxy-6678559b48-mz44r\" (UID: \"e6b692ad-70c0-42bd-ae15-4caddafc8a79\") " pod="openshift-ingress/kube-auth-proxy-6678559b48-mz44r" Jul 16 12:44:05.082865 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:05.082500 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6b692ad-70c0-42bd-ae15-4caddafc8a79-tls-certs\") pod \"kube-auth-proxy-6678559b48-mz44r\" (UID: \"e6b692ad-70c0-42bd-ae15-4caddafc8a79\") " pod="openshift-ingress/kube-auth-proxy-6678559b48-mz44r" Jul 16 12:44:05.082865 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:05.082521 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1432236c-bfc1-4616-b394-08d76a14e655-cert\") pod \"odh-model-controller-7998c67dc4-f6mzx\" (UID: \"1432236c-bfc1-4616-b394-08d76a14e655\") " pod="opendatahub/odh-model-controller-7998c67dc4-f6mzx" Jul 16 12:44:05.082865 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:44:05.082606 2582 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 16 12:44:05.082865 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:05.082615 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2sn5\" (UniqueName: \"kubernetes.io/projected/1432236c-bfc1-4616-b394-08d76a14e655-kube-api-access-m2sn5\") pod \"odh-model-controller-7998c67dc4-f6mzx\" (UID: \"1432236c-bfc1-4616-b394-08d76a14e655\") " pod="opendatahub/odh-model-controller-7998c67dc4-f6mzx" Jul 16 12:44:05.082865 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:44:05.082677 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1432236c-bfc1-4616-b394-08d76a14e655-cert podName:1432236c-bfc1-4616-b394-08d76a14e655 nodeName:}" failed. No retries permitted until 2026-07-16 12:44:05.582653845 +0000 UTC m=+304.060399440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1432236c-bfc1-4616-b394-08d76a14e655-cert") pod "odh-model-controller-7998c67dc4-f6mzx" (UID: "1432236c-bfc1-4616-b394-08d76a14e655") : secret "odh-model-controller-webhook-cert" not found Jul 16 12:44:05.082865 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:05.082706 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-477sr\" (UniqueName: \"kubernetes.io/projected/e6b692ad-70c0-42bd-ae15-4caddafc8a79-kube-api-access-477sr\") pod \"kube-auth-proxy-6678559b48-mz44r\" (UID: \"e6b692ad-70c0-42bd-ae15-4caddafc8a79\") " pod="openshift-ingress/kube-auth-proxy-6678559b48-mz44r" Jul 16 12:44:05.084793 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:05.084772 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6b692ad-70c0-42bd-ae15-4caddafc8a79-tmp\") pod \"kube-auth-proxy-6678559b48-mz44r\" (UID: \"e6b692ad-70c0-42bd-ae15-4caddafc8a79\") " pod="openshift-ingress/kube-auth-proxy-6678559b48-mz44r" Jul 16 12:44:05.085003 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:05.084986 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6b692ad-70c0-42bd-ae15-4caddafc8a79-tls-certs\") pod \"kube-auth-proxy-6678559b48-mz44r\" (UID: \"e6b692ad-70c0-42bd-ae15-4caddafc8a79\") " pod="openshift-ingress/kube-auth-proxy-6678559b48-mz44r" Jul 16 12:44:05.094577 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:05.094551 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2sn5\" (UniqueName: \"kubernetes.io/projected/1432236c-bfc1-4616-b394-08d76a14e655-kube-api-access-m2sn5\") pod \"odh-model-controller-7998c67dc4-f6mzx\" (UID: \"1432236c-bfc1-4616-b394-08d76a14e655\") " pod="opendatahub/odh-model-controller-7998c67dc4-f6mzx" Jul 16 12:44:05.094687 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:05.094670 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-477sr\" (UniqueName: \"kubernetes.io/projected/e6b692ad-70c0-42bd-ae15-4caddafc8a79-kube-api-access-477sr\") pod \"kube-auth-proxy-6678559b48-mz44r\" (UID: \"e6b692ad-70c0-42bd-ae15-4caddafc8a79\") " pod="openshift-ingress/kube-auth-proxy-6678559b48-mz44r" Jul 16 12:44:05.150354 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:05.150322 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6678559b48-mz44r" Jul 16 12:44:05.161420 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:05.161377 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:44:05.257993 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:05.257922 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:44:05.283855 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:05.283834 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6678559b48-mz44r"] Jul 16 12:44:05.285879 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:44:05.285854 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6b692ad_70c0_42bd_ae15_4caddafc8a79.slice/crio-2c66d0f4b6840c947960e2e46ebf9abe4489388aba85487edf64b9cceb154a66 WatchSource:0}: Error finding container 2c66d0f4b6840c947960e2e46ebf9abe4489388aba85487edf64b9cceb154a66: Status 404 returned error can't find the container with id 2c66d0f4b6840c947960e2e46ebf9abe4489388aba85487edf64b9cceb154a66 Jul 16 12:44:05.287780 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:05.287764 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 12:44:05.586502 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:05.586413 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1432236c-bfc1-4616-b394-08d76a14e655-cert\") pod \"odh-model-controller-7998c67dc4-f6mzx\" (UID: \"1432236c-bfc1-4616-b394-08d76a14e655\") " pod="opendatahub/odh-model-controller-7998c67dc4-f6mzx" Jul 16 12:44:05.586653 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:44:05.586540 2582 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 16 12:44:05.586653 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:44:05.586599 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1432236c-bfc1-4616-b394-08d76a14e655-cert podName:1432236c-bfc1-4616-b394-08d76a14e655 nodeName:}" failed. No retries permitted until 2026-07-16 12:44:06.586580776 +0000 UTC m=+305.064326371 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1432236c-bfc1-4616-b394-08d76a14e655-cert") pod "odh-model-controller-7998c67dc4-f6mzx" (UID: "1432236c-bfc1-4616-b394-08d76a14e655") : secret "odh-model-controller-webhook-cert" not found Jul 16 12:44:05.629350 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:05.629321 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:44:05.945899 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:05.945868 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:44:06.049449 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:06.049383 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:44:06.182975 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:06.182934 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6678559b48-mz44r" event={"ID":"e6b692ad-70c0-42bd-ae15-4caddafc8a79","Type":"ContainerStarted","Data":"2c66d0f4b6840c947960e2e46ebf9abe4489388aba85487edf64b9cceb154a66"} Jul 16 12:44:06.197152 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:06.197081 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:44:06.281911 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:06.281881 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:44:06.595657 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:06.595567 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1432236c-bfc1-4616-b394-08d76a14e655-cert\") pod \"odh-model-controller-7998c67dc4-f6mzx\" (UID: \"1432236c-bfc1-4616-b394-08d76a14e655\") " pod="opendatahub/odh-model-controller-7998c67dc4-f6mzx" Jul 16 12:44:06.598145 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:06.598122 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1432236c-bfc1-4616-b394-08d76a14e655-cert\") pod \"odh-model-controller-7998c67dc4-f6mzx\" (UID: \"1432236c-bfc1-4616-b394-08d76a14e655\") " pod="opendatahub/odh-model-controller-7998c67dc4-f6mzx" Jul 16 12:44:06.681677 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:06.681616 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-f6mzx" Jul 16 12:44:06.821801 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:06.821509 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-f6mzx"] Jul 16 12:44:06.824448 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:44:06.824415 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1432236c_bfc1_4616_b394_08d76a14e655.slice/crio-858d8626c29468d22edf450e172fca5d90a8815d02ea51f9b969ad9af7dcab6c WatchSource:0}: Error finding container 858d8626c29468d22edf450e172fca5d90a8815d02ea51f9b969ad9af7dcab6c: Status 404 returned error can't find the container with id 858d8626c29468d22edf450e172fca5d90a8815d02ea51f9b969ad9af7dcab6c Jul 16 12:44:07.187662 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:07.187624 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-f6mzx" event={"ID":"1432236c-bfc1-4616-b394-08d76a14e655","Type":"ContainerStarted","Data":"858d8626c29468d22edf450e172fca5d90a8815d02ea51f9b969ad9af7dcab6c"} Jul 16 12:44:07.457338 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:07.457250 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:44:09.198815 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:09.198740 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6678559b48-mz44r" event={"ID":"e6b692ad-70c0-42bd-ae15-4caddafc8a79","Type":"ContainerStarted","Data":"9f3f3999e264d57d5610fda654e4497442b309624352736a88f9c9d64f484587"} Jul 16 12:44:09.217764 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:09.217706 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6678559b48-mz44r" podStartSLOduration=2.284796351 podStartE2EDuration="5.217683719s" totalImagesPullingTime="2.932887368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:44:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:44:09.215653185 +0000 UTC m=+307.693398812" watchObservedRunningTime="2026-07-16 12:44:09.217683719 +0000 UTC m=+307.695429336" Jul 16 12:44:10.205028 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:10.204990 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-f6mzx" event={"ID":"1432236c-bfc1-4616-b394-08d76a14e655","Type":"ContainerStarted","Data":"4540876e03f0d8165ed8e6cdc785c9aadba4470ba249ccfea71cd411d61a0c5b"} Jul 16 12:44:10.205463 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:10.205084 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-f6mzx" Jul 16 12:44:10.224242 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:10.224191 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-f6mzx" podStartSLOduration=3.54579832 podStartE2EDuration="6.224176422s" totalImagesPullingTime="2.678378102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:44:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:44:10.222888996 +0000 UTC m=+308.700634633" watchObservedRunningTime="2026-07-16 12:44:10.224176422 +0000 UTC m=+308.701922038" Jul 16 12:44:10.592997 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:10.592921 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:44:11.243707 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.243674 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn"] Jul 16 12:44:11.247184 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.247163 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" Jul 16 12:44:11.250532 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.250511 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-r9ht7\"" Jul 16 12:44:11.250646 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.250542 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 16 12:44:11.266660 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.266643 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn"] Jul 16 12:44:11.279831 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.279809 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:44:11.340744 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.340715 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dca87b5-5935-411a-bd0c-ac7335a1f9e8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nglsn\" (UID: \"1dca87b5-5935-411a-bd0c-ac7335a1f9e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" Jul 16 12:44:11.340857 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.340749 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dca87b5-5935-411a-bd0c-ac7335a1f9e8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nglsn\" (UID: \"1dca87b5-5935-411a-bd0c-ac7335a1f9e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" Jul 16 12:44:11.340857 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.340776 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dca87b5-5935-411a-bd0c-ac7335a1f9e8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nglsn\" (UID: \"1dca87b5-5935-411a-bd0c-ac7335a1f9e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" Jul 16 12:44:11.340857 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.340829 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dca87b5-5935-411a-bd0c-ac7335a1f9e8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nglsn\" (UID: \"1dca87b5-5935-411a-bd0c-ac7335a1f9e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" Jul 16 12:44:11.341013 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.340874 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv8wz\" (UniqueName: \"kubernetes.io/projected/1dca87b5-5935-411a-bd0c-ac7335a1f9e8-kube-api-access-lv8wz\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nglsn\" (UID: \"1dca87b5-5935-411a-bd0c-ac7335a1f9e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" Jul 16 12:44:11.341013 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.340989 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dca87b5-5935-411a-bd0c-ac7335a1f9e8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nglsn\" (UID: \"1dca87b5-5935-411a-bd0c-ac7335a1f9e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" Jul 16 12:44:11.341101 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.341032 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dca87b5-5935-411a-bd0c-ac7335a1f9e8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nglsn\" (UID: \"1dca87b5-5935-411a-bd0c-ac7335a1f9e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" Jul 16 12:44:11.341101 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.341059 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dca87b5-5935-411a-bd0c-ac7335a1f9e8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nglsn\" (UID: \"1dca87b5-5935-411a-bd0c-ac7335a1f9e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" Jul 16 12:44:11.341193 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.341109 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dca87b5-5935-411a-bd0c-ac7335a1f9e8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nglsn\" (UID: \"1dca87b5-5935-411a-bd0c-ac7335a1f9e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" Jul 16 12:44:11.442083 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.442044 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv8wz\" (UniqueName: \"kubernetes.io/projected/1dca87b5-5935-411a-bd0c-ac7335a1f9e8-kube-api-access-lv8wz\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nglsn\" (UID: \"1dca87b5-5935-411a-bd0c-ac7335a1f9e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" Jul 16 12:44:11.442279 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.442115 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dca87b5-5935-411a-bd0c-ac7335a1f9e8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nglsn\" (UID: \"1dca87b5-5935-411a-bd0c-ac7335a1f9e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" Jul 16 12:44:11.442279 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.442146 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dca87b5-5935-411a-bd0c-ac7335a1f9e8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nglsn\" (UID: \"1dca87b5-5935-411a-bd0c-ac7335a1f9e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" Jul 16 12:44:11.442279 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.442168 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dca87b5-5935-411a-bd0c-ac7335a1f9e8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nglsn\" (UID: \"1dca87b5-5935-411a-bd0c-ac7335a1f9e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" Jul 16 12:44:11.442279 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.442195 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dca87b5-5935-411a-bd0c-ac7335a1f9e8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nglsn\" (UID: \"1dca87b5-5935-411a-bd0c-ac7335a1f9e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" Jul 16 12:44:11.442513 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.442363 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dca87b5-5935-411a-bd0c-ac7335a1f9e8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nglsn\" (UID: \"1dca87b5-5935-411a-bd0c-ac7335a1f9e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" Jul 16 12:44:11.442513 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.442445 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dca87b5-5935-411a-bd0c-ac7335a1f9e8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nglsn\" (UID: \"1dca87b5-5935-411a-bd0c-ac7335a1f9e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" Jul 16 12:44:11.442513 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.442482 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dca87b5-5935-411a-bd0c-ac7335a1f9e8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nglsn\" (UID: \"1dca87b5-5935-411a-bd0c-ac7335a1f9e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" Jul 16 12:44:11.442656 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.442512 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dca87b5-5935-411a-bd0c-ac7335a1f9e8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nglsn\" (UID: \"1dca87b5-5935-411a-bd0c-ac7335a1f9e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" Jul 16 12:44:11.442656 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.442643 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dca87b5-5935-411a-bd0c-ac7335a1f9e8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nglsn\" (UID: \"1dca87b5-5935-411a-bd0c-ac7335a1f9e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" Jul 16 12:44:11.442754 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.442712 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dca87b5-5935-411a-bd0c-ac7335a1f9e8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nglsn\" (UID: \"1dca87b5-5935-411a-bd0c-ac7335a1f9e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" Jul 16 12:44:11.442847 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.442821 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dca87b5-5935-411a-bd0c-ac7335a1f9e8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nglsn\" (UID: \"1dca87b5-5935-411a-bd0c-ac7335a1f9e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" Jul 16 12:44:11.442954 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.442933 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dca87b5-5935-411a-bd0c-ac7335a1f9e8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nglsn\" (UID: \"1dca87b5-5935-411a-bd0c-ac7335a1f9e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" Jul 16 12:44:11.443005 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.442955 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dca87b5-5935-411a-bd0c-ac7335a1f9e8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nglsn\" (UID: \"1dca87b5-5935-411a-bd0c-ac7335a1f9e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" Jul 16 12:44:11.444763 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.444741 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dca87b5-5935-411a-bd0c-ac7335a1f9e8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nglsn\" (UID: \"1dca87b5-5935-411a-bd0c-ac7335a1f9e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" Jul 16 12:44:11.445055 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.445039 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dca87b5-5935-411a-bd0c-ac7335a1f9e8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nglsn\" (UID: \"1dca87b5-5935-411a-bd0c-ac7335a1f9e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" Jul 16 12:44:11.456903 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.456882 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dca87b5-5935-411a-bd0c-ac7335a1f9e8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nglsn\" (UID: \"1dca87b5-5935-411a-bd0c-ac7335a1f9e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" Jul 16 12:44:11.456903 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.456891 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv8wz\" (UniqueName: \"kubernetes.io/projected/1dca87b5-5935-411a-bd0c-ac7335a1f9e8-kube-api-access-lv8wz\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nglsn\" (UID: \"1dca87b5-5935-411a-bd0c-ac7335a1f9e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" Jul 16 12:44:11.540263 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.540184 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:44:11.557345 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.557314 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" Jul 16 12:44:11.692552 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.692519 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn"] Jul 16 12:44:11.695544 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:44:11.695519 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dca87b5_5935_411a_bd0c_ac7335a1f9e8.slice/crio-0ab69919846051791e1a6553d572a6e05dfd6a9bc04989b9a65b6657f2977bf7 WatchSource:0}: Error finding container 0ab69919846051791e1a6553d572a6e05dfd6a9bc04989b9a65b6657f2977bf7: Status 404 returned error can't find the container with id 0ab69919846051791e1a6553d572a6e05dfd6a9bc04989b9a65b6657f2977bf7 Jul 16 12:44:11.789735 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:11.789704 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:44:12.214924 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:12.214890 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" event={"ID":"1dca87b5-5935-411a-bd0c-ac7335a1f9e8","Type":"ContainerStarted","Data":"0ab69919846051791e1a6553d572a6e05dfd6a9bc04989b9a65b6657f2977bf7"} Jul 16 12:44:13.108506 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:13.108400 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-596f9f8f6b-4kxc8"] Jul 16 12:44:13.112210 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:13.112188 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-596f9f8f6b-4kxc8" Jul 16 12:44:13.123150 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:13.116854 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 16 12:44:13.123150 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:13.117170 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-wmf9c\"" Jul 16 12:44:13.123150 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:13.117340 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 16 12:44:13.123150 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:13.117560 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 16 12:44:13.136970 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:13.136939 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-596f9f8f6b-4kxc8"] Jul 16 12:44:13.258283 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:13.258244 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b76ee530-c81b-43cb-a571-b9a8dcaf1923-cert\") pod \"lws-controller-manager-596f9f8f6b-4kxc8\" (UID: \"b76ee530-c81b-43cb-a571-b9a8dcaf1923\") " pod="openshift-lws-operator/lws-controller-manager-596f9f8f6b-4kxc8" Jul 16 12:44:13.258283 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:13.258292 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b76ee530-c81b-43cb-a571-b9a8dcaf1923-metrics-cert\") pod \"lws-controller-manager-596f9f8f6b-4kxc8\" (UID: \"b76ee530-c81b-43cb-a571-b9a8dcaf1923\") " pod="openshift-lws-operator/lws-controller-manager-596f9f8f6b-4kxc8" Jul 16 12:44:13.258548 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:13.258355 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxx4p\" (UniqueName: \"kubernetes.io/projected/b76ee530-c81b-43cb-a571-b9a8dcaf1923-kube-api-access-gxx4p\") pod \"lws-controller-manager-596f9f8f6b-4kxc8\" (UID: \"b76ee530-c81b-43cb-a571-b9a8dcaf1923\") " pod="openshift-lws-operator/lws-controller-manager-596f9f8f6b-4kxc8" Jul 16 12:44:13.258548 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:13.258510 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b76ee530-c81b-43cb-a571-b9a8dcaf1923-manager-config\") pod \"lws-controller-manager-596f9f8f6b-4kxc8\" (UID: \"b76ee530-c81b-43cb-a571-b9a8dcaf1923\") " pod="openshift-lws-operator/lws-controller-manager-596f9f8f6b-4kxc8" Jul 16 12:44:13.359466 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:13.359366 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b76ee530-c81b-43cb-a571-b9a8dcaf1923-manager-config\") pod \"lws-controller-manager-596f9f8f6b-4kxc8\" (UID: \"b76ee530-c81b-43cb-a571-b9a8dcaf1923\") " pod="openshift-lws-operator/lws-controller-manager-596f9f8f6b-4kxc8" Jul 16 12:44:13.359466 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:13.359462 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b76ee530-c81b-43cb-a571-b9a8dcaf1923-cert\") pod \"lws-controller-manager-596f9f8f6b-4kxc8\" (UID: \"b76ee530-c81b-43cb-a571-b9a8dcaf1923\") " pod="openshift-lws-operator/lws-controller-manager-596f9f8f6b-4kxc8" Jul 16 12:44:13.359695 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:13.359490 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b76ee530-c81b-43cb-a571-b9a8dcaf1923-metrics-cert\") pod \"lws-controller-manager-596f9f8f6b-4kxc8\" (UID: \"b76ee530-c81b-43cb-a571-b9a8dcaf1923\") " pod="openshift-lws-operator/lws-controller-manager-596f9f8f6b-4kxc8" Jul 16 12:44:13.359695 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:13.359521 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxx4p\" (UniqueName: \"kubernetes.io/projected/b76ee530-c81b-43cb-a571-b9a8dcaf1923-kube-api-access-gxx4p\") pod \"lws-controller-manager-596f9f8f6b-4kxc8\" (UID: \"b76ee530-c81b-43cb-a571-b9a8dcaf1923\") " pod="openshift-lws-operator/lws-controller-manager-596f9f8f6b-4kxc8" Jul 16 12:44:13.360212 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:13.360126 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b76ee530-c81b-43cb-a571-b9a8dcaf1923-manager-config\") pod \"lws-controller-manager-596f9f8f6b-4kxc8\" (UID: \"b76ee530-c81b-43cb-a571-b9a8dcaf1923\") " pod="openshift-lws-operator/lws-controller-manager-596f9f8f6b-4kxc8" Jul 16 12:44:13.362267 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:13.362242 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b76ee530-c81b-43cb-a571-b9a8dcaf1923-cert\") pod \"lws-controller-manager-596f9f8f6b-4kxc8\" (UID: \"b76ee530-c81b-43cb-a571-b9a8dcaf1923\") " pod="openshift-lws-operator/lws-controller-manager-596f9f8f6b-4kxc8" Jul 16 12:44:13.362500 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:13.362474 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b76ee530-c81b-43cb-a571-b9a8dcaf1923-metrics-cert\") pod \"lws-controller-manager-596f9f8f6b-4kxc8\" (UID: \"b76ee530-c81b-43cb-a571-b9a8dcaf1923\") " pod="openshift-lws-operator/lws-controller-manager-596f9f8f6b-4kxc8" Jul 16 12:44:13.370725 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:13.370699 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxx4p\" (UniqueName: \"kubernetes.io/projected/b76ee530-c81b-43cb-a571-b9a8dcaf1923-kube-api-access-gxx4p\") pod \"lws-controller-manager-596f9f8f6b-4kxc8\" (UID: \"b76ee530-c81b-43cb-a571-b9a8dcaf1923\") " pod="openshift-lws-operator/lws-controller-manager-596f9f8f6b-4kxc8" Jul 16 12:44:13.432098 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:13.432059 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-596f9f8f6b-4kxc8" Jul 16 12:44:13.589340 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:13.589317 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-596f9f8f6b-4kxc8"] Jul 16 12:44:13.712659 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:13.712619 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:44:14.378800 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:14.378765 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:44:14.473496 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:14.473456 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:44:14.574640 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:44:14.574609 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb76ee530_c81b_43cb_a571_b9a8dcaf1923.slice/crio-7acc0492b30c874ad16d01213b59a8275ca99d15e89e6fb308b4527fd1f81bf5 WatchSource:0}: Error finding container 7acc0492b30c874ad16d01213b59a8275ca99d15e89e6fb308b4527fd1f81bf5: Status 404 returned error can't find the container with id 7acc0492b30c874ad16d01213b59a8275ca99d15e89e6fb308b4527fd1f81bf5 Jul 16 12:44:14.634756 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:14.634716 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 12:44:14.634840 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:14.634794 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 12:44:14.634840 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:14.634831 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 12:44:14.718884 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:14.718857 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:44:15.070370 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:15.070283 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:44:15.232279 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:15.232242 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-596f9f8f6b-4kxc8" event={"ID":"b76ee530-c81b-43cb-a571-b9a8dcaf1923","Type":"ContainerStarted","Data":"7acc0492b30c874ad16d01213b59a8275ca99d15e89e6fb308b4527fd1f81bf5"} Jul 16 12:44:15.233670 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:15.233644 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" event={"ID":"1dca87b5-5935-411a-bd0c-ac7335a1f9e8","Type":"ContainerStarted","Data":"5359f95ee43337a7364d53fa5772128b280420497ce6a0bd13027955b0ec4686"} Jul 16 12:44:15.260249 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:15.260203 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" podStartSLOduration=1.323273846 podStartE2EDuration="4.26018896s" totalImagesPullingTime="2.936915114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:44:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:44:15.258872999 +0000 UTC m=+313.736618614" watchObservedRunningTime="2026-07-16 12:44:15.26018896 +0000 UTC m=+313.737934645" Jul 16 12:44:15.557879 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:15.557848 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" Jul 16 12:44:15.562573 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:15.562548 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" Jul 16 12:44:15.708347 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:15.708312 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:44:15.795767 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:15.795718 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:44:15.898485 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:15.898447 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:44:16.243796 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:16.241425 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" Jul 16 12:44:16.243796 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:16.242737 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nglsn" Jul 16 12:44:17.245713 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:17.245674 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-596f9f8f6b-4kxc8" event={"ID":"b76ee530-c81b-43cb-a571-b9a8dcaf1923","Type":"ContainerStarted","Data":"8a66e9dbde81440e650f4805aadd2c93cf2082e6bc866a1e16ddce0f5986219e"} Jul 16 12:44:17.246124 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:17.245865 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-596f9f8f6b-4kxc8" Jul 16 12:44:17.276936 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:17.276891 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-596f9f8f6b-4kxc8" podStartSLOduration=2.332262747 podStartE2EDuration="4.276874925s" totalImagesPullingTime="1.944612178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:44:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:44:17.275615534 +0000 UTC m=+315.753361161" watchObservedRunningTime="2026-07-16 12:44:17.276874925 +0000 UTC m=+315.754620540" Jul 16 12:44:21.212457 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:21.212425 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-f6mzx" Jul 16 12:44:28.252355 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:44:28.252325 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-596f9f8f6b-4kxc8" Jul 16 12:45:05.490367 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:05.490278 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:45:07.347246 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:07.347207 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:45:21.404828 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:21.404791 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz"] Jul 16 12:45:21.408293 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:21.408275 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz" Jul 16 12:45:21.411036 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:21.411017 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 16 12:45:21.412002 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:21.411985 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-w5vrd\"" Jul 16 12:45:21.412079 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:21.412067 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 16 12:45:21.418709 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:21.418685 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz"] Jul 16 12:45:21.511475 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:21.511442 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fggtn\" (UniqueName: \"kubernetes.io/projected/aa451150-aecc-4259-baa2-b590d4bceed0-kube-api-access-fggtn\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz\" (UID: \"aa451150-aecc-4259-baa2-b590d4bceed0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz" Jul 16 12:45:21.511672 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:21.511503 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa451150-aecc-4259-baa2-b590d4bceed0-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz\" (UID: \"aa451150-aecc-4259-baa2-b590d4bceed0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz" Jul 16 12:45:21.511672 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:21.511522 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa451150-aecc-4259-baa2-b590d4bceed0-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz\" (UID: \"aa451150-aecc-4259-baa2-b590d4bceed0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz" Jul 16 12:45:21.612819 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:21.612776 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa451150-aecc-4259-baa2-b590d4bceed0-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz\" (UID: \"aa451150-aecc-4259-baa2-b590d4bceed0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz" Jul 16 12:45:21.612819 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:21.612816 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa451150-aecc-4259-baa2-b590d4bceed0-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz\" (UID: \"aa451150-aecc-4259-baa2-b590d4bceed0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz" Jul 16 12:45:21.613027 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:21.612881 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fggtn\" (UniqueName: \"kubernetes.io/projected/aa451150-aecc-4259-baa2-b590d4bceed0-kube-api-access-fggtn\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz\" (UID: \"aa451150-aecc-4259-baa2-b590d4bceed0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz" Jul 16 12:45:21.613173 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:21.613153 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa451150-aecc-4259-baa2-b590d4bceed0-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz\" (UID: \"aa451150-aecc-4259-baa2-b590d4bceed0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz" Jul 16 12:45:21.613215 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:21.613176 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa451150-aecc-4259-baa2-b590d4bceed0-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz\" (UID: \"aa451150-aecc-4259-baa2-b590d4bceed0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz" Jul 16 12:45:21.620870 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:21.620839 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fggtn\" (UniqueName: \"kubernetes.io/projected/aa451150-aecc-4259-baa2-b590d4bceed0-kube-api-access-fggtn\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz\" (UID: \"aa451150-aecc-4259-baa2-b590d4bceed0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz" Jul 16 12:45:21.718025 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:21.717937 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz" Jul 16 12:45:21.852266 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:21.852230 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz"] Jul 16 12:45:21.856103 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:45:21.856070 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa451150_aecc_4259_baa2_b590d4bceed0.slice/crio-0b59fff134e24bebe65299feb55bdb587944be67d2a5572230fc18b7b1328a10 WatchSource:0}: Error finding container 0b59fff134e24bebe65299feb55bdb587944be67d2a5572230fc18b7b1328a10: Status 404 returned error can't find the container with id 0b59fff134e24bebe65299feb55bdb587944be67d2a5572230fc18b7b1328a10 Jul 16 12:45:22.009911 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.009830 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp"] Jul 16 12:45:22.013134 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.013113 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp" Jul 16 12:45:22.023067 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.023045 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp"] Jul 16 12:45:22.116670 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.116627 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tv96\" (UniqueName: \"kubernetes.io/projected/38c90c2c-d74a-4c6e-b6a7-f5d82958cc49-kube-api-access-7tv96\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp\" (UID: \"38c90c2c-d74a-4c6e-b6a7-f5d82958cc49\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp" Jul 16 12:45:22.116874 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.116749 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38c90c2c-d74a-4c6e-b6a7-f5d82958cc49-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp\" (UID: \"38c90c2c-d74a-4c6e-b6a7-f5d82958cc49\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp" Jul 16 12:45:22.116874 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.116853 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38c90c2c-d74a-4c6e-b6a7-f5d82958cc49-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp\" (UID: \"38c90c2c-d74a-4c6e-b6a7-f5d82958cc49\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp" Jul 16 12:45:22.217353 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.217314 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tv96\" (UniqueName: \"kubernetes.io/projected/38c90c2c-d74a-4c6e-b6a7-f5d82958cc49-kube-api-access-7tv96\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp\" (UID: \"38c90c2c-d74a-4c6e-b6a7-f5d82958cc49\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp" Jul 16 12:45:22.217540 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.217372 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38c90c2c-d74a-4c6e-b6a7-f5d82958cc49-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp\" (UID: \"38c90c2c-d74a-4c6e-b6a7-f5d82958cc49\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp" Jul 16 12:45:22.217540 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.217498 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38c90c2c-d74a-4c6e-b6a7-f5d82958cc49-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp\" (UID: \"38c90c2c-d74a-4c6e-b6a7-f5d82958cc49\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp" Jul 16 12:45:22.217811 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.217793 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38c90c2c-d74a-4c6e-b6a7-f5d82958cc49-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp\" (UID: \"38c90c2c-d74a-4c6e-b6a7-f5d82958cc49\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp" Jul 16 12:45:22.217844 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.217827 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38c90c2c-d74a-4c6e-b6a7-f5d82958cc49-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp\" (UID: \"38c90c2c-d74a-4c6e-b6a7-f5d82958cc49\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp" Jul 16 12:45:22.225945 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.225911 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tv96\" (UniqueName: \"kubernetes.io/projected/38c90c2c-d74a-4c6e-b6a7-f5d82958cc49-kube-api-access-7tv96\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp\" (UID: \"38c90c2c-d74a-4c6e-b6a7-f5d82958cc49\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp" Jul 16 12:45:22.321947 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.321852 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp" Jul 16 12:45:22.411154 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.411118 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss"] Jul 16 12:45:22.415284 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.415264 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss" Jul 16 12:45:22.422938 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.422910 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss"] Jul 16 12:45:22.452864 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.452836 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp"] Jul 16 12:45:22.455123 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:45:22.455097 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38c90c2c_d74a_4c6e_b6a7_f5d82958cc49.slice/crio-95e21bf037dd67fe3f579dea03a6e9079a86609b4885d98f4f336aec81864b31 WatchSource:0}: Error finding container 95e21bf037dd67fe3f579dea03a6e9079a86609b4885d98f4f336aec81864b31: Status 404 returned error can't find the container with id 95e21bf037dd67fe3f579dea03a6e9079a86609b4885d98f4f336aec81864b31 Jul 16 12:45:22.521206 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.521181 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f992ddba-708b-4d18-9d23-b92f474c885a-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss\" (UID: \"f992ddba-708b-4d18-9d23-b92f474c885a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss" Jul 16 12:45:22.521341 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.521240 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f992ddba-708b-4d18-9d23-b92f474c885a-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss\" (UID: \"f992ddba-708b-4d18-9d23-b92f474c885a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss" Jul 16 12:45:22.521438 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.521337 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj2c4\" (UniqueName: \"kubernetes.io/projected/f992ddba-708b-4d18-9d23-b92f474c885a-kube-api-access-wj2c4\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss\" (UID: \"f992ddba-708b-4d18-9d23-b92f474c885a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss" Jul 16 12:45:22.526746 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.526716 2582 generic.go:356] "Generic (PLEG): container finished" podID="aa451150-aecc-4259-baa2-b590d4bceed0" containerID="3c98e26132d6d3ca10b1b352ef73e3d529ceb2a8d9cc1d898dad8109e761381a" exitCode=0 Jul 16 12:45:22.526862 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.526800 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz" event={"ID":"aa451150-aecc-4259-baa2-b590d4bceed0","Type":"ContainerDied","Data":"3c98e26132d6d3ca10b1b352ef73e3d529ceb2a8d9cc1d898dad8109e761381a"} Jul 16 12:45:22.526862 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.526850 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz" event={"ID":"aa451150-aecc-4259-baa2-b590d4bceed0","Type":"ContainerStarted","Data":"0b59fff134e24bebe65299feb55bdb587944be67d2a5572230fc18b7b1328a10"} Jul 16 12:45:22.528603 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.528582 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp" event={"ID":"38c90c2c-d74a-4c6e-b6a7-f5d82958cc49","Type":"ContainerStarted","Data":"e905735b846c9f3a1e827725fcb456426b17042b75292a9b2170e626d93ce1aa"} Jul 16 12:45:22.528692 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.528621 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp" event={"ID":"38c90c2c-d74a-4c6e-b6a7-f5d82958cc49","Type":"ContainerStarted","Data":"95e21bf037dd67fe3f579dea03a6e9079a86609b4885d98f4f336aec81864b31"} Jul 16 12:45:22.622876 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.622796 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f992ddba-708b-4d18-9d23-b92f474c885a-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss\" (UID: \"f992ddba-708b-4d18-9d23-b92f474c885a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss" Jul 16 12:45:22.622876 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.622841 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj2c4\" (UniqueName: \"kubernetes.io/projected/f992ddba-708b-4d18-9d23-b92f474c885a-kube-api-access-wj2c4\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss\" (UID: \"f992ddba-708b-4d18-9d23-b92f474c885a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss" Jul 16 12:45:22.623077 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.622919 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f992ddba-708b-4d18-9d23-b92f474c885a-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss\" (UID: \"f992ddba-708b-4d18-9d23-b92f474c885a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss" Jul 16 12:45:22.623230 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.623206 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f992ddba-708b-4d18-9d23-b92f474c885a-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss\" (UID: \"f992ddba-708b-4d18-9d23-b92f474c885a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss" Jul 16 12:45:22.623312 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.623270 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f992ddba-708b-4d18-9d23-b92f474c885a-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss\" (UID: \"f992ddba-708b-4d18-9d23-b92f474c885a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss" Jul 16 12:45:22.631778 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.631756 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj2c4\" (UniqueName: \"kubernetes.io/projected/f992ddba-708b-4d18-9d23-b92f474c885a-kube-api-access-wj2c4\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss\" (UID: \"f992ddba-708b-4d18-9d23-b92f474c885a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss" Jul 16 12:45:22.727853 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.727813 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss" Jul 16 12:45:22.806304 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.806273 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6"] Jul 16 12:45:22.810327 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.809958 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6" Jul 16 12:45:22.818664 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.818627 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6"] Jul 16 12:45:22.865117 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.865094 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss"] Jul 16 12:45:22.866947 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:45:22.866921 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf992ddba_708b_4d18_9d23_b92f474c885a.slice/crio-c7469bbe889dc46016c7d4b312f786d1a12417eaae1b09ed3d9ec784adc142c9 WatchSource:0}: Error finding container c7469bbe889dc46016c7d4b312f786d1a12417eaae1b09ed3d9ec784adc142c9: Status 404 returned error can't find the container with id c7469bbe889dc46016c7d4b312f786d1a12417eaae1b09ed3d9ec784adc142c9 Jul 16 12:45:22.925718 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.925679 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e98fbe22-4c45-412d-b87a-010ea40c07c3-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6\" (UID: \"e98fbe22-4c45-412d-b87a-010ea40c07c3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6" Jul 16 12:45:22.925813 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.925742 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e98fbe22-4c45-412d-b87a-010ea40c07c3-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6\" (UID: \"e98fbe22-4c45-412d-b87a-010ea40c07c3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6" Jul 16 12:45:22.925903 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:22.925883 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7rpw\" (UniqueName: \"kubernetes.io/projected/e98fbe22-4c45-412d-b87a-010ea40c07c3-kube-api-access-z7rpw\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6\" (UID: \"e98fbe22-4c45-412d-b87a-010ea40c07c3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6" Jul 16 12:45:23.026987 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:23.026946 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e98fbe22-4c45-412d-b87a-010ea40c07c3-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6\" (UID: \"e98fbe22-4c45-412d-b87a-010ea40c07c3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6" Jul 16 12:45:23.027151 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:23.027001 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e98fbe22-4c45-412d-b87a-010ea40c07c3-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6\" (UID: \"e98fbe22-4c45-412d-b87a-010ea40c07c3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6" Jul 16 12:45:23.027151 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:23.027055 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7rpw\" (UniqueName: \"kubernetes.io/projected/e98fbe22-4c45-412d-b87a-010ea40c07c3-kube-api-access-z7rpw\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6\" (UID: \"e98fbe22-4c45-412d-b87a-010ea40c07c3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6" Jul 16 12:45:23.027439 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:23.027379 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e98fbe22-4c45-412d-b87a-010ea40c07c3-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6\" (UID: \"e98fbe22-4c45-412d-b87a-010ea40c07c3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6" Jul 16 12:45:23.027487 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:23.027415 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e98fbe22-4c45-412d-b87a-010ea40c07c3-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6\" (UID: \"e98fbe22-4c45-412d-b87a-010ea40c07c3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6" Jul 16 12:45:23.035268 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:23.035244 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7rpw\" (UniqueName: \"kubernetes.io/projected/e98fbe22-4c45-412d-b87a-010ea40c07c3-kube-api-access-z7rpw\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6\" (UID: \"e98fbe22-4c45-412d-b87a-010ea40c07c3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6" Jul 16 12:45:23.122001 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:23.121951 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6" Jul 16 12:45:23.252563 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:23.252525 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6"] Jul 16 12:45:23.254617 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:45:23.254597 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode98fbe22_4c45_412d_b87a_010ea40c07c3.slice/crio-f1c90f556b7079919135c93dacaee9479a514d766a93d4372d98f554ecaa96d3 WatchSource:0}: Error finding container f1c90f556b7079919135c93dacaee9479a514d766a93d4372d98f554ecaa96d3: Status 404 returned error can't find the container with id f1c90f556b7079919135c93dacaee9479a514d766a93d4372d98f554ecaa96d3 Jul 16 12:45:23.537968 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:23.537881 2582 generic.go:356] "Generic (PLEG): container finished" podID="38c90c2c-d74a-4c6e-b6a7-f5d82958cc49" containerID="e905735b846c9f3a1e827725fcb456426b17042b75292a9b2170e626d93ce1aa" exitCode=0 Jul 16 12:45:23.538359 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:23.537975 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp" event={"ID":"38c90c2c-d74a-4c6e-b6a7-f5d82958cc49","Type":"ContainerDied","Data":"e905735b846c9f3a1e827725fcb456426b17042b75292a9b2170e626d93ce1aa"} Jul 16 12:45:23.539563 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:23.539542 2582 generic.go:356] "Generic (PLEG): container finished" podID="f992ddba-708b-4d18-9d23-b92f474c885a" containerID="ebe6a0395784c1d977c8b5c3941261c8ddc22f526278373f4604e159b78d5d53" exitCode=0 Jul 16 12:45:23.539677 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:23.539625 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss" event={"ID":"f992ddba-708b-4d18-9d23-b92f474c885a","Type":"ContainerDied","Data":"ebe6a0395784c1d977c8b5c3941261c8ddc22f526278373f4604e159b78d5d53"} Jul 16 12:45:23.539677 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:23.539666 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss" event={"ID":"f992ddba-708b-4d18-9d23-b92f474c885a","Type":"ContainerStarted","Data":"c7469bbe889dc46016c7d4b312f786d1a12417eaae1b09ed3d9ec784adc142c9"} Jul 16 12:45:23.541137 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:23.541115 2582 generic.go:356] "Generic (PLEG): container finished" podID="e98fbe22-4c45-412d-b87a-010ea40c07c3" containerID="b1d332cb3d35a0606ddaae2185f1c2198ec8a820d5955c86bc50fda5592c1c62" exitCode=0 Jul 16 12:45:23.541236 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:23.541163 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6" event={"ID":"e98fbe22-4c45-412d-b87a-010ea40c07c3","Type":"ContainerDied","Data":"b1d332cb3d35a0606ddaae2185f1c2198ec8a820d5955c86bc50fda5592c1c62"} Jul 16 12:45:23.541236 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:23.541197 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6" event={"ID":"e98fbe22-4c45-412d-b87a-010ea40c07c3","Type":"ContainerStarted","Data":"f1c90f556b7079919135c93dacaee9479a514d766a93d4372d98f554ecaa96d3"} Jul 16 12:45:25.552571 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:25.552533 2582 generic.go:356] "Generic (PLEG): container finished" podID="38c90c2c-d74a-4c6e-b6a7-f5d82958cc49" containerID="0d9021c36dcf05c30f9c35990523938b073e5943ccd8eb1debd1e38966954976" exitCode=0 Jul 16 12:45:25.552974 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:25.552601 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp" event={"ID":"38c90c2c-d74a-4c6e-b6a7-f5d82958cc49","Type":"ContainerDied","Data":"0d9021c36dcf05c30f9c35990523938b073e5943ccd8eb1debd1e38966954976"} Jul 16 12:45:25.554537 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:25.554514 2582 generic.go:356] "Generic (PLEG): container finished" podID="aa451150-aecc-4259-baa2-b590d4bceed0" containerID="1fd8427bb646daf3a93187cdd5baa51980888187290cbff5d639246a4636f052" exitCode=0 Jul 16 12:45:25.554617 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:25.554538 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz" event={"ID":"aa451150-aecc-4259-baa2-b590d4bceed0","Type":"ContainerDied","Data":"1fd8427bb646daf3a93187cdd5baa51980888187290cbff5d639246a4636f052"} Jul 16 12:45:25.556544 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:25.556528 2582 generic.go:356] "Generic (PLEG): container finished" podID="f992ddba-708b-4d18-9d23-b92f474c885a" containerID="e75fc1cdf6e7772759ab77275a5846b30dbb8b18e291b58eacd3fff449826916" exitCode=0 Jul 16 12:45:25.556630 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:25.556608 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss" event={"ID":"f992ddba-708b-4d18-9d23-b92f474c885a","Type":"ContainerDied","Data":"e75fc1cdf6e7772759ab77275a5846b30dbb8b18e291b58eacd3fff449826916"} Jul 16 12:45:25.558482 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:25.558463 2582 generic.go:356] "Generic (PLEG): container finished" podID="e98fbe22-4c45-412d-b87a-010ea40c07c3" containerID="371d0ae2a467bce31860da4eedbb168748cbef7e9931cde45716664729f6f1db" exitCode=0 Jul 16 12:45:25.558560 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:25.558507 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6" event={"ID":"e98fbe22-4c45-412d-b87a-010ea40c07c3","Type":"ContainerDied","Data":"371d0ae2a467bce31860da4eedbb168748cbef7e9931cde45716664729f6f1db"} Jul 16 12:45:26.564691 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:26.564655 2582 generic.go:356] "Generic (PLEG): container finished" podID="38c90c2c-d74a-4c6e-b6a7-f5d82958cc49" containerID="9cad97ed7a21fb8a291b46148ff88065dfc4c9f164ee2755257a5a1ede4413ab" exitCode=0 Jul 16 12:45:26.565128 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:26.564744 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp" event={"ID":"38c90c2c-d74a-4c6e-b6a7-f5d82958cc49","Type":"ContainerDied","Data":"9cad97ed7a21fb8a291b46148ff88065dfc4c9f164ee2755257a5a1ede4413ab"} Jul 16 12:45:26.566679 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:26.566658 2582 generic.go:356] "Generic (PLEG): container finished" podID="aa451150-aecc-4259-baa2-b590d4bceed0" containerID="e38ca2952c533add6760b1724da1e77f3c89954a707cc1acb69915f9298ed576" exitCode=0 Jul 16 12:45:26.566802 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:26.566687 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz" event={"ID":"aa451150-aecc-4259-baa2-b590d4bceed0","Type":"ContainerDied","Data":"e38ca2952c533add6760b1724da1e77f3c89954a707cc1acb69915f9298ed576"} Jul 16 12:45:26.573980 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:26.573945 2582 generic.go:356] "Generic (PLEG): container finished" podID="f992ddba-708b-4d18-9d23-b92f474c885a" containerID="0409234ba6280e7d20d29228e4a7c91217802ff5e646c18d4bb4b14f62885408" exitCode=0 Jul 16 12:45:26.574110 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:26.574018 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss" event={"ID":"f992ddba-708b-4d18-9d23-b92f474c885a","Type":"ContainerDied","Data":"0409234ba6280e7d20d29228e4a7c91217802ff5e646c18d4bb4b14f62885408"} Jul 16 12:45:26.576047 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:26.576025 2582 generic.go:356] "Generic (PLEG): container finished" podID="e98fbe22-4c45-412d-b87a-010ea40c07c3" containerID="ca1e74f3d2b2574d0f297ff501564f64b223f5251ae52dd2282b8c4c295b8d6b" exitCode=0 Jul 16 12:45:26.576133 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:26.576061 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6" event={"ID":"e98fbe22-4c45-412d-b87a-010ea40c07c3","Type":"ContainerDied","Data":"ca1e74f3d2b2574d0f297ff501564f64b223f5251ae52dd2282b8c4c295b8d6b"} Jul 16 12:45:27.734659 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:27.734629 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6" Jul 16 12:45:27.770521 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:27.770496 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp" Jul 16 12:45:27.774282 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:27.774258 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss" Jul 16 12:45:27.777367 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:27.777350 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz" Jul 16 12:45:27.866423 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:27.866341 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e98fbe22-4c45-412d-b87a-010ea40c07c3-util\") pod \"e98fbe22-4c45-412d-b87a-010ea40c07c3\" (UID: \"e98fbe22-4c45-412d-b87a-010ea40c07c3\") " Jul 16 12:45:27.866423 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:27.866383 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7rpw\" (UniqueName: \"kubernetes.io/projected/e98fbe22-4c45-412d-b87a-010ea40c07c3-kube-api-access-z7rpw\") pod \"e98fbe22-4c45-412d-b87a-010ea40c07c3\" (UID: \"e98fbe22-4c45-412d-b87a-010ea40c07c3\") " Jul 16 12:45:27.866585 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:27.866487 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e98fbe22-4c45-412d-b87a-010ea40c07c3-bundle\") pod \"e98fbe22-4c45-412d-b87a-010ea40c07c3\" (UID: \"e98fbe22-4c45-412d-b87a-010ea40c07c3\") " Jul 16 12:45:27.867043 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:27.866998 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e98fbe22-4c45-412d-b87a-010ea40c07c3-bundle" (OuterVolumeSpecName: "bundle") pod "e98fbe22-4c45-412d-b87a-010ea40c07c3" (UID: "e98fbe22-4c45-412d-b87a-010ea40c07c3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:45:27.868787 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:27.868766 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e98fbe22-4c45-412d-b87a-010ea40c07c3-kube-api-access-z7rpw" (OuterVolumeSpecName: "kube-api-access-z7rpw") pod "e98fbe22-4c45-412d-b87a-010ea40c07c3" (UID: "e98fbe22-4c45-412d-b87a-010ea40c07c3"). InnerVolumeSpecName "kube-api-access-z7rpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:45:27.871736 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:27.871714 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e98fbe22-4c45-412d-b87a-010ea40c07c3-util" (OuterVolumeSpecName: "util") pod "e98fbe22-4c45-412d-b87a-010ea40c07c3" (UID: "e98fbe22-4c45-412d-b87a-010ea40c07c3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:45:27.967632 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:27.967602 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38c90c2c-d74a-4c6e-b6a7-f5d82958cc49-util\") pod \"38c90c2c-d74a-4c6e-b6a7-f5d82958cc49\" (UID: \"38c90c2c-d74a-4c6e-b6a7-f5d82958cc49\") " Jul 16 12:45:27.967760 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:27.967639 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wj2c4\" (UniqueName: \"kubernetes.io/projected/f992ddba-708b-4d18-9d23-b92f474c885a-kube-api-access-wj2c4\") pod \"f992ddba-708b-4d18-9d23-b92f474c885a\" (UID: \"f992ddba-708b-4d18-9d23-b92f474c885a\") " Jul 16 12:45:27.967760 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:27.967663 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa451150-aecc-4259-baa2-b590d4bceed0-util\") pod \"aa451150-aecc-4259-baa2-b590d4bceed0\" (UID: \"aa451150-aecc-4259-baa2-b590d4bceed0\") " Jul 16 12:45:27.967760 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:27.967691 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7tv96\" (UniqueName: \"kubernetes.io/projected/38c90c2c-d74a-4c6e-b6a7-f5d82958cc49-kube-api-access-7tv96\") pod \"38c90c2c-d74a-4c6e-b6a7-f5d82958cc49\" (UID: \"38c90c2c-d74a-4c6e-b6a7-f5d82958cc49\") " Jul 16 12:45:27.967760 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:27.967715 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f992ddba-708b-4d18-9d23-b92f474c885a-util\") pod \"f992ddba-708b-4d18-9d23-b92f474c885a\" (UID: \"f992ddba-708b-4d18-9d23-b92f474c885a\") " Jul 16 12:45:27.967760 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:27.967755 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f992ddba-708b-4d18-9d23-b92f474c885a-bundle\") pod \"f992ddba-708b-4d18-9d23-b92f474c885a\" (UID: \"f992ddba-708b-4d18-9d23-b92f474c885a\") " Jul 16 12:45:27.967989 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:27.967810 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa451150-aecc-4259-baa2-b590d4bceed0-bundle\") pod \"aa451150-aecc-4259-baa2-b590d4bceed0\" (UID: \"aa451150-aecc-4259-baa2-b590d4bceed0\") " Jul 16 12:45:27.967989 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:27.967834 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38c90c2c-d74a-4c6e-b6a7-f5d82958cc49-bundle\") pod \"38c90c2c-d74a-4c6e-b6a7-f5d82958cc49\" (UID: \"38c90c2c-d74a-4c6e-b6a7-f5d82958cc49\") " Jul 16 12:45:27.967989 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:27.967869 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fggtn\" (UniqueName: \"kubernetes.io/projected/aa451150-aecc-4259-baa2-b590d4bceed0-kube-api-access-fggtn\") pod \"aa451150-aecc-4259-baa2-b590d4bceed0\" (UID: \"aa451150-aecc-4259-baa2-b590d4bceed0\") " Jul 16 12:45:27.968207 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:27.968146 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e98fbe22-4c45-412d-b87a-010ea40c07c3-util\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:45:27.968207 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:27.968165 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7rpw\" (UniqueName: \"kubernetes.io/projected/e98fbe22-4c45-412d-b87a-010ea40c07c3-kube-api-access-z7rpw\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:45:27.968207 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:27.968180 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e98fbe22-4c45-412d-b87a-010ea40c07c3-bundle\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:45:27.968572 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:27.968537 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa451150-aecc-4259-baa2-b590d4bceed0-bundle" (OuterVolumeSpecName: "bundle") pod "aa451150-aecc-4259-baa2-b590d4bceed0" (UID: "aa451150-aecc-4259-baa2-b590d4bceed0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:45:27.968572 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:27.968553 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f992ddba-708b-4d18-9d23-b92f474c885a-bundle" (OuterVolumeSpecName: "bundle") pod "f992ddba-708b-4d18-9d23-b92f474c885a" (UID: "f992ddba-708b-4d18-9d23-b92f474c885a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:45:27.969095 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:27.969052 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38c90c2c-d74a-4c6e-b6a7-f5d82958cc49-bundle" (OuterVolumeSpecName: "bundle") pod "38c90c2c-d74a-4c6e-b6a7-f5d82958cc49" (UID: "38c90c2c-d74a-4c6e-b6a7-f5d82958cc49"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:45:27.970362 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:27.970336 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f992ddba-708b-4d18-9d23-b92f474c885a-kube-api-access-wj2c4" (OuterVolumeSpecName: "kube-api-access-wj2c4") pod "f992ddba-708b-4d18-9d23-b92f474c885a" (UID: "f992ddba-708b-4d18-9d23-b92f474c885a"). InnerVolumeSpecName "kube-api-access-wj2c4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:45:27.970570 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:27.970538 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38c90c2c-d74a-4c6e-b6a7-f5d82958cc49-kube-api-access-7tv96" (OuterVolumeSpecName: "kube-api-access-7tv96") pod "38c90c2c-d74a-4c6e-b6a7-f5d82958cc49" (UID: "38c90c2c-d74a-4c6e-b6a7-f5d82958cc49"). InnerVolumeSpecName "kube-api-access-7tv96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:45:27.970646 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:27.970626 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa451150-aecc-4259-baa2-b590d4bceed0-kube-api-access-fggtn" (OuterVolumeSpecName: "kube-api-access-fggtn") pod "aa451150-aecc-4259-baa2-b590d4bceed0" (UID: "aa451150-aecc-4259-baa2-b590d4bceed0"). InnerVolumeSpecName "kube-api-access-fggtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:45:27.973429 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:27.973383 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa451150-aecc-4259-baa2-b590d4bceed0-util" (OuterVolumeSpecName: "util") pod "aa451150-aecc-4259-baa2-b590d4bceed0" (UID: "aa451150-aecc-4259-baa2-b590d4bceed0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:45:27.977062 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:27.977024 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f992ddba-708b-4d18-9d23-b92f474c885a-util" (OuterVolumeSpecName: "util") pod "f992ddba-708b-4d18-9d23-b92f474c885a" (UID: "f992ddba-708b-4d18-9d23-b92f474c885a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:45:27.977558 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:27.977534 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38c90c2c-d74a-4c6e-b6a7-f5d82958cc49-util" (OuterVolumeSpecName: "util") pod "38c90c2c-d74a-4c6e-b6a7-f5d82958cc49" (UID: "38c90c2c-d74a-4c6e-b6a7-f5d82958cc49"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:45:28.069030 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:28.068994 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38c90c2c-d74a-4c6e-b6a7-f5d82958cc49-util\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:45:28.069030 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:28.069025 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj2c4\" (UniqueName: \"kubernetes.io/projected/f992ddba-708b-4d18-9d23-b92f474c885a-kube-api-access-wj2c4\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:45:28.069214 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:28.069039 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa451150-aecc-4259-baa2-b590d4bceed0-util\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:45:28.069214 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:28.069053 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tv96\" (UniqueName: \"kubernetes.io/projected/38c90c2c-d74a-4c6e-b6a7-f5d82958cc49-kube-api-access-7tv96\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:45:28.069214 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:28.069064 2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f992ddba-708b-4d18-9d23-b92f474c885a-util\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:45:28.069214 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:28.069075 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f992ddba-708b-4d18-9d23-b92f474c885a-bundle\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:45:28.069214 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:28.069085 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa451150-aecc-4259-baa2-b590d4bceed0-bundle\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:45:28.069214 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:28.069095 2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38c90c2c-d74a-4c6e-b6a7-f5d82958cc49-bundle\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:45:28.069214 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:28.069106 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fggtn\" (UniqueName: \"kubernetes.io/projected/aa451150-aecc-4259-baa2-b590d4bceed0-kube-api-access-fggtn\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:45:28.586957 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:28.586927 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6" Jul 16 12:45:28.586957 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:28.586940 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6" event={"ID":"e98fbe22-4c45-412d-b87a-010ea40c07c3","Type":"ContainerDied","Data":"f1c90f556b7079919135c93dacaee9479a514d766a93d4372d98f554ecaa96d3"} Jul 16 12:45:28.587212 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:28.586990 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1c90f556b7079919135c93dacaee9479a514d766a93d4372d98f554ecaa96d3" Jul 16 12:45:28.588912 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:28.588884 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp" Jul 16 12:45:28.588912 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:28.588894 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp" event={"ID":"38c90c2c-d74a-4c6e-b6a7-f5d82958cc49","Type":"ContainerDied","Data":"95e21bf037dd67fe3f579dea03a6e9079a86609b4885d98f4f336aec81864b31"} Jul 16 12:45:28.589120 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:28.588933 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95e21bf037dd67fe3f579dea03a6e9079a86609b4885d98f4f336aec81864b31" Jul 16 12:45:28.591025 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:28.591004 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz" Jul 16 12:45:28.591153 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:28.591026 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz" event={"ID":"aa451150-aecc-4259-baa2-b590d4bceed0","Type":"ContainerDied","Data":"0b59fff134e24bebe65299feb55bdb587944be67d2a5572230fc18b7b1328a10"} Jul 16 12:45:28.591153 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:28.591069 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b59fff134e24bebe65299feb55bdb587944be67d2a5572230fc18b7b1328a10" Jul 16 12:45:28.593503 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:28.593479 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss" event={"ID":"f992ddba-708b-4d18-9d23-b92f474c885a","Type":"ContainerDied","Data":"c7469bbe889dc46016c7d4b312f786d1a12417eaae1b09ed3d9ec784adc142c9"} Jul 16 12:45:28.593628 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:28.593514 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7469bbe889dc46016c7d4b312f786d1a12417eaae1b09ed3d9ec784adc142c9" Jul 16 12:45:28.593628 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:28.593553 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss" Jul 16 12:45:34.159818 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:34.159780 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:45:34.703549 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:34.703517 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:45:40.858640 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:40.858605 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:45:41.592102 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:41.592070 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:45:41.702896 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:41.702862 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:45:42.104336 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:42.104304 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:45:43.966199 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:43.966165 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:45:45.739774 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:45.739745 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:45:47.345669 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:47.345635 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:45:49.022404 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:49.022354 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:45:50.904780 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:50.904743 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:45:51.006076 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:51.006045 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:45:51.300553 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:51.300469 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:45:52.771509 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:52.771472 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:45:54.923547 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:54.923510 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:45:55.028668 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:55.028637 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:45:55.740382 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:55.740348 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:45:55.901679 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:55.901648 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:45:57.287931 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:57.287905 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:45:57.387187 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:57.387155 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:45:58.170217 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:58.170187 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:45:58.269832 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:58.269802 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:45:59.130997 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:59.130969 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:45:59.231063 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:45:59.231031 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:46:04.625588 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:04.625552 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:46:04.722199 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:04.722164 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:46:08.859708 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:08.859668 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:46:08.961588 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:08.961548 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:46:10.290374 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:10.290341 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:46:10.392942 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:10.392915 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:46:11.535967 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:11.535932 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:46:11.637721 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:11.637681 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:46:15.476329 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:15.476292 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:46:15.576772 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:15.576736 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:46:18.568085 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:18.568053 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:46:18.889660 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:18.889625 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:46:19.004247 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:19.004214 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:46:27.087195 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:27.087114 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:46:27.148991 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:27.148958 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:46:29.708659 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:29.708624 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-645f7565bc-5c4pr"] Jul 16 12:46:29.871348 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:29.871316 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:46:29.976228 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:29.976147 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:46:30.072599 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:30.072569 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:46:38.455842 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:38.455812 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:46:38.615522 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:38.615491 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:46:46.159268 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:46.159232 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:46:52.681396 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:52.681359 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:46:52.776530 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:52.776497 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:46:52.830377 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:52.830348 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:46:53.464846 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.464814 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-c66ph"] Jul 16 12:46:53.465091 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.465079 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa451150-aecc-4259-baa2-b590d4bceed0" containerName="pull" Jul 16 12:46:53.465133 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.465092 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa451150-aecc-4259-baa2-b590d4bceed0" containerName="pull" Jul 16 12:46:53.465133 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.465103 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f992ddba-708b-4d18-9d23-b92f474c885a" containerName="extract" Jul 16 12:46:53.465133 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.465108 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f992ddba-708b-4d18-9d23-b92f474c885a" containerName="extract" Jul 16 12:46:53.465133 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.465116 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa451150-aecc-4259-baa2-b590d4bceed0" containerName="util" Jul 16 12:46:53.465133 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.465121 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa451150-aecc-4259-baa2-b590d4bceed0" containerName="util" Jul 16 12:46:53.465133 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.465127 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38c90c2c-d74a-4c6e-b6a7-f5d82958cc49" containerName="pull" Jul 16 12:46:53.465133 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.465132 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="38c90c2c-d74a-4c6e-b6a7-f5d82958cc49" containerName="pull" Jul 16 12:46:53.465333 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.465142 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e98fbe22-4c45-412d-b87a-010ea40c07c3" containerName="util" Jul 16 12:46:53.465333 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.465146 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e98fbe22-4c45-412d-b87a-010ea40c07c3" containerName="util" Jul 16 12:46:53.465333 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.465153 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38c90c2c-d74a-4c6e-b6a7-f5d82958cc49" containerName="util" Jul 16 12:46:53.465333 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.465158 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="38c90c2c-d74a-4c6e-b6a7-f5d82958cc49" containerName="util" Jul 16 12:46:53.465333 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.465164 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f992ddba-708b-4d18-9d23-b92f474c885a" containerName="pull" Jul 16 12:46:53.465333 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.465168 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f992ddba-708b-4d18-9d23-b92f474c885a" containerName="pull" Jul 16 12:46:53.465333 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.465173 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e98fbe22-4c45-412d-b87a-010ea40c07c3" containerName="pull" Jul 16 12:46:53.465333 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.465177 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e98fbe22-4c45-412d-b87a-010ea40c07c3" containerName="pull" Jul 16 12:46:53.465333 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.465184 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e98fbe22-4c45-412d-b87a-010ea40c07c3" containerName="extract" Jul 16 12:46:53.465333 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.465188 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e98fbe22-4c45-412d-b87a-010ea40c07c3" containerName="extract" Jul 16 12:46:53.465333 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.465194 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f992ddba-708b-4d18-9d23-b92f474c885a" containerName="util" Jul 16 12:46:53.465333 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.465199 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f992ddba-708b-4d18-9d23-b92f474c885a" containerName="util" Jul 16 12:46:53.465333 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.465203 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa451150-aecc-4259-baa2-b590d4bceed0" containerName="extract" Jul 16 12:46:53.465333 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.465207 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa451150-aecc-4259-baa2-b590d4bceed0" containerName="extract" Jul 16 12:46:53.465333 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.465214 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38c90c2c-d74a-4c6e-b6a7-f5d82958cc49" containerName="extract" Jul 16 12:46:53.465333 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.465218 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="38c90c2c-d74a-4c6e-b6a7-f5d82958cc49" containerName="extract" Jul 16 12:46:53.465333 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.465268 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="38c90c2c-d74a-4c6e-b6a7-f5d82958cc49" containerName="extract" Jul 16 12:46:53.465333 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.465274 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f992ddba-708b-4d18-9d23-b92f474c885a" containerName="extract" Jul 16 12:46:53.465333 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.465280 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e98fbe22-4c45-412d-b87a-010ea40c07c3" containerName="extract" Jul 16 12:46:53.465333 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.465286 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa451150-aecc-4259-baa2-b590d4bceed0" containerName="extract" Jul 16 12:46:53.468036 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.467997 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-c66ph" Jul 16 12:46:53.470448 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.470429 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 16 12:46:53.470556 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.470429 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zgjb2\"" Jul 16 12:46:53.471290 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.471275 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 16 12:46:53.476195 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.476178 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-c66ph"] Jul 16 12:46:53.494563 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.494539 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:46:53.543351 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.543328 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56fk9\" (UniqueName: \"kubernetes.io/projected/54c7ab1b-01f0-4698-a51e-9b7007a5575b-kube-api-access-56fk9\") pod \"authorino-58cd955675-c66ph\" (UID: \"54c7ab1b-01f0-4698-a51e-9b7007a5575b\") " pod="kuadrant-system/authorino-58cd955675-c66ph" Jul 16 12:46:53.559766 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.559738 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:46:53.644126 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.644099 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56fk9\" (UniqueName: \"kubernetes.io/projected/54c7ab1b-01f0-4698-a51e-9b7007a5575b-kube-api-access-56fk9\") pod \"authorino-58cd955675-c66ph\" (UID: \"54c7ab1b-01f0-4698-a51e-9b7007a5575b\") " pod="kuadrant-system/authorino-58cd955675-c66ph" Jul 16 12:46:53.652171 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.652144 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56fk9\" (UniqueName: \"kubernetes.io/projected/54c7ab1b-01f0-4698-a51e-9b7007a5575b-kube-api-access-56fk9\") pod \"authorino-58cd955675-c66ph\" (UID: \"54c7ab1b-01f0-4698-a51e-9b7007a5575b\") " pod="kuadrant-system/authorino-58cd955675-c66ph" Jul 16 12:46:53.777524 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.777448 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-c66ph" Jul 16 12:46:53.906701 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.906676 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-c66ph"] Jul 16 12:46:53.909099 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:46:53.909072 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54c7ab1b_01f0_4698_a51e_9b7007a5575b.slice/crio-a7c7b22ddbca0fa5819e30627df048ff703eea806c54a9b4ad55c1726afb61b2 WatchSource:0}: Error finding container a7c7b22ddbca0fa5819e30627df048ff703eea806c54a9b4ad55c1726afb61b2: Status 404 returned error can't find the container with id a7c7b22ddbca0fa5819e30627df048ff703eea806c54a9b4ad55c1726afb61b2 Jul 16 12:46:53.981403 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:53.981359 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-c66ph" event={"ID":"54c7ab1b-01f0-4698-a51e-9b7007a5575b","Type":"ContainerStarted","Data":"a7c7b22ddbca0fa5819e30627df048ff703eea806c54a9b4ad55c1726afb61b2"} Jul 16 12:46:54.729338 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:54.729143 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-645f7565bc-5c4pr" podUID="ef74c52a-35d9-4b5c-9186-a494580f4011" containerName="console" containerID="cri-o://6c0156808f4520c2b8b411d7d5532f77f2b3d76f1a162cb3f8b51a4b5101704f" gracePeriod=15 Jul 16 12:46:54.991417 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:54.991169 2582 generic.go:356] "Generic (PLEG): container finished" podID="ef74c52a-35d9-4b5c-9186-a494580f4011" containerID="6c0156808f4520c2b8b411d7d5532f77f2b3d76f1a162cb3f8b51a4b5101704f" exitCode=0 Jul 16 12:46:54.991417 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:54.991243 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-645f7565bc-5c4pr" event={"ID":"ef74c52a-35d9-4b5c-9186-a494580f4011","Type":"ContainerDied","Data":"6c0156808f4520c2b8b411d7d5532f77f2b3d76f1a162cb3f8b51a4b5101704f"} Jul 16 12:46:55.028163 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:55.028137 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-645f7565bc-5c4pr" Jul 16 12:46:55.056698 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:55.056552 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef74c52a-35d9-4b5c-9186-a494580f4011-trusted-ca-bundle\") pod \"ef74c52a-35d9-4b5c-9186-a494580f4011\" (UID: \"ef74c52a-35d9-4b5c-9186-a494580f4011\") " Jul 16 12:46:55.056698 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:55.056611 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmtqs\" (UniqueName: \"kubernetes.io/projected/ef74c52a-35d9-4b5c-9186-a494580f4011-kube-api-access-kmtqs\") pod \"ef74c52a-35d9-4b5c-9186-a494580f4011\" (UID: \"ef74c52a-35d9-4b5c-9186-a494580f4011\") " Jul 16 12:46:55.056698 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:55.056686 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef74c52a-35d9-4b5c-9186-a494580f4011-oauth-serving-cert\") pod \"ef74c52a-35d9-4b5c-9186-a494580f4011\" (UID: \"ef74c52a-35d9-4b5c-9186-a494580f4011\") " Jul 16 12:46:55.056977 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:55.056718 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef74c52a-35d9-4b5c-9186-a494580f4011-console-oauth-config\") pod \"ef74c52a-35d9-4b5c-9186-a494580f4011\" (UID: \"ef74c52a-35d9-4b5c-9186-a494580f4011\") " Jul 16 12:46:55.057907 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:55.057179 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef74c52a-35d9-4b5c-9186-a494580f4011-console-serving-cert\") pod \"ef74c52a-35d9-4b5c-9186-a494580f4011\" (UID: \"ef74c52a-35d9-4b5c-9186-a494580f4011\") " Jul 16 12:46:55.057907 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:55.057173 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef74c52a-35d9-4b5c-9186-a494580f4011-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ef74c52a-35d9-4b5c-9186-a494580f4011" (UID: "ef74c52a-35d9-4b5c-9186-a494580f4011"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:46:55.057907 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:55.057244 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef74c52a-35d9-4b5c-9186-a494580f4011-console-config\") pod \"ef74c52a-35d9-4b5c-9186-a494580f4011\" (UID: \"ef74c52a-35d9-4b5c-9186-a494580f4011\") " Jul 16 12:46:55.057907 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:55.057281 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef74c52a-35d9-4b5c-9186-a494580f4011-service-ca\") pod \"ef74c52a-35d9-4b5c-9186-a494580f4011\" (UID: \"ef74c52a-35d9-4b5c-9186-a494580f4011\") " Jul 16 12:46:55.057907 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:55.057539 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef74c52a-35d9-4b5c-9186-a494580f4011-trusted-ca-bundle\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:46:55.057907 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:55.057825 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef74c52a-35d9-4b5c-9186-a494580f4011-console-config" (OuterVolumeSpecName: "console-config") pod "ef74c52a-35d9-4b5c-9186-a494580f4011" (UID: "ef74c52a-35d9-4b5c-9186-a494580f4011"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:46:55.058286 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:55.057909 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef74c52a-35d9-4b5c-9186-a494580f4011-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ef74c52a-35d9-4b5c-9186-a494580f4011" (UID: "ef74c52a-35d9-4b5c-9186-a494580f4011"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:46:55.058286 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:55.058159 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef74c52a-35d9-4b5c-9186-a494580f4011-service-ca" (OuterVolumeSpecName: "service-ca") pod "ef74c52a-35d9-4b5c-9186-a494580f4011" (UID: "ef74c52a-35d9-4b5c-9186-a494580f4011"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:46:55.059957 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:55.059922 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef74c52a-35d9-4b5c-9186-a494580f4011-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ef74c52a-35d9-4b5c-9186-a494580f4011" (UID: "ef74c52a-35d9-4b5c-9186-a494580f4011"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:46:55.060327 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:55.060301 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef74c52a-35d9-4b5c-9186-a494580f4011-kube-api-access-kmtqs" (OuterVolumeSpecName: "kube-api-access-kmtqs") pod "ef74c52a-35d9-4b5c-9186-a494580f4011" (UID: "ef74c52a-35d9-4b5c-9186-a494580f4011"). InnerVolumeSpecName "kube-api-access-kmtqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:46:55.061012 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:55.060988 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef74c52a-35d9-4b5c-9186-a494580f4011-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ef74c52a-35d9-4b5c-9186-a494580f4011" (UID: "ef74c52a-35d9-4b5c-9186-a494580f4011"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:46:55.158642 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:55.158577 2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef74c52a-35d9-4b5c-9186-a494580f4011-oauth-serving-cert\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:46:55.158642 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:55.158609 2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef74c52a-35d9-4b5c-9186-a494580f4011-console-oauth-config\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:46:55.158642 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:55.158622 2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef74c52a-35d9-4b5c-9186-a494580f4011-console-serving-cert\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:46:55.158642 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:55.158636 2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef74c52a-35d9-4b5c-9186-a494580f4011-console-config\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:46:55.158642 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:55.158647 2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef74c52a-35d9-4b5c-9186-a494580f4011-service-ca\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:46:55.159003 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:55.158659 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmtqs\" (UniqueName: \"kubernetes.io/projected/ef74c52a-35d9-4b5c-9186-a494580f4011-kube-api-access-kmtqs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:46:55.998222 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:55.998181 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-645f7565bc-5c4pr" event={"ID":"ef74c52a-35d9-4b5c-9186-a494580f4011","Type":"ContainerDied","Data":"0674d56022d68b1ee471f6b3f6a9ad2eb87d0034ac860bb67620847dc0d004b6"} Jul 16 12:46:55.998581 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:55.998204 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-645f7565bc-5c4pr" Jul 16 12:46:55.998581 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:55.998256 2582 scope.go:117] "RemoveContainer" containerID="6c0156808f4520c2b8b411d7d5532f77f2b3d76f1a162cb3f8b51a4b5101704f" Jul 16 12:46:56.015657 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:56.015628 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-645f7565bc-5c4pr"] Jul 16 12:46:56.019239 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:56.019216 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-645f7565bc-5c4pr"] Jul 16 12:46:57.004929 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:57.004892 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-c66ph" event={"ID":"54c7ab1b-01f0-4698-a51e-9b7007a5575b","Type":"ContainerStarted","Data":"c3da060d9a434073f4a566fbd07afd5fc0949adf72234ebbbc63f7ac93d2c6cb"} Jul 16 12:46:57.023815 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:57.023720 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-c66ph" podStartSLOduration=1.107002082 podStartE2EDuration="4.023705336s" totalImagesPullingTime="2.916703254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:46:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:46:57.022565219 +0000 UTC m=+475.500310836" watchObservedRunningTime="2026-07-16 12:46:57.023705336 +0000 UTC m=+475.501450952" Jul 16 12:46:57.976898 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:57.976862 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef74c52a-35d9-4b5c-9186-a494580f4011" path="/var/lib/kubelet/pods/ef74c52a-35d9-4b5c-9186-a494580f4011/volumes" Jul 16 12:46:58.830442 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:58.830415 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-c66ph"] Jul 16 12:46:59.015244 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:59.015205 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-c66ph" podUID="54c7ab1b-01f0-4698-a51e-9b7007a5575b" containerName="authorino" containerID="cri-o://c3da060d9a434073f4a566fbd07afd5fc0949adf72234ebbbc63f7ac93d2c6cb" gracePeriod=30 Jul 16 12:46:59.273091 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:59.273070 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-c66ph" Jul 16 12:46:59.398134 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:59.398047 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56fk9\" (UniqueName: \"kubernetes.io/projected/54c7ab1b-01f0-4698-a51e-9b7007a5575b-kube-api-access-56fk9\") pod \"54c7ab1b-01f0-4698-a51e-9b7007a5575b\" (UID: \"54c7ab1b-01f0-4698-a51e-9b7007a5575b\") " Jul 16 12:46:59.400231 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:59.400201 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c7ab1b-01f0-4698-a51e-9b7007a5575b-kube-api-access-56fk9" (OuterVolumeSpecName: "kube-api-access-56fk9") pod "54c7ab1b-01f0-4698-a51e-9b7007a5575b" (UID: "54c7ab1b-01f0-4698-a51e-9b7007a5575b"). InnerVolumeSpecName "kube-api-access-56fk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:46:59.498726 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:46:59.498694 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56fk9\" (UniqueName: \"kubernetes.io/projected/54c7ab1b-01f0-4698-a51e-9b7007a5575b-kube-api-access-56fk9\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:47:00.020440 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:00.020381 2582 generic.go:356] "Generic (PLEG): container finished" podID="54c7ab1b-01f0-4698-a51e-9b7007a5575b" containerID="c3da060d9a434073f4a566fbd07afd5fc0949adf72234ebbbc63f7ac93d2c6cb" exitCode=0 Jul 16 12:47:00.020800 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:00.020445 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-c66ph" Jul 16 12:47:00.020800 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:00.020482 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-c66ph" event={"ID":"54c7ab1b-01f0-4698-a51e-9b7007a5575b","Type":"ContainerDied","Data":"c3da060d9a434073f4a566fbd07afd5fc0949adf72234ebbbc63f7ac93d2c6cb"} Jul 16 12:47:00.020800 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:00.020520 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-c66ph" event={"ID":"54c7ab1b-01f0-4698-a51e-9b7007a5575b","Type":"ContainerDied","Data":"a7c7b22ddbca0fa5819e30627df048ff703eea806c54a9b4ad55c1726afb61b2"} Jul 16 12:47:00.020800 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:00.020539 2582 scope.go:117] "RemoveContainer" containerID="c3da060d9a434073f4a566fbd07afd5fc0949adf72234ebbbc63f7ac93d2c6cb" Jul 16 12:47:00.029050 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:00.029024 2582 scope.go:117] "RemoveContainer" containerID="c3da060d9a434073f4a566fbd07afd5fc0949adf72234ebbbc63f7ac93d2c6cb" Jul 16 12:47:00.029263 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:47:00.029248 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3da060d9a434073f4a566fbd07afd5fc0949adf72234ebbbc63f7ac93d2c6cb\": container with ID starting with c3da060d9a434073f4a566fbd07afd5fc0949adf72234ebbbc63f7ac93d2c6cb not found: ID does not exist" containerID="c3da060d9a434073f4a566fbd07afd5fc0949adf72234ebbbc63f7ac93d2c6cb" Jul 16 12:47:00.029314 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:00.029269 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3da060d9a434073f4a566fbd07afd5fc0949adf72234ebbbc63f7ac93d2c6cb"} err="failed to get container status \"c3da060d9a434073f4a566fbd07afd5fc0949adf72234ebbbc63f7ac93d2c6cb\": rpc error: code = NotFound desc = could not find container \"c3da060d9a434073f4a566fbd07afd5fc0949adf72234ebbbc63f7ac93d2c6cb\": container with ID starting with c3da060d9a434073f4a566fbd07afd5fc0949adf72234ebbbc63f7ac93d2c6cb not found: ID does not exist" Jul 16 12:47:00.039131 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:00.039109 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-c66ph"] Jul 16 12:47:00.043666 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:00.043646 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-c66ph"] Jul 16 12:47:01.977029 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:01.976999 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54c7ab1b-01f0-4698-a51e-9b7007a5575b" path="/var/lib/kubelet/pods/54c7ab1b-01f0-4698-a51e-9b7007a5575b/volumes" Jul 16 12:47:16.016022 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:16.015990 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:47:16.116900 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:16.116872 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:47:27.503479 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:27.503439 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:47:27.603827 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:27.603793 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:47:27.730806 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:27.730762 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:47:27.946061 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:27.946024 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-86skr"] Jul 16 12:47:27.946234 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:27.946183 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:47:27.946303 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:27.946294 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef74c52a-35d9-4b5c-9186-a494580f4011" containerName="console" Jul 16 12:47:27.946340 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:27.946305 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef74c52a-35d9-4b5c-9186-a494580f4011" containerName="console" Jul 16 12:47:27.946340 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:27.946320 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54c7ab1b-01f0-4698-a51e-9b7007a5575b" containerName="authorino" Jul 16 12:47:27.946340 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:27.946325 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="54c7ab1b-01f0-4698-a51e-9b7007a5575b" containerName="authorino" Jul 16 12:47:27.946482 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:27.946375 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef74c52a-35d9-4b5c-9186-a494580f4011" containerName="console" Jul 16 12:47:27.946482 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:27.946382 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="54c7ab1b-01f0-4698-a51e-9b7007a5575b" containerName="authorino" Jul 16 12:47:27.952597 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:27.952581 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-86skr" Jul 16 12:47:27.955070 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:27.955053 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 16 12:47:27.956131 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:27.956110 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zgjb2\"" Jul 16 12:47:27.956309 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:27.956291 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 16 12:47:27.958484 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:27.958460 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-86skr"] Jul 16 12:47:28.119436 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:28.119373 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnjcj\" (UniqueName: \"kubernetes.io/projected/86f36673-05e6-459a-bc3b-a9dfeaa95e87-kube-api-access-wnjcj\") pod \"authorino-5d4b67c446-86skr\" (UID: \"86f36673-05e6-459a-bc3b-a9dfeaa95e87\") " pod="kuadrant-system/authorino-5d4b67c446-86skr" Jul 16 12:47:28.183989 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:28.183956 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-86skr"] Jul 16 12:47:28.184175 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:47:28.184154 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-wnjcj], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-86skr" podUID="86f36673-05e6-459a-bc3b-a9dfeaa95e87" Jul 16 12:47:28.212621 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:28.212549 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-78cdc7b659-g7v2p"] Jul 16 12:47:28.216304 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:28.216286 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78cdc7b659-g7v2p" Jul 16 12:47:28.220003 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:28.219980 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnjcj\" (UniqueName: \"kubernetes.io/projected/86f36673-05e6-459a-bc3b-a9dfeaa95e87-kube-api-access-wnjcj\") pod \"authorino-5d4b67c446-86skr\" (UID: \"86f36673-05e6-459a-bc3b-a9dfeaa95e87\") " pod="kuadrant-system/authorino-5d4b67c446-86skr" Jul 16 12:47:28.226551 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:28.226533 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78cdc7b659-g7v2p"] Jul 16 12:47:28.231004 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:28.230977 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnjcj\" (UniqueName: \"kubernetes.io/projected/86f36673-05e6-459a-bc3b-a9dfeaa95e87-kube-api-access-wnjcj\") pod \"authorino-5d4b67c446-86skr\" (UID: \"86f36673-05e6-459a-bc3b-a9dfeaa95e87\") " pod="kuadrant-system/authorino-5d4b67c446-86skr" Jul 16 12:47:28.320977 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:28.320946 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dvkf\" (UniqueName: \"kubernetes.io/projected/feb18302-0b5a-442d-bdb7-e6542ac3fe2e-kube-api-access-5dvkf\") pod \"authorino-78cdc7b659-g7v2p\" (UID: \"feb18302-0b5a-442d-bdb7-e6542ac3fe2e\") " pod="kuadrant-system/authorino-78cdc7b659-g7v2p" Jul 16 12:47:28.391698 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:28.391665 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-78cdc7b659-g7v2p"] Jul 16 12:47:28.391903 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:47:28.391884 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-5dvkf], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-78cdc7b659-g7v2p" podUID="feb18302-0b5a-442d-bdb7-e6542ac3fe2e" Jul 16 12:47:28.417591 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:28.417558 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f9765cc5f-r7x5v"] Jul 16 12:47:28.421315 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:28.421298 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f9765cc5f-r7x5v" Jul 16 12:47:28.421533 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:28.421512 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dvkf\" (UniqueName: \"kubernetes.io/projected/feb18302-0b5a-442d-bdb7-e6542ac3fe2e-kube-api-access-5dvkf\") pod \"authorino-78cdc7b659-g7v2p\" (UID: \"feb18302-0b5a-442d-bdb7-e6542ac3fe2e\") " pod="kuadrant-system/authorino-78cdc7b659-g7v2p" Jul 16 12:47:28.423556 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:28.423535 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 16 12:47:28.431449 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:28.431429 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f9765cc5f-r7x5v"] Jul 16 12:47:28.434338 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:28.434319 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dvkf\" (UniqueName: \"kubernetes.io/projected/feb18302-0b5a-442d-bdb7-e6542ac3fe2e-kube-api-access-5dvkf\") pod \"authorino-78cdc7b659-g7v2p\" (UID: \"feb18302-0b5a-442d-bdb7-e6542ac3fe2e\") " pod="kuadrant-system/authorino-78cdc7b659-g7v2p" Jul 16 12:47:28.522833 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:28.522751 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgpgh\" (UniqueName: \"kubernetes.io/projected/6df4f150-dc08-445c-9f36-1c65aa0363b5-kube-api-access-cgpgh\") pod \"authorino-5f9765cc5f-r7x5v\" (UID: \"6df4f150-dc08-445c-9f36-1c65aa0363b5\") " pod="kuadrant-system/authorino-5f9765cc5f-r7x5v" Jul 16 12:47:28.522833 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:28.522806 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6df4f150-dc08-445c-9f36-1c65aa0363b5-tls-cert\") pod \"authorino-5f9765cc5f-r7x5v\" (UID: \"6df4f150-dc08-445c-9f36-1c65aa0363b5\") " pod="kuadrant-system/authorino-5f9765cc5f-r7x5v" Jul 16 12:47:28.623376 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:28.623342 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6df4f150-dc08-445c-9f36-1c65aa0363b5-tls-cert\") pod \"authorino-5f9765cc5f-r7x5v\" (UID: \"6df4f150-dc08-445c-9f36-1c65aa0363b5\") " pod="kuadrant-system/authorino-5f9765cc5f-r7x5v" Jul 16 12:47:28.623557 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:28.623432 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgpgh\" (UniqueName: \"kubernetes.io/projected/6df4f150-dc08-445c-9f36-1c65aa0363b5-kube-api-access-cgpgh\") pod \"authorino-5f9765cc5f-r7x5v\" (UID: \"6df4f150-dc08-445c-9f36-1c65aa0363b5\") " pod="kuadrant-system/authorino-5f9765cc5f-r7x5v" Jul 16 12:47:28.625855 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:28.625827 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6df4f150-dc08-445c-9f36-1c65aa0363b5-tls-cert\") pod \"authorino-5f9765cc5f-r7x5v\" (UID: \"6df4f150-dc08-445c-9f36-1c65aa0363b5\") " pod="kuadrant-system/authorino-5f9765cc5f-r7x5v" Jul 16 12:47:28.631818 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:28.631798 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgpgh\" (UniqueName: \"kubernetes.io/projected/6df4f150-dc08-445c-9f36-1c65aa0363b5-kube-api-access-cgpgh\") pod \"authorino-5f9765cc5f-r7x5v\" (UID: \"6df4f150-dc08-445c-9f36-1c65aa0363b5\") " pod="kuadrant-system/authorino-5f9765cc5f-r7x5v" Jul 16 12:47:28.730651 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:28.730603 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f9765cc5f-r7x5v" Jul 16 12:47:28.861212 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:28.861185 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f9765cc5f-r7x5v"] Jul 16 12:47:28.863561 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:47:28.863527 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6df4f150_dc08_445c_9f36_1c65aa0363b5.slice/crio-474d3a04cd214a9867ee2d82a07b26133b8dc6ae5e29cda607aa8ed92ff09f01 WatchSource:0}: Error finding container 474d3a04cd214a9867ee2d82a07b26133b8dc6ae5e29cda607aa8ed92ff09f01: Status 404 returned error can't find the container with id 474d3a04cd214a9867ee2d82a07b26133b8dc6ae5e29cda607aa8ed92ff09f01 Jul 16 12:47:29.155547 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:29.155514 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-86skr" Jul 16 12:47:29.155547 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:29.155529 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f9765cc5f-r7x5v" event={"ID":"6df4f150-dc08-445c-9f36-1c65aa0363b5","Type":"ContainerStarted","Data":"474d3a04cd214a9867ee2d82a07b26133b8dc6ae5e29cda607aa8ed92ff09f01"} Jul 16 12:47:29.155772 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:29.155584 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78cdc7b659-g7v2p" Jul 16 12:47:29.164222 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:29.164202 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-86skr" Jul 16 12:47:29.166770 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:29.166755 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78cdc7b659-g7v2p" Jul 16 12:47:29.329809 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:29.329780 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnjcj\" (UniqueName: \"kubernetes.io/projected/86f36673-05e6-459a-bc3b-a9dfeaa95e87-kube-api-access-wnjcj\") pod \"86f36673-05e6-459a-bc3b-a9dfeaa95e87\" (UID: \"86f36673-05e6-459a-bc3b-a9dfeaa95e87\") " Jul 16 12:47:29.329958 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:29.329878 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dvkf\" (UniqueName: \"kubernetes.io/projected/feb18302-0b5a-442d-bdb7-e6542ac3fe2e-kube-api-access-5dvkf\") pod \"feb18302-0b5a-442d-bdb7-e6542ac3fe2e\" (UID: \"feb18302-0b5a-442d-bdb7-e6542ac3fe2e\") " Jul 16 12:47:29.332064 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:29.332033 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86f36673-05e6-459a-bc3b-a9dfeaa95e87-kube-api-access-wnjcj" (OuterVolumeSpecName: "kube-api-access-wnjcj") pod "86f36673-05e6-459a-bc3b-a9dfeaa95e87" (UID: "86f36673-05e6-459a-bc3b-a9dfeaa95e87"). InnerVolumeSpecName "kube-api-access-wnjcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:47:29.332064 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:29.332042 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feb18302-0b5a-442d-bdb7-e6542ac3fe2e-kube-api-access-5dvkf" (OuterVolumeSpecName: "kube-api-access-5dvkf") pod "feb18302-0b5a-442d-bdb7-e6542ac3fe2e" (UID: "feb18302-0b5a-442d-bdb7-e6542ac3fe2e"). InnerVolumeSpecName "kube-api-access-5dvkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:47:29.430706 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:29.430680 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dvkf\" (UniqueName: \"kubernetes.io/projected/feb18302-0b5a-442d-bdb7-e6542ac3fe2e-kube-api-access-5dvkf\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:47:29.430706 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:29.430708 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnjcj\" (UniqueName: \"kubernetes.io/projected/86f36673-05e6-459a-bc3b-a9dfeaa95e87-kube-api-access-wnjcj\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:47:30.162217 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:30.162182 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f9765cc5f-r7x5v" event={"ID":"6df4f150-dc08-445c-9f36-1c65aa0363b5","Type":"ContainerStarted","Data":"d351d15d1b1c279d56304cd6e4bfbf255c80a83daee1dab60f9718a68868051a"} Jul 16 12:47:30.162217 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:30.162204 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78cdc7b659-g7v2p" Jul 16 12:47:30.162639 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:30.162475 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-86skr" Jul 16 12:47:30.190121 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:30.190071 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5f9765cc5f-r7x5v" podStartSLOduration=1.7007773309999998 podStartE2EDuration="2.190057313s" totalImagesPullingTime="489.279982ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:47:30.18828573 +0000 UTC m=+508.666031349" watchObservedRunningTime="2026-07-16 12:47:30.190057313 +0000 UTC m=+508.667802942" Jul 16 12:47:30.211794 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:30.211767 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-78cdc7b659-g7v2p"] Jul 16 12:47:30.220303 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:30.220280 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-78cdc7b659-g7v2p"] Jul 16 12:47:30.247783 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:30.247719 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-86skr"] Jul 16 12:47:30.251521 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:30.251497 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-86skr"] Jul 16 12:47:31.977871 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:31.977840 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86f36673-05e6-459a-bc3b-a9dfeaa95e87" path="/var/lib/kubelet/pods/86f36673-05e6-459a-bc3b-a9dfeaa95e87/volumes" Jul 16 12:47:31.978233 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:31.978044 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="feb18302-0b5a-442d-bdb7-e6542ac3fe2e" path="/var/lib/kubelet/pods/feb18302-0b5a-442d-bdb7-e6542ac3fe2e/volumes" Jul 16 12:47:34.916971 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:34.916934 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:47:35.012263 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:35.012226 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:47:35.407934 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:35.407907 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:47:38.209465 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:38.209429 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:47:38.312006 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:38.311967 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:47:44.653097 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:44.653065 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:47:44.751301 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:47:44.751271 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:48:10.198830 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:10.198735 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:48:11.088031 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:11.087998 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:48:11.218682 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:11.218655 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:48:11.319119 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:11.319091 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:48:11.405259 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:11.405229 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:48:22.649275 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:22.649240 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:48:22.749058 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:22.749023 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:48:37.965097 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:37.965063 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:48:53.180169 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:53.180134 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-bcx57"] Jul 16 12:48:53.183441 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:53.183425 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bcx57" Jul 16 12:48:53.185913 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:53.185888 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 16 12:48:53.186032 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:53.185938 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 16 12:48:53.186899 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:53.186882 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-7jchv\"" Jul 16 12:48:53.186955 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:53.186888 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 16 12:48:53.194784 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:53.194763 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-bcx57"] Jul 16 12:48:53.271927 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:53.271898 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/971c7ce3-5be4-4a9e-920c-e1620ffbb196-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bcx57\" (UID: \"971c7ce3-5be4-4a9e-920c-e1620ffbb196\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bcx57" Jul 16 12:48:53.272093 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:53.271930 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/971c7ce3-5be4-4a9e-920c-e1620ffbb196-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bcx57\" (UID: \"971c7ce3-5be4-4a9e-920c-e1620ffbb196\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bcx57" Jul 16 12:48:53.272093 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:53.271952 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dntpb\" (UniqueName: \"kubernetes.io/projected/971c7ce3-5be4-4a9e-920c-e1620ffbb196-kube-api-access-dntpb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bcx57\" (UID: \"971c7ce3-5be4-4a9e-920c-e1620ffbb196\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bcx57" Jul 16 12:48:53.272093 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:53.272048 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/971c7ce3-5be4-4a9e-920c-e1620ffbb196-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bcx57\" (UID: \"971c7ce3-5be4-4a9e-920c-e1620ffbb196\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bcx57" Jul 16 12:48:53.272251 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:53.272096 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/971c7ce3-5be4-4a9e-920c-e1620ffbb196-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bcx57\" (UID: \"971c7ce3-5be4-4a9e-920c-e1620ffbb196\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bcx57" Jul 16 12:48:53.272251 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:53.272149 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/971c7ce3-5be4-4a9e-920c-e1620ffbb196-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bcx57\" (UID: \"971c7ce3-5be4-4a9e-920c-e1620ffbb196\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bcx57" Jul 16 12:48:53.291604 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:53.291574 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:48:53.373449 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:53.373414 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/971c7ce3-5be4-4a9e-920c-e1620ffbb196-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bcx57\" (UID: \"971c7ce3-5be4-4a9e-920c-e1620ffbb196\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bcx57" Jul 16 12:48:53.373449 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:53.373447 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/971c7ce3-5be4-4a9e-920c-e1620ffbb196-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bcx57\" (UID: \"971c7ce3-5be4-4a9e-920c-e1620ffbb196\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bcx57" Jul 16 12:48:53.373646 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:53.373466 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dntpb\" (UniqueName: \"kubernetes.io/projected/971c7ce3-5be4-4a9e-920c-e1620ffbb196-kube-api-access-dntpb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bcx57\" (UID: \"971c7ce3-5be4-4a9e-920c-e1620ffbb196\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bcx57" Jul 16 12:48:53.373646 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:53.373581 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/971c7ce3-5be4-4a9e-920c-e1620ffbb196-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bcx57\" (UID: \"971c7ce3-5be4-4a9e-920c-e1620ffbb196\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bcx57" Jul 16 12:48:53.373646 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:53.373623 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/971c7ce3-5be4-4a9e-920c-e1620ffbb196-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bcx57\" (UID: \"971c7ce3-5be4-4a9e-920c-e1620ffbb196\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bcx57" Jul 16 12:48:53.373803 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:53.373655 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/971c7ce3-5be4-4a9e-920c-e1620ffbb196-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bcx57\" (UID: \"971c7ce3-5be4-4a9e-920c-e1620ffbb196\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bcx57" Jul 16 12:48:53.373874 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:53.373855 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/971c7ce3-5be4-4a9e-920c-e1620ffbb196-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bcx57\" (UID: \"971c7ce3-5be4-4a9e-920c-e1620ffbb196\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bcx57" Jul 16 12:48:53.373965 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:53.373941 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/971c7ce3-5be4-4a9e-920c-e1620ffbb196-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bcx57\" (UID: \"971c7ce3-5be4-4a9e-920c-e1620ffbb196\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bcx57" Jul 16 12:48:53.374014 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:53.373994 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/971c7ce3-5be4-4a9e-920c-e1620ffbb196-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bcx57\" (UID: \"971c7ce3-5be4-4a9e-920c-e1620ffbb196\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bcx57" Jul 16 12:48:53.375900 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:53.375879 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/971c7ce3-5be4-4a9e-920c-e1620ffbb196-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bcx57\" (UID: \"971c7ce3-5be4-4a9e-920c-e1620ffbb196\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bcx57" Jul 16 12:48:53.376049 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:53.376033 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/971c7ce3-5be4-4a9e-920c-e1620ffbb196-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bcx57\" (UID: \"971c7ce3-5be4-4a9e-920c-e1620ffbb196\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bcx57" Jul 16 12:48:53.381439 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:53.381417 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dntpb\" (UniqueName: \"kubernetes.io/projected/971c7ce3-5be4-4a9e-920c-e1620ffbb196-kube-api-access-dntpb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bcx57\" (UID: \"971c7ce3-5be4-4a9e-920c-e1620ffbb196\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bcx57" Jul 16 12:48:53.494731 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:53.494651 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bcx57" Jul 16 12:48:53.634846 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:53.634786 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-bcx57"] Jul 16 12:48:53.637413 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:48:53.637360 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod971c7ce3_5be4_4a9e_920c_e1620ffbb196.slice/crio-4cf5cd7c272fbcf0a3eb6291877966317d94fdbefb256e3d1a521f3ece747f90 WatchSource:0}: Error finding container 4cf5cd7c272fbcf0a3eb6291877966317d94fdbefb256e3d1a521f3ece747f90: Status 404 returned error can't find the container with id 4cf5cd7c272fbcf0a3eb6291877966317d94fdbefb256e3d1a521f3ece747f90 Jul 16 12:48:54.551875 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:54.551826 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bcx57" event={"ID":"971c7ce3-5be4-4a9e-920c-e1620ffbb196","Type":"ContainerStarted","Data":"4cf5cd7c272fbcf0a3eb6291877966317d94fdbefb256e3d1a521f3ece747f90"} Jul 16 12:48:55.557662 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:55.557631 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bcx57" event={"ID":"971c7ce3-5be4-4a9e-920c-e1620ffbb196","Type":"ContainerStarted","Data":"d04db6ed33081a51a3a065fac319d4964288d5e10052e43a392469562fc8e106"} Jul 16 12:48:55.575672 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:48:55.575621 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bcx57" podStartSLOduration=1.472644999 podStartE2EDuration="2.575605164s" totalImagesPullingTime="1.102960165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:48:55.573762138 +0000 UTC m=+594.051507774" watchObservedRunningTime="2026-07-16 12:48:55.575605164 +0000 UTC m=+594.053350779" Jul 16 12:49:03.495810 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:49:03.495773 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bcx57" Jul 16 12:49:03.498185 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:49:03.495832 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bcx57" Jul 16 12:49:03.508177 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:49:03.508152 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bcx57" Jul 16 12:49:03.602164 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:49:03.602140 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bcx57" Jul 16 12:49:03.803173 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:49:03.803087 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:49:14.489186 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:49:14.489154 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:49:22.411819 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:49:22.411742 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:49:27.167441 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:49:27.167405 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:49:32.280880 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:49:32.280834 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:49:52.452622 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:49:52.452587 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:50:25.478885 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:25.478848 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-74fd59b54b-6cnkq"] Jul 16 12:50:25.481949 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:25.481930 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74fd59b54b-6cnkq" Jul 16 12:50:25.490216 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:25.490187 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74fd59b54b-6cnkq"] Jul 16 12:50:25.526684 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:25.526648 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d7b42a40-f9e4-41a0-920f-5fdba34878a8-tls-cert\") pod \"authorino-74fd59b54b-6cnkq\" (UID: \"d7b42a40-f9e4-41a0-920f-5fdba34878a8\") " pod="kuadrant-system/authorino-74fd59b54b-6cnkq" Jul 16 12:50:25.526844 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:25.526694 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndlgr\" (UniqueName: \"kubernetes.io/projected/d7b42a40-f9e4-41a0-920f-5fdba34878a8-kube-api-access-ndlgr\") pod \"authorino-74fd59b54b-6cnkq\" (UID: \"d7b42a40-f9e4-41a0-920f-5fdba34878a8\") " pod="kuadrant-system/authorino-74fd59b54b-6cnkq" Jul 16 12:50:25.627848 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:25.627808 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d7b42a40-f9e4-41a0-920f-5fdba34878a8-tls-cert\") pod \"authorino-74fd59b54b-6cnkq\" (UID: \"d7b42a40-f9e4-41a0-920f-5fdba34878a8\") " pod="kuadrant-system/authorino-74fd59b54b-6cnkq" Jul 16 12:50:25.628006 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:25.627856 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndlgr\" (UniqueName: \"kubernetes.io/projected/d7b42a40-f9e4-41a0-920f-5fdba34878a8-kube-api-access-ndlgr\") pod \"authorino-74fd59b54b-6cnkq\" (UID: \"d7b42a40-f9e4-41a0-920f-5fdba34878a8\") " pod="kuadrant-system/authorino-74fd59b54b-6cnkq" Jul 16 12:50:25.630308 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:25.630285 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d7b42a40-f9e4-41a0-920f-5fdba34878a8-tls-cert\") pod \"authorino-74fd59b54b-6cnkq\" (UID: \"d7b42a40-f9e4-41a0-920f-5fdba34878a8\") " pod="kuadrant-system/authorino-74fd59b54b-6cnkq" Jul 16 12:50:25.635178 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:25.635149 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndlgr\" (UniqueName: \"kubernetes.io/projected/d7b42a40-f9e4-41a0-920f-5fdba34878a8-kube-api-access-ndlgr\") pod \"authorino-74fd59b54b-6cnkq\" (UID: \"d7b42a40-f9e4-41a0-920f-5fdba34878a8\") " pod="kuadrant-system/authorino-74fd59b54b-6cnkq" Jul 16 12:50:25.712924 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:25.712892 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:50:25.790977 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:25.790897 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74fd59b54b-6cnkq" Jul 16 12:50:25.922890 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:25.922865 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74fd59b54b-6cnkq"] Jul 16 12:50:25.924725 ip-10-0-140-130 kubenswrapper[2582]: W0716 12:50:25.924699 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7b42a40_f9e4_41a0_920f_5fdba34878a8.slice/crio-677593902b678c43b146d732289f118158fc09c05e9a648e38c31c4de66f49ea WatchSource:0}: Error finding container 677593902b678c43b146d732289f118158fc09c05e9a648e38c31c4de66f49ea: Status 404 returned error can't find the container with id 677593902b678c43b146d732289f118158fc09c05e9a648e38c31c4de66f49ea Jul 16 12:50:25.925934 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:25.925918 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 12:50:25.966103 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:25.966069 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74fd59b54b-6cnkq" event={"ID":"d7b42a40-f9e4-41a0-920f-5fdba34878a8","Type":"ContainerStarted","Data":"677593902b678c43b146d732289f118158fc09c05e9a648e38c31c4de66f49ea"} Jul 16 12:50:27.290704 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:27.290566 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:50:27.662084 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:27.662049 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:50:27.759061 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:27.759021 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:50:27.979266 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:27.979175 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74fd59b54b-6cnkq" event={"ID":"d7b42a40-f9e4-41a0-920f-5fdba34878a8","Type":"ContainerStarted","Data":"4518b99ffba8b945a05b0937ec490247de9294ba1f167843e1c576ea43583ce9"} Jul 16 12:50:27.996335 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:27.996283 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-74fd59b54b-6cnkq" podStartSLOduration=1.888390827 podStartE2EDuration="2.996267003s" totalImagesPullingTime="1.107876176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:50:27.993843951 +0000 UTC m=+686.471589569" watchObservedRunningTime="2026-07-16 12:50:27.996267003 +0000 UTC m=+686.474012619" Jul 16 12:50:28.022537 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:28.022497 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f9765cc5f-r7x5v"] Jul 16 12:50:28.022722 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:28.022697 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5f9765cc5f-r7x5v" podUID="6df4f150-dc08-445c-9f36-1c65aa0363b5" containerName="authorino" containerID="cri-o://d351d15d1b1c279d56304cd6e4bfbf255c80a83daee1dab60f9718a68868051a" gracePeriod=30 Jul 16 12:50:28.265543 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:28.265520 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f9765cc5f-r7x5v" Jul 16 12:50:28.355737 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:28.355705 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgpgh\" (UniqueName: \"kubernetes.io/projected/6df4f150-dc08-445c-9f36-1c65aa0363b5-kube-api-access-cgpgh\") pod \"6df4f150-dc08-445c-9f36-1c65aa0363b5\" (UID: \"6df4f150-dc08-445c-9f36-1c65aa0363b5\") " Jul 16 12:50:28.356090 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:28.355781 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6df4f150-dc08-445c-9f36-1c65aa0363b5-tls-cert\") pod \"6df4f150-dc08-445c-9f36-1c65aa0363b5\" (UID: \"6df4f150-dc08-445c-9f36-1c65aa0363b5\") " Jul 16 12:50:28.357967 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:28.357938 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df4f150-dc08-445c-9f36-1c65aa0363b5-kube-api-access-cgpgh" (OuterVolumeSpecName: "kube-api-access-cgpgh") pod "6df4f150-dc08-445c-9f36-1c65aa0363b5" (UID: "6df4f150-dc08-445c-9f36-1c65aa0363b5"). InnerVolumeSpecName "kube-api-access-cgpgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:50:28.365515 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:28.365495 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6df4f150-dc08-445c-9f36-1c65aa0363b5-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "6df4f150-dc08-445c-9f36-1c65aa0363b5" (UID: "6df4f150-dc08-445c-9f36-1c65aa0363b5"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:50:28.456791 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:28.456748 2582 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6df4f150-dc08-445c-9f36-1c65aa0363b5-tls-cert\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:50:28.456791 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:28.456790 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgpgh\" (UniqueName: \"kubernetes.io/projected/6df4f150-dc08-445c-9f36-1c65aa0363b5-kube-api-access-cgpgh\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 12:50:28.985585 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:28.985545 2582 generic.go:356] "Generic (PLEG): container finished" podID="6df4f150-dc08-445c-9f36-1c65aa0363b5" containerID="d351d15d1b1c279d56304cd6e4bfbf255c80a83daee1dab60f9718a68868051a" exitCode=0 Jul 16 12:50:28.985771 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:28.985613 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f9765cc5f-r7x5v" Jul 16 12:50:28.985771 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:28.985638 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f9765cc5f-r7x5v" event={"ID":"6df4f150-dc08-445c-9f36-1c65aa0363b5","Type":"ContainerDied","Data":"d351d15d1b1c279d56304cd6e4bfbf255c80a83daee1dab60f9718a68868051a"} Jul 16 12:50:28.985771 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:28.985681 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f9765cc5f-r7x5v" event={"ID":"6df4f150-dc08-445c-9f36-1c65aa0363b5","Type":"ContainerDied","Data":"474d3a04cd214a9867ee2d82a07b26133b8dc6ae5e29cda607aa8ed92ff09f01"} Jul 16 12:50:28.985771 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:28.985697 2582 scope.go:117] "RemoveContainer" containerID="d351d15d1b1c279d56304cd6e4bfbf255c80a83daee1dab60f9718a68868051a" Jul 16 12:50:28.995122 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:28.995103 2582 scope.go:117] "RemoveContainer" containerID="d351d15d1b1c279d56304cd6e4bfbf255c80a83daee1dab60f9718a68868051a" Jul 16 12:50:28.995410 ip-10-0-140-130 kubenswrapper[2582]: E0716 12:50:28.995370 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d351d15d1b1c279d56304cd6e4bfbf255c80a83daee1dab60f9718a68868051a\": container with ID starting with d351d15d1b1c279d56304cd6e4bfbf255c80a83daee1dab60f9718a68868051a not found: ID does not exist" containerID="d351d15d1b1c279d56304cd6e4bfbf255c80a83daee1dab60f9718a68868051a" Jul 16 12:50:28.995481 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:28.995435 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d351d15d1b1c279d56304cd6e4bfbf255c80a83daee1dab60f9718a68868051a"} err="failed to get container status \"d351d15d1b1c279d56304cd6e4bfbf255c80a83daee1dab60f9718a68868051a\": rpc error: code = NotFound desc = could not find container \"d351d15d1b1c279d56304cd6e4bfbf255c80a83daee1dab60f9718a68868051a\": container with ID starting with d351d15d1b1c279d56304cd6e4bfbf255c80a83daee1dab60f9718a68868051a not found: ID does not exist" Jul 16 12:50:29.009281 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:29.009258 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f9765cc5f-r7x5v"] Jul 16 12:50:29.013221 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:29.013197 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5f9765cc5f-r7x5v"] Jul 16 12:50:29.977290 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:50:29.977260 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6df4f150-dc08-445c-9f36-1c65aa0363b5" path="/var/lib/kubelet/pods/6df4f150-dc08-445c-9f36-1c65aa0363b5/volumes" Jul 16 12:52:41.460283 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:52:41.460249 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:52:42.691348 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:52:42.691309 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:53:02.700948 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:53:02.700914 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:53:20.750287 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:53:20.750254 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:53:20.879974 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:53:20.879940 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:53:21.010770 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:53:21.010690 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:53:28.132786 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:53:28.132749 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:53:28.239544 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:53:28.239503 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:53:28.337930 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:53:28.337899 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:53:28.526629 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:53:28.526540 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:53:34.699819 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:53:34.699785 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:53:34.799323 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:53:34.799297 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:54:02.009142 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:54:02.009105 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:54:02.158671 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:54:02.158639 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:55:25.562727 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:55:25.562640 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:56:03.904377 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:56:03.904317 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 12:59:38.683832 ip-10-0-140-130 kubenswrapper[2582]: I0716 12:59:38.683794 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:01:02.421618 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:01:02.421566 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:06.061215 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:06.061182 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:23.136952 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:23.136858 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:23.319670 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:23.319628 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:27.696312 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:27.696278 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:28.961107 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:28.961076 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:29.064749 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.064718 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:29.137068 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.136891 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt"] Jul 16 13:04:29.143033 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.141492 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6df4f150-dc08-445c-9f36-1c65aa0363b5" containerName="authorino" Jul 16 13:04:29.143033 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.141529 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6df4f150-dc08-445c-9f36-1c65aa0363b5" containerName="authorino" Jul 16 13:04:29.143033 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.141703 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6df4f150-dc08-445c-9f36-1c65aa0363b5" containerName="authorino" Jul 16 13:04:29.145285 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.145262 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:29.147700 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.147674 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-60a47609-openshift-default-dockercfg-7fst7\"" Jul 16 13:04:29.160127 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.160106 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt"] Jul 16 13:04:29.211503 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.211382 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cb490fe-ff70-495f-b51d-1e86e8882d0b-istiod-ca-cert\") pod \"e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:29.211503 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.211481 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cb490fe-ff70-495f-b51d-1e86e8882d0b-istio-data\") pod \"e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:29.211744 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.211511 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cb490fe-ff70-495f-b51d-1e86e8882d0b-istio-token\") pod \"e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:29.211744 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.211537 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cb490fe-ff70-495f-b51d-1e86e8882d0b-workload-certs\") pod \"e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:29.211744 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.211592 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cb490fe-ff70-495f-b51d-1e86e8882d0b-istio-podinfo\") pod \"e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:29.211744 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.211616 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb490fe-ff70-495f-b51d-1e86e8882d0b-workload-socket\") pod \"e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:29.211744 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.211643 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cb490fe-ff70-495f-b51d-1e86e8882d0b-istio-envoy\") pod \"e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:29.211744 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.211662 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b88kn\" (UniqueName: \"kubernetes.io/projected/5cb490fe-ff70-495f-b51d-1e86e8882d0b-kube-api-access-b88kn\") pod \"e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:29.211744 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.211684 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb490fe-ff70-495f-b51d-1e86e8882d0b-credential-socket\") pod \"e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:29.246839 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.246801 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:29.312305 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.312269 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cb490fe-ff70-495f-b51d-1e86e8882d0b-istio-token\") pod \"e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:29.312305 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.312309 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cb490fe-ff70-495f-b51d-1e86e8882d0b-workload-certs\") pod \"e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:29.312591 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.312333 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cb490fe-ff70-495f-b51d-1e86e8882d0b-istio-podinfo\") pod \"e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:29.312591 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.312354 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb490fe-ff70-495f-b51d-1e86e8882d0b-workload-socket\") pod \"e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:29.312591 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.312376 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cb490fe-ff70-495f-b51d-1e86e8882d0b-istio-envoy\") pod \"e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:29.312591 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.312418 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b88kn\" (UniqueName: \"kubernetes.io/projected/5cb490fe-ff70-495f-b51d-1e86e8882d0b-kube-api-access-b88kn\") pod \"e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:29.312591 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.312440 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb490fe-ff70-495f-b51d-1e86e8882d0b-credential-socket\") pod \"e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:29.312591 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.312466 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cb490fe-ff70-495f-b51d-1e86e8882d0b-istiod-ca-cert\") pod \"e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:29.312898 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.312605 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cb490fe-ff70-495f-b51d-1e86e8882d0b-istio-data\") pod \"e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:29.312898 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.312787 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb490fe-ff70-495f-b51d-1e86e8882d0b-workload-socket\") pod \"e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:29.312898 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.312891 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb490fe-ff70-495f-b51d-1e86e8882d0b-credential-socket\") pod \"e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:29.313072 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.312910 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cb490fe-ff70-495f-b51d-1e86e8882d0b-istio-data\") pod \"e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:29.313072 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.312950 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cb490fe-ff70-495f-b51d-1e86e8882d0b-workload-certs\") pod \"e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:29.313328 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.313311 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cb490fe-ff70-495f-b51d-1e86e8882d0b-istiod-ca-cert\") pod \"e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:29.314978 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.314955 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cb490fe-ff70-495f-b51d-1e86e8882d0b-istio-envoy\") pod \"e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:29.315080 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.315036 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cb490fe-ff70-495f-b51d-1e86e8882d0b-istio-podinfo\") pod \"e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:29.322187 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.322165 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cb490fe-ff70-495f-b51d-1e86e8882d0b-istio-token\") pod \"e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:29.323257 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.323237 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b88kn\" (UniqueName: \"kubernetes.io/projected/5cb490fe-ff70-495f-b51d-1e86e8882d0b-kube-api-access-b88kn\") pod \"e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:29.357807 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.357751 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:29.454997 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.454958 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:29.516490 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.516460 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:29.597232 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.597197 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt"] Jul 16 13:04:29.598567 ip-10-0-140-130 kubenswrapper[2582]: W0716 13:04:29.598520 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cb490fe_ff70_495f_b51d_1e86e8882d0b.slice/crio-d3ad6292d1c5cdb3dcd11985eea91dbfbda407457dc50d5120184c50e3eb807f WatchSource:0}: Error finding container d3ad6292d1c5cdb3dcd11985eea91dbfbda407457dc50d5120184c50e3eb807f: Status 404 returned error can't find the container with id d3ad6292d1c5cdb3dcd11985eea91dbfbda407457dc50d5120184c50e3eb807f Jul 16 13:04:29.600758 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.600725 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:04:29.600863 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.600806 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:04:29.600863 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.600852 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:04:29.844127 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.844034 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" event={"ID":"5cb490fe-ff70-495f-b51d-1e86e8882d0b","Type":"ContainerStarted","Data":"8cf9f6b1d763400b566ca99bf816265d2dc96576a31725b388a1225f55d64aa5"} Jul 16 13:04:29.844127 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.844096 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" event={"ID":"5cb490fe-ff70-495f-b51d-1e86e8882d0b","Type":"ContainerStarted","Data":"d3ad6292d1c5cdb3dcd11985eea91dbfbda407457dc50d5120184c50e3eb807f"} Jul 16 13:04:29.866673 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.866621 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" podStartSLOduration=0.866602247 podStartE2EDuration="866.602247ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:04:29.865670593 +0000 UTC m=+1528.343416254" watchObservedRunningTime="2026-07-16 13:04:29.866602247 +0000 UTC m=+1528.344347855" Jul 16 13:04:29.878796 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:29.878771 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:30.092182 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:30.092134 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:30.206194 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:30.206160 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:30.456276 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:30.456192 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:30.460938 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:30.460914 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:30.848780 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:30.848702 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:30.849601 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:30.849581 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:33.659091 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:33.659041 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:35.450692 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:35.450650 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt"] Jul 16 13:04:35.451093 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:35.450883 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" podUID="5cb490fe-ff70-495f-b51d-1e86e8882d0b" containerName="istio-proxy" containerID="cri-o://8cf9f6b1d763400b566ca99bf816265d2dc96576a31725b388a1225f55d64aa5" gracePeriod=30 Jul 16 13:04:35.476676 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:35.476641 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:35.576836 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:35.576802 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:37.918415 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:37.917844 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:41.395499 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.395475 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:41.399422 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.399375 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:41.501103 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.501010 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:41.519039 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.519014 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cb490fe-ff70-495f-b51d-1e86e8882d0b-istio-token\") pod \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " Jul 16 13:04:41.519122 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.519058 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cb490fe-ff70-495f-b51d-1e86e8882d0b-istio-data\") pod \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " Jul 16 13:04:41.519122 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.519080 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb490fe-ff70-495f-b51d-1e86e8882d0b-credential-socket\") pod \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " Jul 16 13:04:41.519122 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.519105 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cb490fe-ff70-495f-b51d-1e86e8882d0b-workload-certs\") pod \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " Jul 16 13:04:41.519278 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.519134 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cb490fe-ff70-495f-b51d-1e86e8882d0b-istio-envoy\") pod \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " Jul 16 13:04:41.519278 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.519157 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb490fe-ff70-495f-b51d-1e86e8882d0b-workload-socket\") pod \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " Jul 16 13:04:41.519278 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.519178 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cb490fe-ff70-495f-b51d-1e86e8882d0b-istiod-ca-cert\") pod \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " Jul 16 13:04:41.519278 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.519233 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b88kn\" (UniqueName: \"kubernetes.io/projected/5cb490fe-ff70-495f-b51d-1e86e8882d0b-kube-api-access-b88kn\") pod \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " Jul 16 13:04:41.519278 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.519265 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cb490fe-ff70-495f-b51d-1e86e8882d0b-istio-podinfo\") pod \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\" (UID: \"5cb490fe-ff70-495f-b51d-1e86e8882d0b\") " Jul 16 13:04:41.519567 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.519460 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb490fe-ff70-495f-b51d-1e86e8882d0b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5cb490fe-ff70-495f-b51d-1e86e8882d0b" (UID: "5cb490fe-ff70-495f-b51d-1e86e8882d0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:04:41.519695 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.519666 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb490fe-ff70-495f-b51d-1e86e8882d0b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5cb490fe-ff70-495f-b51d-1e86e8882d0b" (UID: "5cb490fe-ff70-495f-b51d-1e86e8882d0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:04:41.519752 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.519682 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cb490fe-ff70-495f-b51d-1e86e8882d0b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5cb490fe-ff70-495f-b51d-1e86e8882d0b" (UID: "5cb490fe-ff70-495f-b51d-1e86e8882d0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:04:41.519870 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.519830 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb490fe-ff70-495f-b51d-1e86e8882d0b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5cb490fe-ff70-495f-b51d-1e86e8882d0b" (UID: "5cb490fe-ff70-495f-b51d-1e86e8882d0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:04:41.520000 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.519935 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb490fe-ff70-495f-b51d-1e86e8882d0b-istio-data" (OuterVolumeSpecName: "istio-data") pod "5cb490fe-ff70-495f-b51d-1e86e8882d0b" (UID: "5cb490fe-ff70-495f-b51d-1e86e8882d0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:04:41.521674 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.521641 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cb490fe-ff70-495f-b51d-1e86e8882d0b-istio-token" (OuterVolumeSpecName: "istio-token") pod "5cb490fe-ff70-495f-b51d-1e86e8882d0b" (UID: "5cb490fe-ff70-495f-b51d-1e86e8882d0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:04:41.521674 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.521659 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb490fe-ff70-495f-b51d-1e86e8882d0b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5cb490fe-ff70-495f-b51d-1e86e8882d0b" (UID: "5cb490fe-ff70-495f-b51d-1e86e8882d0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:04:41.521806 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.521720 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cb490fe-ff70-495f-b51d-1e86e8882d0b-kube-api-access-b88kn" (OuterVolumeSpecName: "kube-api-access-b88kn") pod "5cb490fe-ff70-495f-b51d-1e86e8882d0b" (UID: "5cb490fe-ff70-495f-b51d-1e86e8882d0b"). InnerVolumeSpecName "kube-api-access-b88kn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:04:41.521806 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.521756 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5cb490fe-ff70-495f-b51d-1e86e8882d0b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5cb490fe-ff70-495f-b51d-1e86e8882d0b" (UID: "5cb490fe-ff70-495f-b51d-1e86e8882d0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 13:04:41.603128 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.603091 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:41.620317 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.620276 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cb490fe-ff70-495f-b51d-1e86e8882d0b-istio-envoy\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:04:41.620317 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.620314 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb490fe-ff70-495f-b51d-1e86e8882d0b-workload-socket\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:04:41.620514 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.620325 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cb490fe-ff70-495f-b51d-1e86e8882d0b-istiod-ca-cert\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:04:41.620514 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.620335 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b88kn\" (UniqueName: \"kubernetes.io/projected/5cb490fe-ff70-495f-b51d-1e86e8882d0b-kube-api-access-b88kn\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:04:41.620514 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.620345 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cb490fe-ff70-495f-b51d-1e86e8882d0b-istio-podinfo\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:04:41.620514 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.620354 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cb490fe-ff70-495f-b51d-1e86e8882d0b-istio-token\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:04:41.620514 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.620362 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cb490fe-ff70-495f-b51d-1e86e8882d0b-istio-data\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:04:41.620514 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.620370 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb490fe-ff70-495f-b51d-1e86e8882d0b-credential-socket\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:04:41.620514 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.620378 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cb490fe-ff70-495f-b51d-1e86e8882d0b-workload-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:04:41.700882 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.700844 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:41.744718 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.744684 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz"] Jul 16 13:04:41.745125 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.745108 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cb490fe-ff70-495f-b51d-1e86e8882d0b" containerName="istio-proxy" Jul 16 13:04:41.745222 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.745126 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cb490fe-ff70-495f-b51d-1e86e8882d0b" containerName="istio-proxy" Jul 16 13:04:41.745284 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.745222 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cb490fe-ff70-495f-b51d-1e86e8882d0b" containerName="istio-proxy" Jul 16 13:04:41.748028 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.748004 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:41.750349 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.750329 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-5a470a1b-gw-openshift-default-dockercfg-7t8ld\"" Jul 16 13:04:41.759649 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.759625 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz"] Jul 16 13:04:41.781166 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.781139 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:41.822213 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.822183 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f634cef4-1591-4051-922f-128ed55842d3-istio-token\") pod \"e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:41.822354 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.822217 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f634cef4-1591-4051-922f-128ed55842d3-istio-podinfo\") pod \"e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:41.822354 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.822244 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f634cef4-1591-4051-922f-128ed55842d3-workload-socket\") pod \"e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:41.822354 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.822324 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f634cef4-1591-4051-922f-128ed55842d3-workload-certs\") pod \"e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:41.822583 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.822357 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f634cef4-1591-4051-922f-128ed55842d3-credential-socket\") pod \"e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:41.822583 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.822374 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f634cef4-1591-4051-922f-128ed55842d3-istio-data\") pod \"e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:41.822583 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.822422 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f634cef4-1591-4051-922f-128ed55842d3-istiod-ca-cert\") pod \"e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:41.822583 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.822442 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn78f\" (UniqueName: \"kubernetes.io/projected/f634cef4-1591-4051-922f-128ed55842d3-kube-api-access-dn78f\") pod \"e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:41.822583 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.822468 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f634cef4-1591-4051-922f-128ed55842d3-istio-envoy\") pod \"e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:41.862095 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.862063 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:41.904001 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.903973 2582 generic.go:356] "Generic (PLEG): container finished" podID="5cb490fe-ff70-495f-b51d-1e86e8882d0b" containerID="8cf9f6b1d763400b566ca99bf816265d2dc96576a31725b388a1225f55d64aa5" exitCode=0 Jul 16 13:04:41.904138 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.904005 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" event={"ID":"5cb490fe-ff70-495f-b51d-1e86e8882d0b","Type":"ContainerDied","Data":"8cf9f6b1d763400b566ca99bf816265d2dc96576a31725b388a1225f55d64aa5"} Jul 16 13:04:41.904138 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.904021 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" Jul 16 13:04:41.904138 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.904059 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt" event={"ID":"5cb490fe-ff70-495f-b51d-1e86e8882d0b","Type":"ContainerDied","Data":"d3ad6292d1c5cdb3dcd11985eea91dbfbda407457dc50d5120184c50e3eb807f"} Jul 16 13:04:41.904138 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.904081 2582 scope.go:117] "RemoveContainer" containerID="8cf9f6b1d763400b566ca99bf816265d2dc96576a31725b388a1225f55d64aa5" Jul 16 13:04:41.913760 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.913745 2582 scope.go:117] "RemoveContainer" containerID="8cf9f6b1d763400b566ca99bf816265d2dc96576a31725b388a1225f55d64aa5" Jul 16 13:04:41.913986 ip-10-0-140-130 kubenswrapper[2582]: E0716 13:04:41.913968 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cf9f6b1d763400b566ca99bf816265d2dc96576a31725b388a1225f55d64aa5\": container with ID starting with 8cf9f6b1d763400b566ca99bf816265d2dc96576a31725b388a1225f55d64aa5 not found: ID does not exist" containerID="8cf9f6b1d763400b566ca99bf816265d2dc96576a31725b388a1225f55d64aa5" Jul 16 13:04:41.914030 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.913994 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cf9f6b1d763400b566ca99bf816265d2dc96576a31725b388a1225f55d64aa5"} err="failed to get container status \"8cf9f6b1d763400b566ca99bf816265d2dc96576a31725b388a1225f55d64aa5\": rpc error: code = NotFound desc = could not find container \"8cf9f6b1d763400b566ca99bf816265d2dc96576a31725b388a1225f55d64aa5\": container with ID starting with 8cf9f6b1d763400b566ca99bf816265d2dc96576a31725b388a1225f55d64aa5 not found: ID does not exist" Jul 16 13:04:41.922996 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.922977 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f634cef4-1591-4051-922f-128ed55842d3-workload-socket\") pod \"e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:41.923107 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.923022 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f634cef4-1591-4051-922f-128ed55842d3-workload-certs\") pod \"e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:41.923162 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.923147 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f634cef4-1591-4051-922f-128ed55842d3-credential-socket\") pod \"e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:41.923213 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.923174 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f634cef4-1591-4051-922f-128ed55842d3-istio-data\") pod \"e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:41.923213 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.923205 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f634cef4-1591-4051-922f-128ed55842d3-istiod-ca-cert\") pod \"e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:41.923343 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.923315 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn78f\" (UniqueName: \"kubernetes.io/projected/f634cef4-1591-4051-922f-128ed55842d3-kube-api-access-dn78f\") pod \"e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:41.923438 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.923403 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f634cef4-1591-4051-922f-128ed55842d3-workload-certs\") pod \"e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:41.923438 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.923416 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f634cef4-1591-4051-922f-128ed55842d3-istio-envoy\") pod \"e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:41.923562 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.923491 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f634cef4-1591-4051-922f-128ed55842d3-istio-token\") pod \"e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:41.923562 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.923514 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f634cef4-1591-4051-922f-128ed55842d3-istio-podinfo\") pod \"e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:41.923673 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.923624 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f634cef4-1591-4051-922f-128ed55842d3-workload-socket\") pod \"e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:41.923727 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.923667 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f634cef4-1591-4051-922f-128ed55842d3-credential-socket\") pod \"e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:41.923727 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.923692 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f634cef4-1591-4051-922f-128ed55842d3-istio-data\") pod \"e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:41.924050 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.924030 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f634cef4-1591-4051-922f-128ed55842d3-istiod-ca-cert\") pod \"e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:41.925944 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.925926 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f634cef4-1591-4051-922f-128ed55842d3-istio-podinfo\") pod \"e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:41.926025 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.925964 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt"] Jul 16 13:04:41.926025 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.926000 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f634cef4-1591-4051-922f-128ed55842d3-istio-envoy\") pod \"e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:41.930627 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.930605 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-60a47609-openshift-default-6cf89b7669-p6fzt"] Jul 16 13:04:41.931767 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.931749 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn78f\" (UniqueName: \"kubernetes.io/projected/f634cef4-1591-4051-922f-128ed55842d3-kube-api-access-dn78f\") pod \"e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:41.932215 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.932195 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f634cef4-1591-4051-922f-128ed55842d3-istio-token\") pod \"e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:41.977683 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:41.977658 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cb490fe-ff70-495f-b51d-1e86e8882d0b" path="/var/lib/kubelet/pods/5cb490fe-ff70-495f-b51d-1e86e8882d0b/volumes" Jul 16 13:04:42.000172 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:42.000147 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:42.059045 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:42.058985 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:42.123826 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:42.123772 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:42.199209 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:42.199187 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz"] Jul 16 13:04:42.201473 ip-10-0-140-130 kubenswrapper[2582]: W0716 13:04:42.201449 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf634cef4_1591_4051_922f_128ed55842d3.slice/crio-53c1e0a0e3b0e7503e9a4a04fbfe5fe085456c146858fcbff166efce8e5fa45d WatchSource:0}: Error finding container 53c1e0a0e3b0e7503e9a4a04fbfe5fe085456c146858fcbff166efce8e5fa45d: Status 404 returned error can't find the container with id 53c1e0a0e3b0e7503e9a4a04fbfe5fe085456c146858fcbff166efce8e5fa45d Jul 16 13:04:42.203482 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:42.203450 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:04:42.203588 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:42.203511 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:04:42.203588 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:42.203538 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:04:42.701592 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:42.701553 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:42.800824 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:42.800760 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:42.910494 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:42.910461 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" event={"ID":"f634cef4-1591-4051-922f-128ed55842d3","Type":"ContainerStarted","Data":"74652b283f61171b3fb777808018c4c05d8112f401989461cd1f0a31117cfb79"} Jul 16 13:04:42.910673 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:42.910512 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" event={"ID":"f634cef4-1591-4051-922f-128ed55842d3","Type":"ContainerStarted","Data":"53c1e0a0e3b0e7503e9a4a04fbfe5fe085456c146858fcbff166efce8e5fa45d"} Jul 16 13:04:42.930300 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:42.930253 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" podStartSLOduration=1.930238905 podStartE2EDuration="1.930238905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:04:42.929661084 +0000 UTC m=+1541.407406718" watchObservedRunningTime="2026-07-16 13:04:42.930238905 +0000 UTC m=+1541.407984520" Jul 16 13:04:43.059997 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:43.059910 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:43.064748 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:43.064722 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:43.255504 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:43.255468 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:43.915330 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:43.915294 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:43.916307 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:43.916288 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:45.373125 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:45.373095 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:47.636095 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:47.636057 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz"] Jul 16 13:04:47.636565 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:47.636275 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" podUID="f634cef4-1591-4051-922f-128ed55842d3" containerName="istio-proxy" containerID="cri-o://74652b283f61171b3fb777808018c4c05d8112f401989461cd1f0a31117cfb79" gracePeriod=30 Jul 16 13:04:47.662064 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:47.662034 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:47.763101 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:47.763069 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:49.264424 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:49.264377 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:53.687627 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:53.687595 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:53.788140 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:53.788113 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:53.871012 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:53.868598 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w"] Jul 16 13:04:53.872222 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:53.872195 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:04:53.875004 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:53.874982 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-b0d8b7c2-openshift-default-dockercfg-psbtj\"" Jul 16 13:04:53.882889 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:53.882867 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w"] Jul 16 13:04:53.966084 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:53.966049 2582 generic.go:356] "Generic (PLEG): container finished" podID="f634cef4-1591-4051-922f-128ed55842d3" containerID="74652b283f61171b3fb777808018c4c05d8112f401989461cd1f0a31117cfb79" exitCode=0 Jul 16 13:04:53.966279 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:53.966111 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" event={"ID":"f634cef4-1591-4051-922f-128ed55842d3","Type":"ContainerDied","Data":"74652b283f61171b3fb777808018c4c05d8112f401989461cd1f0a31117cfb79"} Jul 16 13:04:54.039981 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.039908 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f767df7e-814d-4f2f-bd4f-600c6addbe2b-istio-podinfo\") pod \"e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:04:54.039981 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.039945 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f767df7e-814d-4f2f-bd4f-600c6addbe2b-istio-envoy\") pod \"e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:04:54.040155 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.039984 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f767df7e-814d-4f2f-bd4f-600c6addbe2b-istio-token\") pod \"e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:04:54.040155 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.040034 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f767df7e-814d-4f2f-bd4f-600c6addbe2b-credential-socket\") pod \"e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:04:54.040155 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.040072 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f767df7e-814d-4f2f-bd4f-600c6addbe2b-istiod-ca-cert\") pod \"e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:04:54.040155 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.040115 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f767df7e-814d-4f2f-bd4f-600c6addbe2b-workload-certs\") pod \"e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:04:54.040155 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.040139 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjxrm\" (UniqueName: \"kubernetes.io/projected/f767df7e-814d-4f2f-bd4f-600c6addbe2b-kube-api-access-tjxrm\") pod \"e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:04:54.040350 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.040177 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f767df7e-814d-4f2f-bd4f-600c6addbe2b-istio-data\") pod \"e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:04:54.040350 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.040203 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f767df7e-814d-4f2f-bd4f-600c6addbe2b-workload-socket\") pod \"e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:04:54.077598 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.077574 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:54.141057 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.141016 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f767df7e-814d-4f2f-bd4f-600c6addbe2b-istio-data\") pod \"e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:04:54.141228 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.141064 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f767df7e-814d-4f2f-bd4f-600c6addbe2b-workload-socket\") pod \"e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:04:54.141228 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.141098 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f767df7e-814d-4f2f-bd4f-600c6addbe2b-istio-podinfo\") pod \"e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:04:54.141228 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.141119 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f767df7e-814d-4f2f-bd4f-600c6addbe2b-istio-envoy\") pod \"e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:04:54.141228 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.141156 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f767df7e-814d-4f2f-bd4f-600c6addbe2b-istio-token\") pod \"e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:04:54.141228 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.141195 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f767df7e-814d-4f2f-bd4f-600c6addbe2b-credential-socket\") pod \"e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:04:54.141228 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.141219 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f767df7e-814d-4f2f-bd4f-600c6addbe2b-istiod-ca-cert\") pod \"e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:04:54.141582 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.141281 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f767df7e-814d-4f2f-bd4f-600c6addbe2b-workload-certs\") pod \"e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:04:54.141582 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.141311 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjxrm\" (UniqueName: \"kubernetes.io/projected/f767df7e-814d-4f2f-bd4f-600c6addbe2b-kube-api-access-tjxrm\") pod \"e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:04:54.141582 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.141507 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f767df7e-814d-4f2f-bd4f-600c6addbe2b-istio-data\") pod \"e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:04:54.141719 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.141578 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f767df7e-814d-4f2f-bd4f-600c6addbe2b-workload-socket\") pod \"e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:04:54.141719 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.141624 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f767df7e-814d-4f2f-bd4f-600c6addbe2b-credential-socket\") pod \"e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:04:54.141792 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.141757 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f767df7e-814d-4f2f-bd4f-600c6addbe2b-workload-certs\") pod \"e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:04:54.142039 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.142023 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f767df7e-814d-4f2f-bd4f-600c6addbe2b-istiod-ca-cert\") pod \"e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:04:54.144097 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.144070 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f767df7e-814d-4f2f-bd4f-600c6addbe2b-istio-envoy\") pod \"e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:04:54.144232 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.144209 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f767df7e-814d-4f2f-bd4f-600c6addbe2b-istio-podinfo\") pod \"e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:04:54.149716 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.149692 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f767df7e-814d-4f2f-bd4f-600c6addbe2b-istio-token\") pod \"e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:04:54.149863 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.149839 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjxrm\" (UniqueName: \"kubernetes.io/projected/f767df7e-814d-4f2f-bd4f-600c6addbe2b-kube-api-access-tjxrm\") pod \"e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:04:54.191947 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.191921 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:04:54.242468 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.242437 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f634cef4-1591-4051-922f-128ed55842d3-istio-envoy\") pod \"f634cef4-1591-4051-922f-128ed55842d3\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " Jul 16 13:04:54.242603 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.242496 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f634cef4-1591-4051-922f-128ed55842d3-credential-socket\") pod \"f634cef4-1591-4051-922f-128ed55842d3\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " Jul 16 13:04:54.242603 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.242536 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f634cef4-1591-4051-922f-128ed55842d3-istio-token\") pod \"f634cef4-1591-4051-922f-128ed55842d3\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " Jul 16 13:04:54.242603 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.242598 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f634cef4-1591-4051-922f-128ed55842d3-istio-podinfo\") pod \"f634cef4-1591-4051-922f-128ed55842d3\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " Jul 16 13:04:54.242780 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.242638 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f634cef4-1591-4051-922f-128ed55842d3-istiod-ca-cert\") pod \"f634cef4-1591-4051-922f-128ed55842d3\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " Jul 16 13:04:54.242780 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.242678 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f634cef4-1591-4051-922f-128ed55842d3-workload-socket\") pod \"f634cef4-1591-4051-922f-128ed55842d3\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " Jul 16 13:04:54.242780 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.242700 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f634cef4-1591-4051-922f-128ed55842d3-istio-data\") pod \"f634cef4-1591-4051-922f-128ed55842d3\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " Jul 16 13:04:54.242780 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.242741 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f634cef4-1591-4051-922f-128ed55842d3-workload-certs\") pod \"f634cef4-1591-4051-922f-128ed55842d3\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " Jul 16 13:04:54.242780 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.242764 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dn78f\" (UniqueName: \"kubernetes.io/projected/f634cef4-1591-4051-922f-128ed55842d3-kube-api-access-dn78f\") pod \"f634cef4-1591-4051-922f-128ed55842d3\" (UID: \"f634cef4-1591-4051-922f-128ed55842d3\") " Jul 16 13:04:54.243675 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.243328 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f634cef4-1591-4051-922f-128ed55842d3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f634cef4-1591-4051-922f-128ed55842d3" (UID: "f634cef4-1591-4051-922f-128ed55842d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:04:54.243675 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.243541 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f634cef4-1591-4051-922f-128ed55842d3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f634cef4-1591-4051-922f-128ed55842d3" (UID: "f634cef4-1591-4051-922f-128ed55842d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:04:54.243924 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.243849 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f634cef4-1591-4051-922f-128ed55842d3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f634cef4-1591-4051-922f-128ed55842d3" (UID: "f634cef4-1591-4051-922f-128ed55842d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:04:54.244075 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.244038 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f634cef4-1591-4051-922f-128ed55842d3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f634cef4-1591-4051-922f-128ed55842d3" (UID: "f634cef4-1591-4051-922f-128ed55842d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:04:54.244421 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.244351 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f634cef4-1591-4051-922f-128ed55842d3-istio-data" (OuterVolumeSpecName: "istio-data") pod "f634cef4-1591-4051-922f-128ed55842d3" (UID: "f634cef4-1591-4051-922f-128ed55842d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:04:54.246617 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.246590 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f634cef4-1591-4051-922f-128ed55842d3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f634cef4-1591-4051-922f-128ed55842d3" (UID: "f634cef4-1591-4051-922f-128ed55842d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:04:54.246825 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.246786 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f634cef4-1591-4051-922f-128ed55842d3-kube-api-access-dn78f" (OuterVolumeSpecName: "kube-api-access-dn78f") pod "f634cef4-1591-4051-922f-128ed55842d3" (UID: "f634cef4-1591-4051-922f-128ed55842d3"). InnerVolumeSpecName "kube-api-access-dn78f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:04:54.247040 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.247009 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f634cef4-1591-4051-922f-128ed55842d3-istio-token" (OuterVolumeSpecName: "istio-token") pod "f634cef4-1591-4051-922f-128ed55842d3" (UID: "f634cef4-1591-4051-922f-128ed55842d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:04:54.248489 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.248456 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f634cef4-1591-4051-922f-128ed55842d3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f634cef4-1591-4051-922f-128ed55842d3" (UID: "f634cef4-1591-4051-922f-128ed55842d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 13:04:54.325850 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.325821 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w"] Jul 16 13:04:54.326898 ip-10-0-140-130 kubenswrapper[2582]: W0716 13:04:54.326872 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf767df7e_814d_4f2f_bd4f_600c6addbe2b.slice/crio-9e4d36ac270c5a9e2fb76033b94bca8a1fe19d99b65b7f76dfc09300487bcc78 WatchSource:0}: Error finding container 9e4d36ac270c5a9e2fb76033b94bca8a1fe19d99b65b7f76dfc09300487bcc78: Status 404 returned error can't find the container with id 9e4d36ac270c5a9e2fb76033b94bca8a1fe19d99b65b7f76dfc09300487bcc78 Jul 16 13:04:54.329112 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.329080 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:04:54.329206 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.329145 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:04:54.329206 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.329173 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:04:54.343959 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.343937 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f634cef4-1591-4051-922f-128ed55842d3-workload-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:04:54.344038 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.343968 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn78f\" (UniqueName: \"kubernetes.io/projected/f634cef4-1591-4051-922f-128ed55842d3-kube-api-access-dn78f\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:04:54.344038 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.343983 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f634cef4-1591-4051-922f-128ed55842d3-istio-envoy\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:04:54.344038 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.343997 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f634cef4-1591-4051-922f-128ed55842d3-credential-socket\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:04:54.344038 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.344010 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f634cef4-1591-4051-922f-128ed55842d3-istio-token\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:04:54.344038 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.344024 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f634cef4-1591-4051-922f-128ed55842d3-istio-podinfo\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:04:54.344194 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.344037 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f634cef4-1591-4051-922f-128ed55842d3-istiod-ca-cert\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:04:54.344194 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.344051 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f634cef4-1591-4051-922f-128ed55842d3-workload-socket\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:04:54.344194 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.344061 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f634cef4-1591-4051-922f-128ed55842d3-istio-data\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:04:54.752742 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.752695 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:54.855083 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.855058 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:54.972837 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.972805 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" event={"ID":"f767df7e-814d-4f2f-bd4f-600c6addbe2b","Type":"ContainerStarted","Data":"33324d158e50e569ab9f21c3212b71aebdf04769a72c7a2e6339cd3228b7afce"} Jul 16 13:04:54.973044 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.972848 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" event={"ID":"f767df7e-814d-4f2f-bd4f-600c6addbe2b","Type":"ContainerStarted","Data":"9e4d36ac270c5a9e2fb76033b94bca8a1fe19d99b65b7f76dfc09300487bcc78"} Jul 16 13:04:54.974117 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.974097 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" event={"ID":"f634cef4-1591-4051-922f-128ed55842d3","Type":"ContainerDied","Data":"53c1e0a0e3b0e7503e9a4a04fbfe5fe085456c146858fcbff166efce8e5fa45d"} Jul 16 13:04:54.974179 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.974134 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz" Jul 16 13:04:54.974179 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.974150 2582 scope.go:117] "RemoveContainer" containerID="74652b283f61171b3fb777808018c4c05d8112f401989461cd1f0a31117cfb79" Jul 16 13:04:54.993889 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:54.993851 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" podStartSLOduration=1.9938373459999998 podStartE2EDuration="1.993837346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:04:54.991527693 +0000 UTC m=+1553.469273365" watchObservedRunningTime="2026-07-16 13:04:54.993837346 +0000 UTC m=+1553.471582961" Jul 16 13:04:55.004992 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:55.004929 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz"] Jul 16 13:04:55.006951 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:55.006927 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-5a470a1b-gw-openshift-default-bb58b99f5-ctrhz"] Jul 16 13:04:55.192440 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:55.192407 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:04:55.197311 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:55.197287 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:04:55.977326 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:55.977290 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f634cef4-1591-4051-922f-128ed55842d3" path="/var/lib/kubelet/pods/f634cef4-1591-4051-922f-128ed55842d3/volumes" Jul 16 13:04:55.979932 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:55.979909 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:04:55.980984 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:55.980966 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:04:57.575462 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:57.575427 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:59.531491 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:59.531454 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w"] Jul 16 13:04:59.531906 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:59.531648 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" podUID="f767df7e-814d-4f2f-bd4f-600c6addbe2b" containerName="istio-proxy" containerID="cri-o://33324d158e50e569ab9f21c3212b71aebdf04769a72c7a2e6339cd3228b7afce" gracePeriod=30 Jul 16 13:04:59.560536 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:59.560496 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:04:59.664189 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:04:59.664157 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:05:01.441176 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:01.441146 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:05:02.444399 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:02.444345 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:05:05.626268 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:05.626232 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:05:05.692575 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:05.692527 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:05:05.783613 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:05.783592 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:05:05.846081 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:05.846003 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f767df7e-814d-4f2f-bd4f-600c6addbe2b-istio-envoy\") pod \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " Jul 16 13:05:05.846081 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:05.846046 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f767df7e-814d-4f2f-bd4f-600c6addbe2b-istio-podinfo\") pod \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " Jul 16 13:05:05.846081 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:05.846076 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjxrm\" (UniqueName: \"kubernetes.io/projected/f767df7e-814d-4f2f-bd4f-600c6addbe2b-kube-api-access-tjxrm\") pod \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " Jul 16 13:05:05.846353 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:05.846099 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f767df7e-814d-4f2f-bd4f-600c6addbe2b-workload-socket\") pod \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " Jul 16 13:05:05.846353 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:05.846135 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f767df7e-814d-4f2f-bd4f-600c6addbe2b-istio-token\") pod \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " Jul 16 13:05:05.846353 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:05.846171 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f767df7e-814d-4f2f-bd4f-600c6addbe2b-istio-data\") pod \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " Jul 16 13:05:05.846353 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:05.846202 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f767df7e-814d-4f2f-bd4f-600c6addbe2b-workload-certs\") pod \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " Jul 16 13:05:05.846353 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:05.846230 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f767df7e-814d-4f2f-bd4f-600c6addbe2b-credential-socket\") pod \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " Jul 16 13:05:05.846353 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:05.846273 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f767df7e-814d-4f2f-bd4f-600c6addbe2b-istiod-ca-cert\") pod \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\" (UID: \"f767df7e-814d-4f2f-bd4f-600c6addbe2b\") " Jul 16 13:05:05.846745 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:05.846458 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f767df7e-814d-4f2f-bd4f-600c6addbe2b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f767df7e-814d-4f2f-bd4f-600c6addbe2b" (UID: "f767df7e-814d-4f2f-bd4f-600c6addbe2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:05:05.846745 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:05.846600 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f767df7e-814d-4f2f-bd4f-600c6addbe2b-workload-socket\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:05:05.846745 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:05.846590 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f767df7e-814d-4f2f-bd4f-600c6addbe2b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f767df7e-814d-4f2f-bd4f-600c6addbe2b" (UID: "f767df7e-814d-4f2f-bd4f-600c6addbe2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:05:05.846890 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:05.846769 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f767df7e-814d-4f2f-bd4f-600c6addbe2b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f767df7e-814d-4f2f-bd4f-600c6addbe2b" (UID: "f767df7e-814d-4f2f-bd4f-600c6addbe2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:05:05.847378 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:05.847356 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f767df7e-814d-4f2f-bd4f-600c6addbe2b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f767df7e-814d-4f2f-bd4f-600c6addbe2b" (UID: "f767df7e-814d-4f2f-bd4f-600c6addbe2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:05:05.847534 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:05.847511 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f767df7e-814d-4f2f-bd4f-600c6addbe2b-istio-data" (OuterVolumeSpecName: "istio-data") pod "f767df7e-814d-4f2f-bd4f-600c6addbe2b" (UID: "f767df7e-814d-4f2f-bd4f-600c6addbe2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:05:05.848438 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:05.848413 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f767df7e-814d-4f2f-bd4f-600c6addbe2b-kube-api-access-tjxrm" (OuterVolumeSpecName: "kube-api-access-tjxrm") pod "f767df7e-814d-4f2f-bd4f-600c6addbe2b" (UID: "f767df7e-814d-4f2f-bd4f-600c6addbe2b"). InnerVolumeSpecName "kube-api-access-tjxrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:05:05.848529 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:05.848413 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f767df7e-814d-4f2f-bd4f-600c6addbe2b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f767df7e-814d-4f2f-bd4f-600c6addbe2b" (UID: "f767df7e-814d-4f2f-bd4f-600c6addbe2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 13:05:05.848857 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:05.848830 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f767df7e-814d-4f2f-bd4f-600c6addbe2b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f767df7e-814d-4f2f-bd4f-600c6addbe2b" (UID: "f767df7e-814d-4f2f-bd4f-600c6addbe2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:05:05.848857 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:05.848839 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f767df7e-814d-4f2f-bd4f-600c6addbe2b-istio-token" (OuterVolumeSpecName: "istio-token") pod "f767df7e-814d-4f2f-bd4f-600c6addbe2b" (UID: "f767df7e-814d-4f2f-bd4f-600c6addbe2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:05:05.947704 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:05.947676 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f767df7e-814d-4f2f-bd4f-600c6addbe2b-istio-token\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:05:05.947704 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:05.947700 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f767df7e-814d-4f2f-bd4f-600c6addbe2b-istio-data\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:05:05.947704 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:05.947710 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f767df7e-814d-4f2f-bd4f-600c6addbe2b-workload-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:05:05.947898 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:05.947717 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f767df7e-814d-4f2f-bd4f-600c6addbe2b-credential-socket\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:05:05.947898 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:05.947727 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f767df7e-814d-4f2f-bd4f-600c6addbe2b-istiod-ca-cert\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:05:05.947898 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:05.947735 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f767df7e-814d-4f2f-bd4f-600c6addbe2b-istio-envoy\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:05:05.947898 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:05.947742 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f767df7e-814d-4f2f-bd4f-600c6addbe2b-istio-podinfo\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:05:05.947898 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:05.947750 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjxrm\" (UniqueName: \"kubernetes.io/projected/f767df7e-814d-4f2f-bd4f-600c6addbe2b-kube-api-access-tjxrm\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:05:06.037786 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:06.037753 2582 generic.go:356] "Generic (PLEG): container finished" podID="f767df7e-814d-4f2f-bd4f-600c6addbe2b" containerID="33324d158e50e569ab9f21c3212b71aebdf04769a72c7a2e6339cd3228b7afce" exitCode=0 Jul 16 13:05:06.037887 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:06.037798 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" Jul 16 13:05:06.037887 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:06.037838 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" event={"ID":"f767df7e-814d-4f2f-bd4f-600c6addbe2b","Type":"ContainerDied","Data":"33324d158e50e569ab9f21c3212b71aebdf04769a72c7a2e6339cd3228b7afce"} Jul 16 13:05:06.037973 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:06.037889 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w" event={"ID":"f767df7e-814d-4f2f-bd4f-600c6addbe2b","Type":"ContainerDied","Data":"9e4d36ac270c5a9e2fb76033b94bca8a1fe19d99b65b7f76dfc09300487bcc78"} Jul 16 13:05:06.037973 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:06.037913 2582 scope.go:117] "RemoveContainer" containerID="33324d158e50e569ab9f21c3212b71aebdf04769a72c7a2e6339cd3228b7afce" Jul 16 13:05:06.046162 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:06.046145 2582 scope.go:117] "RemoveContainer" containerID="33324d158e50e569ab9f21c3212b71aebdf04769a72c7a2e6339cd3228b7afce" Jul 16 13:05:06.046432 ip-10-0-140-130 kubenswrapper[2582]: E0716 13:05:06.046412 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33324d158e50e569ab9f21c3212b71aebdf04769a72c7a2e6339cd3228b7afce\": container with ID starting with 33324d158e50e569ab9f21c3212b71aebdf04769a72c7a2e6339cd3228b7afce not found: ID does not exist" containerID="33324d158e50e569ab9f21c3212b71aebdf04769a72c7a2e6339cd3228b7afce" Jul 16 13:05:06.046512 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:06.046441 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33324d158e50e569ab9f21c3212b71aebdf04769a72c7a2e6339cd3228b7afce"} err="failed to get container status \"33324d158e50e569ab9f21c3212b71aebdf04769a72c7a2e6339cd3228b7afce\": rpc error: code = NotFound desc = could not find container \"33324d158e50e569ab9f21c3212b71aebdf04769a72c7a2e6339cd3228b7afce\": container with ID starting with 33324d158e50e569ab9f21c3212b71aebdf04769a72c7a2e6339cd3228b7afce not found: ID does not exist" Jul 16 13:05:06.053656 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:06.053632 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w"] Jul 16 13:05:06.059630 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:06.059611 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-b0d8b7c2-openshift-default-88857cbd7-49l4w"] Jul 16 13:05:06.125684 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:06.125657 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:05:06.220596 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:06.220568 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:05:06.322740 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:06.322706 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:05:07.749154 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:07.749119 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:05:07.976992 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:07.976947 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f767df7e-814d-4f2f-bd4f-600c6addbe2b" path="/var/lib/kubelet/pods/f767df7e-814d-4f2f-bd4f-600c6addbe2b/volumes" Jul 16 13:05:14.197312 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:14.197269 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:05:14.297766 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:14.297730 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:05:14.404017 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:14.403983 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:05:14.851202 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:14.851174 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:05:40.889134 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:40.889101 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:05:46.937487 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:46.937451 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:05:47.040151 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:47.040112 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:05:47.143330 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:05:47.143297 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:03.079656 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:03.079626 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:08.911436 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:08.911377 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:09.013497 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:09.013466 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:09.112201 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:09.112160 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:19.608660 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:19.608623 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:20.389519 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:20.389485 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:26.172986 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:26.172949 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:26.269010 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:26.268976 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:26.371015 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:26.370980 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:26.468926 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:26.468841 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:27.265859 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:27.265827 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:27.366141 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:27.366104 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:27.466916 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:27.466883 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:29.137739 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:29.137708 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:35.083229 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:35.083187 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:35.759763 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:35.759723 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:35.860829 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:35.860794 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:35.964685 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:35.964649 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:41.682336 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:41.682301 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:41.782771 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:41.782740 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:41.948001 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:41.947901 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:42.236918 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.236846 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs"] Jul 16 13:06:42.237128 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.237116 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f634cef4-1591-4051-922f-128ed55842d3" containerName="istio-proxy" Jul 16 13:06:42.237173 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.237129 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f634cef4-1591-4051-922f-128ed55842d3" containerName="istio-proxy" Jul 16 13:06:42.237173 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.237140 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f767df7e-814d-4f2f-bd4f-600c6addbe2b" containerName="istio-proxy" Jul 16 13:06:42.237173 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.237145 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f767df7e-814d-4f2f-bd4f-600c6addbe2b" containerName="istio-proxy" Jul 16 13:06:42.237268 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.237202 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f767df7e-814d-4f2f-bd4f-600c6addbe2b" containerName="istio-proxy" Jul 16 13:06:42.237268 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.237211 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f634cef4-1591-4051-922f-128ed55842d3" containerName="istio-proxy" Jul 16 13:06:42.238989 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.238973 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:06:42.242605 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.242582 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-41581b25-openshift-default-dockercfg-v5xsk\"" Jul 16 13:06:42.271652 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.271626 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:42.278569 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.278551 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs"] Jul 16 13:06:42.427962 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.427927 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dcd169f-3c4a-4534-8504-2cdbc418bc95-istio-data\") pod \"e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:06:42.427962 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.427975 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dcd169f-3c4a-4534-8504-2cdbc418bc95-credential-socket\") pod \"e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:06:42.428237 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.427991 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dcd169f-3c4a-4534-8504-2cdbc418bc95-istio-envoy\") pod \"e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:06:42.428237 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.428114 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dcd169f-3c4a-4534-8504-2cdbc418bc95-workload-socket\") pod \"e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:06:42.428237 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.428157 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dcd169f-3c4a-4534-8504-2cdbc418bc95-istio-podinfo\") pod \"e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:06:42.428237 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.428185 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dcd169f-3c4a-4534-8504-2cdbc418bc95-istiod-ca-cert\") pod \"e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:06:42.428237 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.428224 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thkf4\" (UniqueName: \"kubernetes.io/projected/5dcd169f-3c4a-4534-8504-2cdbc418bc95-kube-api-access-thkf4\") pod \"e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:06:42.428492 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.428261 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dcd169f-3c4a-4534-8504-2cdbc418bc95-workload-certs\") pod \"e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:06:42.428492 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.428303 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dcd169f-3c4a-4534-8504-2cdbc418bc95-istio-token\") pod \"e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:06:42.528735 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.528649 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dcd169f-3c4a-4534-8504-2cdbc418bc95-istio-token\") pod \"e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:06:42.528735 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.528691 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dcd169f-3c4a-4534-8504-2cdbc418bc95-istio-data\") pod \"e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:06:42.528735 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.528717 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dcd169f-3c4a-4534-8504-2cdbc418bc95-credential-socket\") pod \"e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:06:42.529015 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.528826 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dcd169f-3c4a-4534-8504-2cdbc418bc95-istio-envoy\") pod \"e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:06:42.529015 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.528873 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dcd169f-3c4a-4534-8504-2cdbc418bc95-workload-socket\") pod \"e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:06:42.529015 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.528908 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dcd169f-3c4a-4534-8504-2cdbc418bc95-istio-podinfo\") pod \"e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:06:42.529015 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.528938 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dcd169f-3c4a-4534-8504-2cdbc418bc95-istiod-ca-cert\") pod \"e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:06:42.529015 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.528975 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thkf4\" (UniqueName: \"kubernetes.io/projected/5dcd169f-3c4a-4534-8504-2cdbc418bc95-kube-api-access-thkf4\") pod \"e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:06:42.529015 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.529009 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dcd169f-3c4a-4534-8504-2cdbc418bc95-workload-certs\") pod \"e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:06:42.529411 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.529356 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dcd169f-3c4a-4534-8504-2cdbc418bc95-credential-socket\") pod \"e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:06:42.529532 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.529423 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dcd169f-3c4a-4534-8504-2cdbc418bc95-istio-data\") pod \"e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:06:42.529532 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.529446 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dcd169f-3c4a-4534-8504-2cdbc418bc95-workload-socket\") pod \"e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:06:42.529532 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.529500 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dcd169f-3c4a-4534-8504-2cdbc418bc95-workload-certs\") pod \"e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:06:42.529872 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.529811 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dcd169f-3c4a-4534-8504-2cdbc418bc95-istiod-ca-cert\") pod \"e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:06:42.531274 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.531253 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dcd169f-3c4a-4534-8504-2cdbc418bc95-istio-envoy\") pod \"e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:06:42.531375 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.531356 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dcd169f-3c4a-4534-8504-2cdbc418bc95-istio-podinfo\") pod \"e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:06:42.539352 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.539332 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dcd169f-3c4a-4534-8504-2cdbc418bc95-istio-token\") pod \"e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:06:42.539813 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.539795 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thkf4\" (UniqueName: \"kubernetes.io/projected/5dcd169f-3c4a-4534-8504-2cdbc418bc95-kube-api-access-thkf4\") pod \"e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:06:42.548726 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.548706 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:06:42.687226 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.687198 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs"] Jul 16 13:06:42.689344 ip-10-0-140-130 kubenswrapper[2582]: W0716 13:06:42.689319 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dcd169f_3c4a_4534_8504_2cdbc418bc95.slice/crio-e868e88a4d1832891feec6409b2004dd6802dc9e5bff696b74b97098cb82caff WatchSource:0}: Error finding container e868e88a4d1832891feec6409b2004dd6802dc9e5bff696b74b97098cb82caff: Status 404 returned error can't find the container with id e868e88a4d1832891feec6409b2004dd6802dc9e5bff696b74b97098cb82caff Jul 16 13:06:42.691334 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.691304 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:06:42.691443 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.691363 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:06:42.691443 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.691417 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:06:42.791273 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:42.791191 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:43.485151 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:43.485112 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" event={"ID":"5dcd169f-3c4a-4534-8504-2cdbc418bc95","Type":"ContainerStarted","Data":"14a65fe495c5ad4d61e9e1869f713cf296a994d4f687b5927dba560d1e3eaeaf"} Jul 16 13:06:43.485323 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:43.485165 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" event={"ID":"5dcd169f-3c4a-4534-8504-2cdbc418bc95","Type":"ContainerStarted","Data":"e868e88a4d1832891feec6409b2004dd6802dc9e5bff696b74b97098cb82caff"} Jul 16 13:06:43.506458 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:43.506380 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" podStartSLOduration=1.50636528 podStartE2EDuration="1.50636528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:06:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:06:43.504763139 +0000 UTC m=+1661.982508775" watchObservedRunningTime="2026-07-16 13:06:43.50636528 +0000 UTC m=+1661.984110896" Jul 16 13:06:43.549883 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:43.549849 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:06:43.554877 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:43.554848 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:06:44.435844 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:44.435799 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:44.489932 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:44.489892 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:06:44.490773 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:44.490753 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:06:44.645726 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:44.645683 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:44.738674 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:44.738588 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:46.579261 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:46.579220 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:46.684974 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:46.684946 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:49.970543 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:49.970504 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:50.642890 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:50.642849 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:50.739684 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:50.739642 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:51.078638 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:51.078547 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:56.939555 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:56.939514 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:57.043560 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:57.043515 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:06:57.624714 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:06:57.624683 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:07:00.532558 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:00.532517 2582 patch_prober.go:28] interesting pod/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 16 13:07:00.532929 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:00.532602 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" podUID="5dcd169f-3c4a-4534-8504-2cdbc418bc95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 16 13:07:01.575293 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:01.575216 2582 generic.go:356] "Generic (PLEG): container finished" podID="5dcd169f-3c4a-4534-8504-2cdbc418bc95" containerID="14a65fe495c5ad4d61e9e1869f713cf296a994d4f687b5927dba560d1e3eaeaf" exitCode=0 Jul 16 13:07:01.575293 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:01.575263 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" event={"ID":"5dcd169f-3c4a-4534-8504-2cdbc418bc95","Type":"ContainerDied","Data":"14a65fe495c5ad4d61e9e1869f713cf296a994d4f687b5927dba560d1e3eaeaf"} Jul 16 13:07:01.575744 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:01.575692 2582 scope.go:117] "RemoveContainer" containerID="14a65fe495c5ad4d61e9e1869f713cf296a994d4f687b5927dba560d1e3eaeaf" Jul 16 13:07:01.576661 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:01.576632 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:07:01.576760 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:01.576715 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:07:01.576760 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:01.576746 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:07:02.581461 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:02.581427 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" event={"ID":"5dcd169f-3c4a-4534-8504-2cdbc418bc95","Type":"ContainerStarted","Data":"71931c06f40f31da1142e20d2f4fda134f8d5f162968b05e0cab68535174c9df"} Jul 16 13:07:03.549435 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:03.549364 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:07:03.549687 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:03.549664 2582 patch_prober.go:28] interesting pod/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 16 13:07:03.549733 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:03.549715 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" podUID="5dcd169f-3c4a-4534-8504-2cdbc418bc95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 16 13:07:03.870454 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:03.870365 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:07:03.988099 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:03.988060 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:07:04.135223 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:04.135191 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs"] Jul 16 13:07:04.176792 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:04.176761 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:07:04.373651 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:04.373618 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:07:04.549369 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:04.549288 2582 patch_prober.go:28] interesting pod/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 16 13:07:04.549369 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:04.549345 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" podUID="5dcd169f-3c4a-4534-8504-2cdbc418bc95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 16 13:07:04.568446 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:04.568411 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:07:04.591444 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:04.591405 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" podUID="5dcd169f-3c4a-4534-8504-2cdbc418bc95" containerName="istio-proxy" containerID="cri-o://71931c06f40f31da1142e20d2f4fda134f8d5f162968b05e0cab68535174c9df" gracePeriod=30 Jul 16 13:07:04.669530 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:04.669497 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:07:04.771403 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:04.771353 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:07:10.404164 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:10.404131 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:07:10.509193 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:10.509163 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:07:10.608239 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:10.608205 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:07:10.794676 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:10.794600 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:07:11.528382 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.528359 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:07:11.587619 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.587532 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dcd169f-3c4a-4534-8504-2cdbc418bc95-istio-envoy\") pod \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " Jul 16 13:07:11.587619 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.587566 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dcd169f-3c4a-4534-8504-2cdbc418bc95-workload-socket\") pod \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " Jul 16 13:07:11.587619 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.587590 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dcd169f-3c4a-4534-8504-2cdbc418bc95-istiod-ca-cert\") pod \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " Jul 16 13:07:11.587874 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.587641 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dcd169f-3c4a-4534-8504-2cdbc418bc95-istio-token\") pod \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " Jul 16 13:07:11.587874 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.587670 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dcd169f-3c4a-4534-8504-2cdbc418bc95-istio-podinfo\") pod \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " Jul 16 13:07:11.587874 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.587732 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dcd169f-3c4a-4534-8504-2cdbc418bc95-istio-data\") pod \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " Jul 16 13:07:11.587874 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.587813 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dcd169f-3c4a-4534-8504-2cdbc418bc95-workload-certs\") pod \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " Jul 16 13:07:11.587874 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.587825 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dcd169f-3c4a-4534-8504-2cdbc418bc95-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5dcd169f-3c4a-4534-8504-2cdbc418bc95" (UID: "5dcd169f-3c4a-4534-8504-2cdbc418bc95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:07:11.588149 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.587876 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thkf4\" (UniqueName: \"kubernetes.io/projected/5dcd169f-3c4a-4534-8504-2cdbc418bc95-kube-api-access-thkf4\") pod \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " Jul 16 13:07:11.588149 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.587938 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dcd169f-3c4a-4534-8504-2cdbc418bc95-credential-socket\") pod \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\" (UID: \"5dcd169f-3c4a-4534-8504-2cdbc418bc95\") " Jul 16 13:07:11.588149 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.588116 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dcd169f-3c4a-4534-8504-2cdbc418bc95-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5dcd169f-3c4a-4534-8504-2cdbc418bc95" (UID: "5dcd169f-3c4a-4534-8504-2cdbc418bc95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:07:11.588149 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.588125 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dcd169f-3c4a-4534-8504-2cdbc418bc95-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5dcd169f-3c4a-4534-8504-2cdbc418bc95" (UID: "5dcd169f-3c4a-4534-8504-2cdbc418bc95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:07:11.588354 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.588243 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dcd169f-3c4a-4534-8504-2cdbc418bc95-workload-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:07:11.588354 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.588261 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dcd169f-3c4a-4534-8504-2cdbc418bc95-workload-socket\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:07:11.588354 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.588275 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dcd169f-3c4a-4534-8504-2cdbc418bc95-istiod-ca-cert\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:07:11.588354 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.588278 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dcd169f-3c4a-4534-8504-2cdbc418bc95-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5dcd169f-3c4a-4534-8504-2cdbc418bc95" (UID: "5dcd169f-3c4a-4534-8504-2cdbc418bc95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:07:11.588758 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.588510 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dcd169f-3c4a-4534-8504-2cdbc418bc95-istio-data" (OuterVolumeSpecName: "istio-data") pod "5dcd169f-3c4a-4534-8504-2cdbc418bc95" (UID: "5dcd169f-3c4a-4534-8504-2cdbc418bc95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:07:11.590064 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.590032 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5dcd169f-3c4a-4534-8504-2cdbc418bc95-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5dcd169f-3c4a-4534-8504-2cdbc418bc95" (UID: "5dcd169f-3c4a-4534-8504-2cdbc418bc95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 13:07:11.590210 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.590186 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dcd169f-3c4a-4534-8504-2cdbc418bc95-kube-api-access-thkf4" (OuterVolumeSpecName: "kube-api-access-thkf4") pod "5dcd169f-3c4a-4534-8504-2cdbc418bc95" (UID: "5dcd169f-3c4a-4534-8504-2cdbc418bc95"). InnerVolumeSpecName "kube-api-access-thkf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:07:11.590376 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.590299 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dcd169f-3c4a-4534-8504-2cdbc418bc95-istio-token" (OuterVolumeSpecName: "istio-token") pod "5dcd169f-3c4a-4534-8504-2cdbc418bc95" (UID: "5dcd169f-3c4a-4534-8504-2cdbc418bc95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:07:11.590376 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.590310 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dcd169f-3c4a-4534-8504-2cdbc418bc95-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5dcd169f-3c4a-4534-8504-2cdbc418bc95" (UID: "5dcd169f-3c4a-4534-8504-2cdbc418bc95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:07:11.631084 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.631055 2582 generic.go:356] "Generic (PLEG): container finished" podID="5dcd169f-3c4a-4534-8504-2cdbc418bc95" containerID="71931c06f40f31da1142e20d2f4fda134f8d5f162968b05e0cab68535174c9df" exitCode=0 Jul 16 13:07:11.631185 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.631108 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" Jul 16 13:07:11.631185 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.631107 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" event={"ID":"5dcd169f-3c4a-4534-8504-2cdbc418bc95","Type":"ContainerDied","Data":"71931c06f40f31da1142e20d2f4fda134f8d5f162968b05e0cab68535174c9df"} Jul 16 13:07:11.631259 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.631243 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs" event={"ID":"5dcd169f-3c4a-4534-8504-2cdbc418bc95","Type":"ContainerDied","Data":"e868e88a4d1832891feec6409b2004dd6802dc9e5bff696b74b97098cb82caff"} Jul 16 13:07:11.631295 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.631273 2582 scope.go:117] "RemoveContainer" containerID="71931c06f40f31da1142e20d2f4fda134f8d5f162968b05e0cab68535174c9df" Jul 16 13:07:11.640094 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.640078 2582 scope.go:117] "RemoveContainer" containerID="14a65fe495c5ad4d61e9e1869f713cf296a994d4f687b5927dba560d1e3eaeaf" Jul 16 13:07:11.647716 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.647699 2582 scope.go:117] "RemoveContainer" containerID="71931c06f40f31da1142e20d2f4fda134f8d5f162968b05e0cab68535174c9df" Jul 16 13:07:11.647936 ip-10-0-140-130 kubenswrapper[2582]: E0716 13:07:11.647921 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71931c06f40f31da1142e20d2f4fda134f8d5f162968b05e0cab68535174c9df\": container with ID starting with 71931c06f40f31da1142e20d2f4fda134f8d5f162968b05e0cab68535174c9df not found: ID does not exist" containerID="71931c06f40f31da1142e20d2f4fda134f8d5f162968b05e0cab68535174c9df" Jul 16 13:07:11.647982 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.647944 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71931c06f40f31da1142e20d2f4fda134f8d5f162968b05e0cab68535174c9df"} err="failed to get container status \"71931c06f40f31da1142e20d2f4fda134f8d5f162968b05e0cab68535174c9df\": rpc error: code = NotFound desc = could not find container \"71931c06f40f31da1142e20d2f4fda134f8d5f162968b05e0cab68535174c9df\": container with ID starting with 71931c06f40f31da1142e20d2f4fda134f8d5f162968b05e0cab68535174c9df not found: ID does not exist" Jul 16 13:07:11.647982 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.647960 2582 scope.go:117] "RemoveContainer" containerID="14a65fe495c5ad4d61e9e1869f713cf296a994d4f687b5927dba560d1e3eaeaf" Jul 16 13:07:11.648204 ip-10-0-140-130 kubenswrapper[2582]: E0716 13:07:11.648186 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14a65fe495c5ad4d61e9e1869f713cf296a994d4f687b5927dba560d1e3eaeaf\": container with ID starting with 14a65fe495c5ad4d61e9e1869f713cf296a994d4f687b5927dba560d1e3eaeaf not found: ID does not exist" containerID="14a65fe495c5ad4d61e9e1869f713cf296a994d4f687b5927dba560d1e3eaeaf" Jul 16 13:07:11.648243 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.648209 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a65fe495c5ad4d61e9e1869f713cf296a994d4f687b5927dba560d1e3eaeaf"} err="failed to get container status \"14a65fe495c5ad4d61e9e1869f713cf296a994d4f687b5927dba560d1e3eaeaf\": rpc error: code = NotFound desc = could not find container \"14a65fe495c5ad4d61e9e1869f713cf296a994d4f687b5927dba560d1e3eaeaf\": container with ID starting with 14a65fe495c5ad4d61e9e1869f713cf296a994d4f687b5927dba560d1e3eaeaf not found: ID does not exist" Jul 16 13:07:11.664689 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.664659 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs"] Jul 16 13:07:11.684847 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.684826 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-41581b25-openshift-default-77f7d6f5dc-gzlrs"] Jul 16 13:07:11.689003 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.688987 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dcd169f-3c4a-4534-8504-2cdbc418bc95-istio-podinfo\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:07:11.689064 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.689006 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dcd169f-3c4a-4534-8504-2cdbc418bc95-istio-data\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:07:11.689064 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.689016 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thkf4\" (UniqueName: \"kubernetes.io/projected/5dcd169f-3c4a-4534-8504-2cdbc418bc95-kube-api-access-thkf4\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:07:11.689064 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.689026 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dcd169f-3c4a-4534-8504-2cdbc418bc95-credential-socket\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:07:11.689064 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.689034 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dcd169f-3c4a-4534-8504-2cdbc418bc95-istio-envoy\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:07:11.689064 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.689042 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dcd169f-3c4a-4534-8504-2cdbc418bc95-istio-token\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:07:11.977726 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:11.977688 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dcd169f-3c4a-4534-8504-2cdbc418bc95" path="/var/lib/kubelet/pods/5dcd169f-3c4a-4534-8504-2cdbc418bc95/volumes" Jul 16 13:07:16.874780 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:16.874746 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:07:16.974822 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:16.974762 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:07:24.563773 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:24.563693 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:07:24.768872 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:24.768828 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:07:25.430525 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:25.430492 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:07:25.531777 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:25.531740 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:07:25.691317 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:25.691231 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:07:27.047626 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:27.047594 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:07:33.107287 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:33.107249 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:07:33.207450 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:33.207420 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:07:33.873069 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:33.873032 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:07:41.416887 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:41.416846 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:07:41.517377 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:41.517345 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:07:50.775258 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:50.775219 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:07:51.297009 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:07:51.296974 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:08:06.014600 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:06.014564 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:08:06.111304 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:06.111267 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:08:12.103144 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:12.103112 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:08:18.784168 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:18.784121 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:08:19.492657 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.492621 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m"] Jul 16 13:08:19.492930 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.492918 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dcd169f-3c4a-4534-8504-2cdbc418bc95" containerName="istio-proxy" Jul 16 13:08:19.492981 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.492940 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dcd169f-3c4a-4534-8504-2cdbc418bc95" containerName="istio-proxy" Jul 16 13:08:19.492981 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.492950 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dcd169f-3c4a-4534-8504-2cdbc418bc95" containerName="istio-proxy" Jul 16 13:08:19.492981 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.492955 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dcd169f-3c4a-4534-8504-2cdbc418bc95" containerName="istio-proxy" Jul 16 13:08:19.493080 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.493012 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dcd169f-3c4a-4534-8504-2cdbc418bc95" containerName="istio-proxy" Jul 16 13:08:19.495699 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.495681 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:19.498009 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.497985 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ae18e9-openshift-default-dockercfg-rhwsq\"" Jul 16 13:08:19.516973 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.516941 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m"] Jul 16 13:08:19.619916 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.619871 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a017ea1-3051-43f5-8794-b9ed3d60ab28-credential-socket\") pod \"e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:19.619916 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.619919 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a017ea1-3051-43f5-8794-b9ed3d60ab28-workload-certs\") pod \"e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:19.620161 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.619940 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jpjf\" (UniqueName: \"kubernetes.io/projected/7a017ea1-3051-43f5-8794-b9ed3d60ab28-kube-api-access-5jpjf\") pod \"e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:19.620161 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.620012 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a017ea1-3051-43f5-8794-b9ed3d60ab28-istio-envoy\") pod \"e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:19.620161 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.620056 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a017ea1-3051-43f5-8794-b9ed3d60ab28-istio-podinfo\") pod \"e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:19.620161 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.620079 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a017ea1-3051-43f5-8794-b9ed3d60ab28-istiod-ca-cert\") pod \"e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:19.620161 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.620097 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a017ea1-3051-43f5-8794-b9ed3d60ab28-workload-socket\") pod \"e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:19.620161 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.620154 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a017ea1-3051-43f5-8794-b9ed3d60ab28-istio-token\") pod \"e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:19.620349 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.620174 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a017ea1-3051-43f5-8794-b9ed3d60ab28-istio-data\") pod \"e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:19.720960 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.720889 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a017ea1-3051-43f5-8794-b9ed3d60ab28-credential-socket\") pod \"e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:19.720960 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.720951 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a017ea1-3051-43f5-8794-b9ed3d60ab28-workload-certs\") pod \"e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:19.721216 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.720975 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jpjf\" (UniqueName: \"kubernetes.io/projected/7a017ea1-3051-43f5-8794-b9ed3d60ab28-kube-api-access-5jpjf\") pod \"e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:19.721216 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.721012 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a017ea1-3051-43f5-8794-b9ed3d60ab28-istio-envoy\") pod \"e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:19.721216 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.721044 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a017ea1-3051-43f5-8794-b9ed3d60ab28-istio-podinfo\") pod \"e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:19.721216 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.721072 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a017ea1-3051-43f5-8794-b9ed3d60ab28-istiod-ca-cert\") pod \"e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:19.721216 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.721096 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a017ea1-3051-43f5-8794-b9ed3d60ab28-workload-socket\") pod \"e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:19.721216 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.721128 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a017ea1-3051-43f5-8794-b9ed3d60ab28-istio-token\") pod \"e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:19.721216 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.721156 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a017ea1-3051-43f5-8794-b9ed3d60ab28-istio-data\") pod \"e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:19.721680 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.721305 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a017ea1-3051-43f5-8794-b9ed3d60ab28-credential-socket\") pod \"e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:19.721680 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.721554 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a017ea1-3051-43f5-8794-b9ed3d60ab28-istio-data\") pod \"e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:19.721844 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.721817 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a017ea1-3051-43f5-8794-b9ed3d60ab28-workload-certs\") pod \"e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:19.722057 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.722037 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a017ea1-3051-43f5-8794-b9ed3d60ab28-workload-socket\") pod \"e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:19.722150 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.722128 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a017ea1-3051-43f5-8794-b9ed3d60ab28-istiod-ca-cert\") pod \"e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:19.723753 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.723734 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a017ea1-3051-43f5-8794-b9ed3d60ab28-istio-podinfo\") pod \"e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:19.723899 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.723882 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a017ea1-3051-43f5-8794-b9ed3d60ab28-istio-envoy\") pod \"e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:19.729655 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.729623 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a017ea1-3051-43f5-8794-b9ed3d60ab28-istio-token\") pod \"e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:19.729844 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.729822 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jpjf\" (UniqueName: \"kubernetes.io/projected/7a017ea1-3051-43f5-8794-b9ed3d60ab28-kube-api-access-5jpjf\") pod \"e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:19.806024 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.805927 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:19.947748 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.947720 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m"] Jul 16 13:08:19.949650 ip-10-0-140-130 kubenswrapper[2582]: W0716 13:08:19.949621 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a017ea1_3051_43f5_8794_b9ed3d60ab28.slice/crio-b2506bcea3f284c7c6fd76a4ba518bbc86c636f6de9f7478162582443adfc401 WatchSource:0}: Error finding container b2506bcea3f284c7c6fd76a4ba518bbc86c636f6de9f7478162582443adfc401: Status 404 returned error can't find the container with id b2506bcea3f284c7c6fd76a4ba518bbc86c636f6de9f7478162582443adfc401 Jul 16 13:08:19.951726 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.951697 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:08:19.951821 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.951762 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:08:19.951821 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.951792 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:08:19.957030 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:19.957009 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" event={"ID":"7a017ea1-3051-43f5-8794-b9ed3d60ab28","Type":"ContainerStarted","Data":"b2506bcea3f284c7c6fd76a4ba518bbc86c636f6de9f7478162582443adfc401"} Jul 16 13:08:20.054450 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:20.054372 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:08:20.158115 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:20.158076 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:08:20.963659 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:20.963619 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" event={"ID":"7a017ea1-3051-43f5-8794-b9ed3d60ab28","Type":"ContainerStarted","Data":"5b2b99c1313c330caab567cebc178a978dfb78a8ed4035a94f5c3c49b7d918c2"} Jul 16 13:08:20.992274 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:20.992229 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" podStartSLOduration=1.99221355 podStartE2EDuration="1.99221355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:08:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:08:20.989554295 +0000 UTC m=+1759.467299910" watchObservedRunningTime="2026-07-16 13:08:20.99221355 +0000 UTC m=+1759.469959166" Jul 16 13:08:21.806673 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:21.806639 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:21.811892 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:21.811862 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:21.968475 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:21.968440 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:21.969332 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:21.969312 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:21.975101 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:21.975078 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:08:22.064705 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:22.064618 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:08:22.169359 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:22.169326 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:08:22.475141 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:22.475090 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:08:23.656817 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:23.656780 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:08:23.731213 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:23.731173 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:08:27.481740 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:27.481705 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:08:27.824647 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:27.824558 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:08:28.081948 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.081871 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4"] Jul 16 13:08:28.082451 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.082436 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dcd169f-3c4a-4534-8504-2cdbc418bc95" containerName="istio-proxy" Jul 16 13:08:28.085567 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.085549 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:28.088033 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.088011 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-cb4eee-openshift-default-dockercfg-hzfql\"" Jul 16 13:08:28.095182 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.095161 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4"] Jul 16 13:08:28.111530 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.111502 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:08:28.191517 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.191474 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12e2f5b8-c482-43fa-a69e-57c639af1ff8-istio-data\") pod \"e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:28.191710 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.191554 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e2f5b8-c482-43fa-a69e-57c639af1ff8-workload-socket\") pod \"e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:28.191710 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.191602 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12e2f5b8-c482-43fa-a69e-57c639af1ff8-istiod-ca-cert\") pod \"e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:28.191710 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.191634 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12e2f5b8-c482-43fa-a69e-57c639af1ff8-istio-envoy\") pod \"e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:28.191710 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.191674 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e2f5b8-c482-43fa-a69e-57c639af1ff8-workload-certs\") pod \"e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:28.191905 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.191711 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6bcn\" (UniqueName: \"kubernetes.io/projected/12e2f5b8-c482-43fa-a69e-57c639af1ff8-kube-api-access-m6bcn\") pod \"e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:28.191905 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.191752 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12e2f5b8-c482-43fa-a69e-57c639af1ff8-istio-podinfo\") pod \"e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:28.192003 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.191897 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12e2f5b8-c482-43fa-a69e-57c639af1ff8-istio-token\") pod \"e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:28.192003 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.191933 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e2f5b8-c482-43fa-a69e-57c639af1ff8-credential-socket\") pod \"e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:28.244894 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.244847 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:08:28.292785 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.292745 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12e2f5b8-c482-43fa-a69e-57c639af1ff8-istio-podinfo\") pod \"e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:28.292974 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.292814 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12e2f5b8-c482-43fa-a69e-57c639af1ff8-istio-token\") pod \"e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:28.292974 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.292841 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e2f5b8-c482-43fa-a69e-57c639af1ff8-credential-socket\") pod \"e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:28.292974 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.292879 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12e2f5b8-c482-43fa-a69e-57c639af1ff8-istio-data\") pod \"e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:28.292974 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.292906 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e2f5b8-c482-43fa-a69e-57c639af1ff8-workload-socket\") pod \"e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:28.292974 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.292946 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12e2f5b8-c482-43fa-a69e-57c639af1ff8-istiod-ca-cert\") pod \"e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:28.293205 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.292988 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12e2f5b8-c482-43fa-a69e-57c639af1ff8-istio-envoy\") pod \"e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:28.293241 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.293044 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e2f5b8-c482-43fa-a69e-57c639af1ff8-workload-certs\") pod \"e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:28.293549 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.293516 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6bcn\" (UniqueName: \"kubernetes.io/projected/12e2f5b8-c482-43fa-a69e-57c639af1ff8-kube-api-access-m6bcn\") pod \"e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:28.293671 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.293645 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e2f5b8-c482-43fa-a69e-57c639af1ff8-workload-socket\") pod \"e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:28.293740 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.293718 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12e2f5b8-c482-43fa-a69e-57c639af1ff8-istio-data\") pod \"e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:28.293954 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.293928 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e2f5b8-c482-43fa-a69e-57c639af1ff8-workload-certs\") pod \"e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:28.294094 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.294029 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e2f5b8-c482-43fa-a69e-57c639af1ff8-credential-socket\") pod \"e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:28.294741 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.294722 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12e2f5b8-c482-43fa-a69e-57c639af1ff8-istiod-ca-cert\") pod \"e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:28.296301 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.296276 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12e2f5b8-c482-43fa-a69e-57c639af1ff8-istio-envoy\") pod \"e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:28.296508 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.296486 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12e2f5b8-c482-43fa-a69e-57c639af1ff8-istio-podinfo\") pod \"e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:28.301571 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.301542 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12e2f5b8-c482-43fa-a69e-57c639af1ff8-istio-token\") pod \"e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:28.303082 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.303058 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6bcn\" (UniqueName: \"kubernetes.io/projected/12e2f5b8-c482-43fa-a69e-57c639af1ff8-kube-api-access-m6bcn\") pod \"e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:28.397356 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.397323 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:28.538733 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.538705 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4"] Jul 16 13:08:28.540890 ip-10-0-140-130 kubenswrapper[2582]: W0716 13:08:28.540865 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12e2f5b8_c482_43fa_a69e_57c639af1ff8.slice/crio-8702c0d478f18342a91c8bbbd24b34ee14533cdafd5b93b8e491c5b061fe9b56 WatchSource:0}: Error finding container 8702c0d478f18342a91c8bbbd24b34ee14533cdafd5b93b8e491c5b061fe9b56: Status 404 returned error can't find the container with id 8702c0d478f18342a91c8bbbd24b34ee14533cdafd5b93b8e491c5b061fe9b56 Jul 16 13:08:28.543034 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.543002 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:08:28.543099 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.543070 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:08:28.543132 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:28.543100 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:08:29.001180 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:29.001143 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" event={"ID":"12e2f5b8-c482-43fa-a69e-57c639af1ff8","Type":"ContainerStarted","Data":"36c0d800f085d760c0a58a643c9c8e1e0fd752dd5038db61a11a65a4daa63988"} Jul 16 13:08:29.001425 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:29.001200 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" event={"ID":"12e2f5b8-c482-43fa-a69e-57c639af1ff8","Type":"ContainerStarted","Data":"8702c0d478f18342a91c8bbbd24b34ee14533cdafd5b93b8e491c5b061fe9b56"} Jul 16 13:08:29.021488 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:29.021440 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" podStartSLOduration=1.021425568 podStartE2EDuration="1.021425568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:08:29.020231799 +0000 UTC m=+1767.497977436" watchObservedRunningTime="2026-07-16 13:08:29.021425568 +0000 UTC m=+1767.499171183" Jul 16 13:08:29.398083 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:29.398043 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:29.403358 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:29.403331 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:29.912583 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:29.912550 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:08:30.005852 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:30.005822 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:30.006840 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:30.006818 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:30.011896 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:30.011872 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:08:30.212930 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:30.212837 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:08:30.650551 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:30.650514 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:08:30.765189 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:30.765146 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:08:31.095836 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:31.095751 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:08:35.529380 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:35.529341 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:08:40.057868 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:40.057837 2582 generic.go:356] "Generic (PLEG): container finished" podID="7a017ea1-3051-43f5-8794-b9ed3d60ab28" containerID="5b2b99c1313c330caab567cebc178a978dfb78a8ed4035a94f5c3c49b7d918c2" exitCode=0 Jul 16 13:08:40.058221 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:40.057892 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" event={"ID":"7a017ea1-3051-43f5-8794-b9ed3d60ab28","Type":"ContainerDied","Data":"5b2b99c1313c330caab567cebc178a978dfb78a8ed4035a94f5c3c49b7d918c2"} Jul 16 13:08:40.058264 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:40.058225 2582 scope.go:117] "RemoveContainer" containerID="5b2b99c1313c330caab567cebc178a978dfb78a8ed4035a94f5c3c49b7d918c2" Jul 16 13:08:40.059179 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:40.059152 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:08:40.059240 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:40.059216 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:08:40.059307 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:40.059249 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:08:41.064680 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:41.064643 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" event={"ID":"7a017ea1-3051-43f5-8794-b9ed3d60ab28","Type":"ContainerStarted","Data":"2b4a16aa69d5d9f0adf969358583993fb6cbf20b3b806ca25a4edad9ce9c2bee"} Jul 16 13:08:41.807214 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:41.807166 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:41.807455 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:41.807305 2582 patch_prober.go:28] interesting pod/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 16 13:08:41.807455 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:41.807364 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" podUID="7a017ea1-3051-43f5-8794-b9ed3d60ab28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 16 13:08:42.806822 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:42.806775 2582 patch_prober.go:28] interesting pod/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 16 13:08:42.807304 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:42.806846 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" podUID="7a017ea1-3051-43f5-8794-b9ed3d60ab28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 16 13:08:43.806920 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:43.806875 2582 patch_prober.go:28] interesting pod/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 16 13:08:43.807434 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:43.806957 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" podUID="7a017ea1-3051-43f5-8794-b9ed3d60ab28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 16 13:08:44.316326 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:44.316285 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:08:44.419138 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:44.419038 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:08:44.812416 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:44.806658 2582 patch_prober.go:28] interesting pod/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 16 13:08:44.812416 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:44.806721 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" podUID="7a017ea1-3051-43f5-8794-b9ed3d60ab28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 16 13:08:45.809876 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:45.809842 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:45.810303 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:45.810285 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:45.810974 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:45.810951 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:08:46.047532 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:46.047492 2582 patch_prober.go:28] interesting pod/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 16 13:08:46.047932 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:46.047541 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" podUID="12e2f5b8-c482-43fa-a69e-57c639af1ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 16 13:08:48.099848 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:48.099813 2582 generic.go:356] "Generic (PLEG): container finished" podID="12e2f5b8-c482-43fa-a69e-57c639af1ff8" containerID="36c0d800f085d760c0a58a643c9c8e1e0fd752dd5038db61a11a65a4daa63988" exitCode=0 Jul 16 13:08:48.099848 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:48.099850 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" event={"ID":"12e2f5b8-c482-43fa-a69e-57c639af1ff8","Type":"ContainerDied","Data":"36c0d800f085d760c0a58a643c9c8e1e0fd752dd5038db61a11a65a4daa63988"} Jul 16 13:08:48.100291 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:48.100180 2582 scope.go:117] "RemoveContainer" containerID="36c0d800f085d760c0a58a643c9c8e1e0fd752dd5038db61a11a65a4daa63988" Jul 16 13:08:48.101279 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:48.101251 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:08:48.101349 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:48.101322 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:08:48.101402 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:48.101354 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:08:49.109686 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:49.109643 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" event={"ID":"12e2f5b8-c482-43fa-a69e-57c639af1ff8","Type":"ContainerStarted","Data":"125995e3425ef33ce153aace8a302f6dc47efa9912fa78901e048e884e836cbc"} Jul 16 13:08:49.397952 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:49.397912 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:49.398406 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:49.398357 2582 patch_prober.go:28] interesting pod/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 16 13:08:49.398641 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:49.398608 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" podUID="12e2f5b8-c482-43fa-a69e-57c639af1ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 16 13:08:50.397865 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:50.397827 2582 patch_prober.go:28] interesting pod/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 16 13:08:50.398438 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:50.398382 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" podUID="12e2f5b8-c482-43fa-a69e-57c639af1ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 16 13:08:51.398311 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:51.398275 2582 patch_prober.go:28] interesting pod/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 16 13:08:51.398903 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:51.398873 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" podUID="12e2f5b8-c482-43fa-a69e-57c639af1ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 16 13:08:52.179073 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:52.179040 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:08:52.321192 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:52.321119 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:08:52.398856 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:52.398774 2582 patch_prober.go:28] interesting pod/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 16 13:08:52.399414 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:52.399363 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" podUID="12e2f5b8-c482-43fa-a69e-57c639af1ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 16 13:08:53.401162 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:53.401132 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:53.401608 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:53.401424 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:53.402188 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:53.402170 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:08:58.580996 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:58.580960 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr"] Jul 16 13:08:58.583067 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:58.583048 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr" Jul 16 13:08:58.586513 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:58.586488 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ae18e9\"/\"default-dockercfg-5l7dz\"" Jul 16 13:08:58.586655 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:58.586545 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ae18e9\"/\"openshift-service-ca.crt\"" Jul 16 13:08:58.586655 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:58.586618 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ae18e9\"/\"kube-root-ca.crt\"" Jul 16 13:08:58.586881 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:58.586865 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ae18e9\"/\"test-model-ae18e9-kserve-self-signed-certs\"" Jul 16 13:08:58.593945 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:58.593922 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr"] Jul 16 13:08:58.617935 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:58.617911 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:08:58.673122 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:58.673095 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9ad7ef6-16fe-485b-b9d9-72262db8d504-model-cache\") pod \"test-model-ae18e9-kserve-74fdbc595-kr6mr\" (UID: \"a9ad7ef6-16fe-485b-b9d9-72262db8d504\") " pod="ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr" Jul 16 13:08:58.673122 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:58.673123 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9ad7ef6-16fe-485b-b9d9-72262db8d504-tls-certs\") pod \"test-model-ae18e9-kserve-74fdbc595-kr6mr\" (UID: \"a9ad7ef6-16fe-485b-b9d9-72262db8d504\") " pod="ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr" Jul 16 13:08:58.673275 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:58.673155 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9ad7ef6-16fe-485b-b9d9-72262db8d504-home\") pod \"test-model-ae18e9-kserve-74fdbc595-kr6mr\" (UID: \"a9ad7ef6-16fe-485b-b9d9-72262db8d504\") " pod="ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr" Jul 16 13:08:58.673275 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:58.673240 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jgp4\" (UniqueName: \"kubernetes.io/projected/a9ad7ef6-16fe-485b-b9d9-72262db8d504-kube-api-access-5jgp4\") pod \"test-model-ae18e9-kserve-74fdbc595-kr6mr\" (UID: \"a9ad7ef6-16fe-485b-b9d9-72262db8d504\") " pod="ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr" Jul 16 13:08:58.673405 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:58.673289 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9ad7ef6-16fe-485b-b9d9-72262db8d504-tmp-dir\") pod \"test-model-ae18e9-kserve-74fdbc595-kr6mr\" (UID: \"a9ad7ef6-16fe-485b-b9d9-72262db8d504\") " pod="ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr" Jul 16 13:08:58.673405 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:58.673319 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9ad7ef6-16fe-485b-b9d9-72262db8d504-dshm\") pod \"test-model-ae18e9-kserve-74fdbc595-kr6mr\" (UID: \"a9ad7ef6-16fe-485b-b9d9-72262db8d504\") " pod="ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr" Jul 16 13:08:58.774299 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:58.774268 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9ad7ef6-16fe-485b-b9d9-72262db8d504-model-cache\") pod \"test-model-ae18e9-kserve-74fdbc595-kr6mr\" (UID: \"a9ad7ef6-16fe-485b-b9d9-72262db8d504\") " pod="ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr" Jul 16 13:08:58.774299 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:58.774301 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9ad7ef6-16fe-485b-b9d9-72262db8d504-tls-certs\") pod \"test-model-ae18e9-kserve-74fdbc595-kr6mr\" (UID: \"a9ad7ef6-16fe-485b-b9d9-72262db8d504\") " pod="ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr" Jul 16 13:08:58.774587 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:58.774336 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9ad7ef6-16fe-485b-b9d9-72262db8d504-home\") pod \"test-model-ae18e9-kserve-74fdbc595-kr6mr\" (UID: \"a9ad7ef6-16fe-485b-b9d9-72262db8d504\") " pod="ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr" Jul 16 13:08:58.774855 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:58.774548 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jgp4\" (UniqueName: \"kubernetes.io/projected/a9ad7ef6-16fe-485b-b9d9-72262db8d504-kube-api-access-5jgp4\") pod \"test-model-ae18e9-kserve-74fdbc595-kr6mr\" (UID: \"a9ad7ef6-16fe-485b-b9d9-72262db8d504\") " pod="ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr" Jul 16 13:08:58.774950 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:58.774860 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9ad7ef6-16fe-485b-b9d9-72262db8d504-model-cache\") pod \"test-model-ae18e9-kserve-74fdbc595-kr6mr\" (UID: \"a9ad7ef6-16fe-485b-b9d9-72262db8d504\") " pod="ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr" Jul 16 13:08:58.774950 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:58.774814 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9ad7ef6-16fe-485b-b9d9-72262db8d504-home\") pod \"test-model-ae18e9-kserve-74fdbc595-kr6mr\" (UID: \"a9ad7ef6-16fe-485b-b9d9-72262db8d504\") " pod="ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr" Jul 16 13:08:58.774950 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:58.774908 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9ad7ef6-16fe-485b-b9d9-72262db8d504-tmp-dir\") pod \"test-model-ae18e9-kserve-74fdbc595-kr6mr\" (UID: \"a9ad7ef6-16fe-485b-b9d9-72262db8d504\") " pod="ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr" Jul 16 13:08:58.774950 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:58.774936 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9ad7ef6-16fe-485b-b9d9-72262db8d504-dshm\") pod \"test-model-ae18e9-kserve-74fdbc595-kr6mr\" (UID: \"a9ad7ef6-16fe-485b-b9d9-72262db8d504\") " pod="ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr" Jul 16 13:08:58.775222 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:58.775191 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9ad7ef6-16fe-485b-b9d9-72262db8d504-tmp-dir\") pod \"test-model-ae18e9-kserve-74fdbc595-kr6mr\" (UID: \"a9ad7ef6-16fe-485b-b9d9-72262db8d504\") " pod="ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr" Jul 16 13:08:58.777644 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:58.777622 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9ad7ef6-16fe-485b-b9d9-72262db8d504-dshm\") pod \"test-model-ae18e9-kserve-74fdbc595-kr6mr\" (UID: \"a9ad7ef6-16fe-485b-b9d9-72262db8d504\") " pod="ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr" Jul 16 13:08:58.777958 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:58.777937 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9ad7ef6-16fe-485b-b9d9-72262db8d504-tls-certs\") pod \"test-model-ae18e9-kserve-74fdbc595-kr6mr\" (UID: \"a9ad7ef6-16fe-485b-b9d9-72262db8d504\") " pod="ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr" Jul 16 13:08:58.782366 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:58.782337 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jgp4\" (UniqueName: \"kubernetes.io/projected/a9ad7ef6-16fe-485b-b9d9-72262db8d504-kube-api-access-5jgp4\") pod \"test-model-ae18e9-kserve-74fdbc595-kr6mr\" (UID: \"a9ad7ef6-16fe-485b-b9d9-72262db8d504\") " pod="ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr" Jul 16 13:08:58.892564 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:58.892539 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr" Jul 16 13:08:59.026348 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:59.026321 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr"] Jul 16 13:08:59.028485 ip-10-0-140-130 kubenswrapper[2582]: W0716 13:08:59.028456 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9ad7ef6_16fe_485b_b9d9_72262db8d504.slice/crio-e7a543adc32444963a1f10a38a7ec5e9d4e84520913d8002db22bb7bc276eb20 WatchSource:0}: Error finding container e7a543adc32444963a1f10a38a7ec5e9d4e84520913d8002db22bb7bc276eb20: Status 404 returned error can't find the container with id e7a543adc32444963a1f10a38a7ec5e9d4e84520913d8002db22bb7bc276eb20 Jul 16 13:08:59.165679 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:59.165592 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr" event={"ID":"a9ad7ef6-16fe-485b-b9d9-72262db8d504","Type":"ContainerStarted","Data":"f9436acb3e7d8a6a0d0754e303cbdb7fd5caba59aa8ce32c529db362aae12e1e"} Jul 16 13:08:59.165679 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:59.165632 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr" event={"ID":"a9ad7ef6-16fe-485b-b9d9-72262db8d504","Type":"ContainerStarted","Data":"e7a543adc32444963a1f10a38a7ec5e9d4e84520913d8002db22bb7bc276eb20"} Jul 16 13:08:59.187124 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:08:59.187075 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr" podStartSLOduration=1.187062254 podStartE2EDuration="1.187062254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:08:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:08:59.185730495 +0000 UTC m=+1797.663476122" watchObservedRunningTime="2026-07-16 13:08:59.187062254 +0000 UTC m=+1797.664807869" Jul 16 13:09:08.893368 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:08.893326 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr" Jul 16 13:09:08.893983 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:08.893411 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr" Jul 16 13:09:08.905916 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:08.905888 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr" Jul 16 13:09:09.224989 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:09.224894 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr" Jul 16 13:09:10.652772 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:10.652736 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:09:19.624852 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:19.624815 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:09:39.227410 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:39.227351 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr"] Jul 16 13:09:39.257465 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:39.257436 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:09:40.199816 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:40.199782 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:09:40.673669 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:40.673633 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:09:45.729943 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:45.729906 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:09:54.248528 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:54.248475 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr" podUID="a9ad7ef6-16fe-485b-b9d9-72262db8d504" containerName="main" containerID="cri-o://f9436acb3e7d8a6a0d0754e303cbdb7fd5caba59aa8ce32c529db362aae12e1e" gracePeriod=45 Jul 16 13:09:54.427726 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:54.427692 2582 generic.go:356] "Generic (PLEG): container finished" podID="a9ad7ef6-16fe-485b-b9d9-72262db8d504" containerID="f9436acb3e7d8a6a0d0754e303cbdb7fd5caba59aa8ce32c529db362aae12e1e" exitCode=0 Jul 16 13:09:54.427855 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:54.427765 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr" event={"ID":"a9ad7ef6-16fe-485b-b9d9-72262db8d504","Type":"ContainerDied","Data":"f9436acb3e7d8a6a0d0754e303cbdb7fd5caba59aa8ce32c529db362aae12e1e"} Jul 16 13:09:54.495882 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:54.495856 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr" Jul 16 13:09:54.548669 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:54.548597 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9ad7ef6-16fe-485b-b9d9-72262db8d504-tls-certs\") pod \"a9ad7ef6-16fe-485b-b9d9-72262db8d504\" (UID: \"a9ad7ef6-16fe-485b-b9d9-72262db8d504\") " Jul 16 13:09:54.548669 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:54.548637 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9ad7ef6-16fe-485b-b9d9-72262db8d504-model-cache\") pod \"a9ad7ef6-16fe-485b-b9d9-72262db8d504\" (UID: \"a9ad7ef6-16fe-485b-b9d9-72262db8d504\") " Jul 16 13:09:54.548867 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:54.548668 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9ad7ef6-16fe-485b-b9d9-72262db8d504-tmp-dir\") pod \"a9ad7ef6-16fe-485b-b9d9-72262db8d504\" (UID: \"a9ad7ef6-16fe-485b-b9d9-72262db8d504\") " Jul 16 13:09:54.548867 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:54.548708 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9ad7ef6-16fe-485b-b9d9-72262db8d504-dshm\") pod \"a9ad7ef6-16fe-485b-b9d9-72262db8d504\" (UID: \"a9ad7ef6-16fe-485b-b9d9-72262db8d504\") " Jul 16 13:09:54.548867 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:54.548732 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jgp4\" (UniqueName: \"kubernetes.io/projected/a9ad7ef6-16fe-485b-b9d9-72262db8d504-kube-api-access-5jgp4\") pod \"a9ad7ef6-16fe-485b-b9d9-72262db8d504\" (UID: \"a9ad7ef6-16fe-485b-b9d9-72262db8d504\") " Jul 16 13:09:54.548867 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:54.548763 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9ad7ef6-16fe-485b-b9d9-72262db8d504-home\") pod \"a9ad7ef6-16fe-485b-b9d9-72262db8d504\" (UID: \"a9ad7ef6-16fe-485b-b9d9-72262db8d504\") " Jul 16 13:09:54.549077 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:54.548911 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ad7ef6-16fe-485b-b9d9-72262db8d504-model-cache" (OuterVolumeSpecName: "model-cache") pod "a9ad7ef6-16fe-485b-b9d9-72262db8d504" (UID: "a9ad7ef6-16fe-485b-b9d9-72262db8d504"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:09:54.549077 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:54.549002 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ad7ef6-16fe-485b-b9d9-72262db8d504-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a9ad7ef6-16fe-485b-b9d9-72262db8d504" (UID: "a9ad7ef6-16fe-485b-b9d9-72262db8d504"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:09:54.549172 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:54.549072 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ad7ef6-16fe-485b-b9d9-72262db8d504-home" (OuterVolumeSpecName: "home") pod "a9ad7ef6-16fe-485b-b9d9-72262db8d504" (UID: "a9ad7ef6-16fe-485b-b9d9-72262db8d504"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:09:54.549172 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:54.549127 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9ad7ef6-16fe-485b-b9d9-72262db8d504-model-cache\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:09:54.549172 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:54.549144 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9ad7ef6-16fe-485b-b9d9-72262db8d504-tmp-dir\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:09:54.550931 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:54.550903 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ad7ef6-16fe-485b-b9d9-72262db8d504-dshm" (OuterVolumeSpecName: "dshm") pod "a9ad7ef6-16fe-485b-b9d9-72262db8d504" (UID: "a9ad7ef6-16fe-485b-b9d9-72262db8d504"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:09:54.551036 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:54.550945 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9ad7ef6-16fe-485b-b9d9-72262db8d504-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a9ad7ef6-16fe-485b-b9d9-72262db8d504" (UID: "a9ad7ef6-16fe-485b-b9d9-72262db8d504"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 13:09:54.551036 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:54.550990 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9ad7ef6-16fe-485b-b9d9-72262db8d504-kube-api-access-5jgp4" (OuterVolumeSpecName: "kube-api-access-5jgp4") pod "a9ad7ef6-16fe-485b-b9d9-72262db8d504" (UID: "a9ad7ef6-16fe-485b-b9d9-72262db8d504"). InnerVolumeSpecName "kube-api-access-5jgp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:09:54.649688 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:54.649652 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9ad7ef6-16fe-485b-b9d9-72262db8d504-tls-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:09:54.649688 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:54.649678 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9ad7ef6-16fe-485b-b9d9-72262db8d504-dshm\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:09:54.649688 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:54.649688 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jgp4\" (UniqueName: \"kubernetes.io/projected/a9ad7ef6-16fe-485b-b9d9-72262db8d504-kube-api-access-5jgp4\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:09:54.649897 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:54.649698 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9ad7ef6-16fe-485b-b9d9-72262db8d504-home\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:09:55.433299 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:55.433267 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr" Jul 16 13:09:55.433743 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:55.433295 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr" event={"ID":"a9ad7ef6-16fe-485b-b9d9-72262db8d504","Type":"ContainerDied","Data":"e7a543adc32444963a1f10a38a7ec5e9d4e84520913d8002db22bb7bc276eb20"} Jul 16 13:09:55.433743 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:55.433346 2582 scope.go:117] "RemoveContainer" containerID="f9436acb3e7d8a6a0d0754e303cbdb7fd5caba59aa8ce32c529db362aae12e1e" Jul 16 13:09:55.457350 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:55.454949 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr"] Jul 16 13:09:55.459993 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:55.459966 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ae18e9/test-model-ae18e9-kserve-74fdbc595-kr6mr"] Jul 16 13:09:55.976910 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:09:55.976879 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9ad7ef6-16fe-485b-b9d9-72262db8d504" path="/var/lib/kubelet/pods/a9ad7ef6-16fe-485b-b9d9-72262db8d504/volumes" Jul 16 13:10:23.091331 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:23.091239 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:10:23.330738 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:23.330705 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m"] Jul 16 13:10:23.330968 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:23.330944 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" podUID="7a017ea1-3051-43f5-8794-b9ed3d60ab28" containerName="istio-proxy" containerID="cri-o://2b4a16aa69d5d9f0adf969358583993fb6cbf20b3b806ca25a4edad9ce9c2bee" gracePeriod=30 Jul 16 13:10:23.352536 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:23.352459 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:10:23.588637 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:23.588599 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:10:24.159258 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:24.159220 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:10:28.572626 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.572602 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:10:28.600812 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.600738 2582 generic.go:356] "Generic (PLEG): container finished" podID="7a017ea1-3051-43f5-8794-b9ed3d60ab28" containerID="2b4a16aa69d5d9f0adf969358583993fb6cbf20b3b806ca25a4edad9ce9c2bee" exitCode=0 Jul 16 13:10:28.600812 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.600783 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" Jul 16 13:10:28.600963 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.600786 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" event={"ID":"7a017ea1-3051-43f5-8794-b9ed3d60ab28","Type":"ContainerDied","Data":"2b4a16aa69d5d9f0adf969358583993fb6cbf20b3b806ca25a4edad9ce9c2bee"} Jul 16 13:10:28.600963 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.600914 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m" event={"ID":"7a017ea1-3051-43f5-8794-b9ed3d60ab28","Type":"ContainerDied","Data":"b2506bcea3f284c7c6fd76a4ba518bbc86c636f6de9f7478162582443adfc401"} Jul 16 13:10:28.600963 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.600948 2582 scope.go:117] "RemoveContainer" containerID="2b4a16aa69d5d9f0adf969358583993fb6cbf20b3b806ca25a4edad9ce9c2bee" Jul 16 13:10:28.609581 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.609557 2582 scope.go:117] "RemoveContainer" containerID="5b2b99c1313c330caab567cebc178a978dfb78a8ed4035a94f5c3c49b7d918c2" Jul 16 13:10:28.617565 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.617549 2582 scope.go:117] "RemoveContainer" containerID="2b4a16aa69d5d9f0adf969358583993fb6cbf20b3b806ca25a4edad9ce9c2bee" Jul 16 13:10:28.617801 ip-10-0-140-130 kubenswrapper[2582]: E0716 13:10:28.617784 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b4a16aa69d5d9f0adf969358583993fb6cbf20b3b806ca25a4edad9ce9c2bee\": container with ID starting with 2b4a16aa69d5d9f0adf969358583993fb6cbf20b3b806ca25a4edad9ce9c2bee not found: ID does not exist" containerID="2b4a16aa69d5d9f0adf969358583993fb6cbf20b3b806ca25a4edad9ce9c2bee" Jul 16 13:10:28.617878 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.617810 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b4a16aa69d5d9f0adf969358583993fb6cbf20b3b806ca25a4edad9ce9c2bee"} err="failed to get container status \"2b4a16aa69d5d9f0adf969358583993fb6cbf20b3b806ca25a4edad9ce9c2bee\": rpc error: code = NotFound desc = could not find container \"2b4a16aa69d5d9f0adf969358583993fb6cbf20b3b806ca25a4edad9ce9c2bee\": container with ID starting with 2b4a16aa69d5d9f0adf969358583993fb6cbf20b3b806ca25a4edad9ce9c2bee not found: ID does not exist" Jul 16 13:10:28.617878 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.617831 2582 scope.go:117] "RemoveContainer" containerID="5b2b99c1313c330caab567cebc178a978dfb78a8ed4035a94f5c3c49b7d918c2" Jul 16 13:10:28.618055 ip-10-0-140-130 kubenswrapper[2582]: E0716 13:10:28.618040 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b2b99c1313c330caab567cebc178a978dfb78a8ed4035a94f5c3c49b7d918c2\": container with ID starting with 5b2b99c1313c330caab567cebc178a978dfb78a8ed4035a94f5c3c49b7d918c2 not found: ID does not exist" containerID="5b2b99c1313c330caab567cebc178a978dfb78a8ed4035a94f5c3c49b7d918c2" Jul 16 13:10:28.618094 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.618059 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b2b99c1313c330caab567cebc178a978dfb78a8ed4035a94f5c3c49b7d918c2"} err="failed to get container status \"5b2b99c1313c330caab567cebc178a978dfb78a8ed4035a94f5c3c49b7d918c2\": rpc error: code = NotFound desc = could not find container \"5b2b99c1313c330caab567cebc178a978dfb78a8ed4035a94f5c3c49b7d918c2\": container with ID starting with 5b2b99c1313c330caab567cebc178a978dfb78a8ed4035a94f5c3c49b7d918c2 not found: ID does not exist" Jul 16 13:10:28.624946 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.624929 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a017ea1-3051-43f5-8794-b9ed3d60ab28-credential-socket\") pod \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " Jul 16 13:10:28.625014 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.624954 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a017ea1-3051-43f5-8794-b9ed3d60ab28-istio-envoy\") pod \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " Jul 16 13:10:28.625014 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.624980 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a017ea1-3051-43f5-8794-b9ed3d60ab28-workload-certs\") pod \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " Jul 16 13:10:28.625102 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.625015 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a017ea1-3051-43f5-8794-b9ed3d60ab28-istio-token\") pod \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " Jul 16 13:10:28.625102 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.625043 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a017ea1-3051-43f5-8794-b9ed3d60ab28-istio-podinfo\") pod \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " Jul 16 13:10:28.625102 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.625070 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a017ea1-3051-43f5-8794-b9ed3d60ab28-istio-data\") pod \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " Jul 16 13:10:28.625232 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.625101 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jpjf\" (UniqueName: \"kubernetes.io/projected/7a017ea1-3051-43f5-8794-b9ed3d60ab28-kube-api-access-5jpjf\") pod \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " Jul 16 13:10:28.625232 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.625131 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a017ea1-3051-43f5-8794-b9ed3d60ab28-workload-socket\") pod \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " Jul 16 13:10:28.625232 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.625163 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a017ea1-3051-43f5-8794-b9ed3d60ab28-istiod-ca-cert\") pod \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\" (UID: \"7a017ea1-3051-43f5-8794-b9ed3d60ab28\") " Jul 16 13:10:28.625232 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.625171 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a017ea1-3051-43f5-8794-b9ed3d60ab28-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7a017ea1-3051-43f5-8794-b9ed3d60ab28" (UID: "7a017ea1-3051-43f5-8794-b9ed3d60ab28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:10:28.625514 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.625376 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a017ea1-3051-43f5-8794-b9ed3d60ab28-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7a017ea1-3051-43f5-8794-b9ed3d60ab28" (UID: "7a017ea1-3051-43f5-8794-b9ed3d60ab28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:10:28.625514 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.625412 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a017ea1-3051-43f5-8794-b9ed3d60ab28-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7a017ea1-3051-43f5-8794-b9ed3d60ab28" (UID: "7a017ea1-3051-43f5-8794-b9ed3d60ab28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:10:28.625635 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.625535 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a017ea1-3051-43f5-8794-b9ed3d60ab28-workload-socket\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:10:28.625635 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.625607 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a017ea1-3051-43f5-8794-b9ed3d60ab28-credential-socket\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:10:28.625635 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.625623 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a017ea1-3051-43f5-8794-b9ed3d60ab28-workload-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:10:28.625820 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.625671 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a017ea1-3051-43f5-8794-b9ed3d60ab28-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7a017ea1-3051-43f5-8794-b9ed3d60ab28" (UID: "7a017ea1-3051-43f5-8794-b9ed3d60ab28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:10:28.626672 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.626644 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a017ea1-3051-43f5-8794-b9ed3d60ab28-istio-data" (OuterVolumeSpecName: "istio-data") pod "7a017ea1-3051-43f5-8794-b9ed3d60ab28" (UID: "7a017ea1-3051-43f5-8794-b9ed3d60ab28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:10:28.627295 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.627265 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a017ea1-3051-43f5-8794-b9ed3d60ab28-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7a017ea1-3051-43f5-8794-b9ed3d60ab28" (UID: "7a017ea1-3051-43f5-8794-b9ed3d60ab28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:10:28.627441 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.627366 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a017ea1-3051-43f5-8794-b9ed3d60ab28-kube-api-access-5jpjf" (OuterVolumeSpecName: "kube-api-access-5jpjf") pod "7a017ea1-3051-43f5-8794-b9ed3d60ab28" (UID: "7a017ea1-3051-43f5-8794-b9ed3d60ab28"). InnerVolumeSpecName "kube-api-access-5jpjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:10:28.627686 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.627657 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a017ea1-3051-43f5-8794-b9ed3d60ab28-istio-token" (OuterVolumeSpecName: "istio-token") pod "7a017ea1-3051-43f5-8794-b9ed3d60ab28" (UID: "7a017ea1-3051-43f5-8794-b9ed3d60ab28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:10:28.627900 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.627876 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a017ea1-3051-43f5-8794-b9ed3d60ab28-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7a017ea1-3051-43f5-8794-b9ed3d60ab28" (UID: "7a017ea1-3051-43f5-8794-b9ed3d60ab28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 13:10:28.725973 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.725928 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a017ea1-3051-43f5-8794-b9ed3d60ab28-istio-envoy\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:10:28.725973 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.725968 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a017ea1-3051-43f5-8794-b9ed3d60ab28-istio-token\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:10:28.725973 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.725978 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a017ea1-3051-43f5-8794-b9ed3d60ab28-istio-podinfo\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:10:28.725973 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.725986 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a017ea1-3051-43f5-8794-b9ed3d60ab28-istio-data\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:10:28.726224 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.725996 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jpjf\" (UniqueName: \"kubernetes.io/projected/7a017ea1-3051-43f5-8794-b9ed3d60ab28-kube-api-access-5jpjf\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:10:28.726224 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.726005 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a017ea1-3051-43f5-8794-b9ed3d60ab28-istiod-ca-cert\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:10:28.925097 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.925070 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m"] Jul 16 13:10:28.932605 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:28.932579 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ae18e9-openshift-default-867d77fd5f-ddn2m"] Jul 16 13:10:29.977220 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:29.977175 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a017ea1-3051-43f5-8794-b9ed3d60ab28" path="/var/lib/kubelet/pods/7a017ea1-3051-43f5-8794-b9ed3d60ab28/volumes" Jul 16 13:10:30.215417 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:30.215357 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:10:30.317439 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:30.317324 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:10:30.474683 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:30.474649 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4"] Jul 16 13:10:30.474867 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:30.474842 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" podUID="12e2f5b8-c482-43fa-a69e-57c639af1ff8" containerName="istio-proxy" containerID="cri-o://125995e3425ef33ce153aace8a302f6dc47efa9912fa78901e048e884e836cbc" gracePeriod=30 Jul 16 13:10:30.516052 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:30.516023 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:10:30.724683 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:30.724649 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:10:31.087159 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.087047 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:10:31.374073 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.373989 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq"] Jul 16 13:10:31.374366 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.374348 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a017ea1-3051-43f5-8794-b9ed3d60ab28" containerName="istio-proxy" Jul 16 13:10:31.374481 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.374369 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a017ea1-3051-43f5-8794-b9ed3d60ab28" containerName="istio-proxy" Jul 16 13:10:31.374481 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.374422 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a017ea1-3051-43f5-8794-b9ed3d60ab28" containerName="istio-proxy" Jul 16 13:10:31.374481 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.374432 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a017ea1-3051-43f5-8794-b9ed3d60ab28" containerName="istio-proxy" Jul 16 13:10:31.374481 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.374444 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9ad7ef6-16fe-485b-b9d9-72262db8d504" containerName="main" Jul 16 13:10:31.374481 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.374452 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9ad7ef6-16fe-485b-b9d9-72262db8d504" containerName="main" Jul 16 13:10:31.374723 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.374536 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9ad7ef6-16fe-485b-b9d9-72262db8d504" containerName="main" Jul 16 13:10:31.374723 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.374552 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a017ea1-3051-43f5-8794-b9ed3d60ab28" containerName="istio-proxy" Jul 16 13:10:31.374723 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.374564 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a017ea1-3051-43f5-8794-b9ed3d60ab28" containerName="istio-proxy" Jul 16 13:10:31.379153 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.379131 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:31.381434 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.381409 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-213b02-openshift-default-dockercfg-95xfz\"" Jul 16 13:10:31.388809 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.388787 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq"] Jul 16 13:10:31.413199 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.413172 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:10:31.450834 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.450798 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-istio-data\") pod \"e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:31.450964 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.450845 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-istio-token\") pod \"e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:31.450964 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.450926 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-istio-podinfo\") pod \"e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:31.451083 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.450970 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-istiod-ca-cert\") pod \"e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:31.451083 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.450993 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-workload-socket\") pod \"e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:31.451083 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.451071 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-credential-socket\") pod \"e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:31.451250 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.451107 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-workload-certs\") pod \"e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:31.451250 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.451135 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6znp2\" (UniqueName: \"kubernetes.io/projected/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-kube-api-access-6znp2\") pod \"e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:31.451250 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.451190 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-istio-envoy\") pod \"e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:31.510466 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.510434 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:10:31.552507 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.552471 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-istio-podinfo\") pod \"e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:31.552648 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.552556 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-istiod-ca-cert\") pod \"e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:31.552648 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.552587 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-workload-socket\") pod \"e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:31.552648 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.552638 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-credential-socket\") pod \"e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:31.552814 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.552664 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-workload-certs\") pod \"e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:31.552814 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.552691 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6znp2\" (UniqueName: \"kubernetes.io/projected/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-kube-api-access-6znp2\") pod \"e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:31.552814 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.552752 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-istio-envoy\") pod \"e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:31.552814 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.552801 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-istio-data\") pod \"e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:31.553042 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.552832 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-istio-token\") pod \"e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:31.553155 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.553130 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-workload-socket\") pod \"e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:31.553451 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.553423 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-istiod-ca-cert\") pod \"e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:31.553590 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.553522 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-credential-socket\") pod \"e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:31.553765 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.553743 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-istio-data\") pod \"e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:31.553913 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.553896 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-workload-certs\") pod \"e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:31.555715 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.555690 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-istio-podinfo\") pod \"e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:31.555871 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.555856 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-istio-envoy\") pod \"e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:31.564510 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.564484 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-istio-token\") pod \"e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:31.565044 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.565021 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6znp2\" (UniqueName: \"kubernetes.io/projected/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-kube-api-access-6znp2\") pod \"e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:31.690499 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.690469 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:31.830649 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.830624 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq"] Jul 16 13:10:31.832045 ip-10-0-140-130 kubenswrapper[2582]: W0716 13:10:31.832019 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ef90b2_69d9_4c1d_b7fa_cecdfca899dc.slice/crio-c6f2ad95bc90023b5c789ee7b9b275a584afb87b5765b062d8f7ced423938269 WatchSource:0}: Error finding container c6f2ad95bc90023b5c789ee7b9b275a584afb87b5765b062d8f7ced423938269: Status 404 returned error can't find the container with id c6f2ad95bc90023b5c789ee7b9b275a584afb87b5765b062d8f7ced423938269 Jul 16 13:10:31.834123 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.834095 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:10:31.834178 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.834156 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:10:31.834217 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:31.834184 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:10:32.620938 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:32.620899 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" event={"ID":"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc","Type":"ContainerStarted","Data":"fe976bfdc12f4065efe6e12e5a9322c34ecb68b17b6ebc67481d58148c1907a2"} Jul 16 13:10:32.621339 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:32.620950 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" event={"ID":"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc","Type":"ContainerStarted","Data":"c6f2ad95bc90023b5c789ee7b9b275a584afb87b5765b062d8f7ced423938269"} Jul 16 13:10:32.647057 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:32.647008 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" podStartSLOduration=1.6469929479999998 podStartE2EDuration="1.646992948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:10:32.643875901 +0000 UTC m=+1891.121621517" watchObservedRunningTime="2026-07-16 13:10:32.646992948 +0000 UTC m=+1891.124738564" Jul 16 13:10:32.690891 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:32.690863 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:32.695555 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:32.695530 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:33.626259 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:33.626216 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:33.627163 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:33.627142 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:33.667482 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:33.667452 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:10:33.762692 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:33.762653 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:10:33.916076 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:33.915986 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:10:34.100660 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:34.100628 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:10:34.364668 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:34.364578 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:10:34.467511 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:34.467483 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:10:35.640667 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:35.640637 2582 generic.go:356] "Generic (PLEG): container finished" podID="12e2f5b8-c482-43fa-a69e-57c639af1ff8" containerID="125995e3425ef33ce153aace8a302f6dc47efa9912fa78901e048e884e836cbc" exitCode=0 Jul 16 13:10:35.641066 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:35.640711 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" event={"ID":"12e2f5b8-c482-43fa-a69e-57c639af1ff8","Type":"ContainerDied","Data":"125995e3425ef33ce153aace8a302f6dc47efa9912fa78901e048e884e836cbc"} Jul 16 13:10:35.641066 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:35.640775 2582 scope.go:117] "RemoveContainer" containerID="36c0d800f085d760c0a58a643c9c8e1e0fd752dd5038db61a11a65a4daa63988" Jul 16 13:10:35.731472 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:35.731450 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:10:35.790348 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:35.790276 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12e2f5b8-c482-43fa-a69e-57c639af1ff8-istio-envoy\") pod \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " Jul 16 13:10:35.790348 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:35.790320 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e2f5b8-c482-43fa-a69e-57c639af1ff8-workload-certs\") pod \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " Jul 16 13:10:35.790595 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:35.790359 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12e2f5b8-c482-43fa-a69e-57c639af1ff8-istio-podinfo\") pod \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " Jul 16 13:10:35.790595 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:35.790449 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e2f5b8-c482-43fa-a69e-57c639af1ff8-credential-socket\") pod \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " Jul 16 13:10:35.790595 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:35.790487 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12e2f5b8-c482-43fa-a69e-57c639af1ff8-istio-data\") pod \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " Jul 16 13:10:35.790595 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:35.790510 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12e2f5b8-c482-43fa-a69e-57c639af1ff8-istiod-ca-cert\") pod \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " Jul 16 13:10:35.790595 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:35.790548 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12e2f5b8-c482-43fa-a69e-57c639af1ff8-istio-token\") pod \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " Jul 16 13:10:35.790595 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:35.790575 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e2f5b8-c482-43fa-a69e-57c639af1ff8-workload-socket\") pod \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " Jul 16 13:10:35.790877 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:35.790605 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6bcn\" (UniqueName: \"kubernetes.io/projected/12e2f5b8-c482-43fa-a69e-57c639af1ff8-kube-api-access-m6bcn\") pod \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\" (UID: \"12e2f5b8-c482-43fa-a69e-57c639af1ff8\") " Jul 16 13:10:35.790877 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:35.790647 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e2f5b8-c482-43fa-a69e-57c639af1ff8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "12e2f5b8-c482-43fa-a69e-57c639af1ff8" (UID: "12e2f5b8-c482-43fa-a69e-57c639af1ff8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:10:35.790877 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:35.790767 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e2f5b8-c482-43fa-a69e-57c639af1ff8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "12e2f5b8-c482-43fa-a69e-57c639af1ff8" (UID: "12e2f5b8-c482-43fa-a69e-57c639af1ff8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:10:35.791035 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:35.790879 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e2f5b8-c482-43fa-a69e-57c639af1ff8-workload-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:10:35.791035 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:35.790897 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e2f5b8-c482-43fa-a69e-57c639af1ff8-credential-socket\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:10:35.791035 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:35.790892 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12e2f5b8-c482-43fa-a69e-57c639af1ff8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "12e2f5b8-c482-43fa-a69e-57c639af1ff8" (UID: "12e2f5b8-c482-43fa-a69e-57c639af1ff8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:10:35.791252 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:35.791222 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e2f5b8-c482-43fa-a69e-57c639af1ff8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "12e2f5b8-c482-43fa-a69e-57c639af1ff8" (UID: "12e2f5b8-c482-43fa-a69e-57c639af1ff8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:10:35.791770 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:35.791745 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e2f5b8-c482-43fa-a69e-57c639af1ff8-istio-data" (OuterVolumeSpecName: "istio-data") pod "12e2f5b8-c482-43fa-a69e-57c639af1ff8" (UID: "12e2f5b8-c482-43fa-a69e-57c639af1ff8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:10:35.793056 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:35.793036 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e2f5b8-c482-43fa-a69e-57c639af1ff8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "12e2f5b8-c482-43fa-a69e-57c639af1ff8" (UID: "12e2f5b8-c482-43fa-a69e-57c639af1ff8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:10:35.793056 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:35.793039 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e2f5b8-c482-43fa-a69e-57c639af1ff8-kube-api-access-m6bcn" (OuterVolumeSpecName: "kube-api-access-m6bcn") pod "12e2f5b8-c482-43fa-a69e-57c639af1ff8" (UID: "12e2f5b8-c482-43fa-a69e-57c639af1ff8"). InnerVolumeSpecName "kube-api-access-m6bcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:10:35.793327 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:35.793300 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e2f5b8-c482-43fa-a69e-57c639af1ff8-istio-token" (OuterVolumeSpecName: "istio-token") pod "12e2f5b8-c482-43fa-a69e-57c639af1ff8" (UID: "12e2f5b8-c482-43fa-a69e-57c639af1ff8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:10:35.793469 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:35.793358 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12e2f5b8-c482-43fa-a69e-57c639af1ff8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "12e2f5b8-c482-43fa-a69e-57c639af1ff8" (UID: "12e2f5b8-c482-43fa-a69e-57c639af1ff8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 13:10:35.891469 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:35.891437 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12e2f5b8-c482-43fa-a69e-57c639af1ff8-istio-podinfo\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:10:35.891469 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:35.891463 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12e2f5b8-c482-43fa-a69e-57c639af1ff8-istio-data\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:10:35.891469 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:35.891473 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12e2f5b8-c482-43fa-a69e-57c639af1ff8-istiod-ca-cert\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:10:35.891692 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:35.891481 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12e2f5b8-c482-43fa-a69e-57c639af1ff8-istio-token\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:10:35.891692 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:35.891489 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e2f5b8-c482-43fa-a69e-57c639af1ff8-workload-socket\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:10:35.891692 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:35.891497 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6bcn\" (UniqueName: \"kubernetes.io/projected/12e2f5b8-c482-43fa-a69e-57c639af1ff8-kube-api-access-m6bcn\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:10:35.891692 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:35.891509 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12e2f5b8-c482-43fa-a69e-57c639af1ff8-istio-envoy\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:10:36.646281 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:36.646247 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" Jul 16 13:10:36.646772 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:36.646278 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4" event={"ID":"12e2f5b8-c482-43fa-a69e-57c639af1ff8","Type":"ContainerDied","Data":"8702c0d478f18342a91c8bbbd24b34ee14533cdafd5b93b8e491c5b061fe9b56"} Jul 16 13:10:36.646772 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:36.646329 2582 scope.go:117] "RemoveContainer" containerID="125995e3425ef33ce153aace8a302f6dc47efa9912fa78901e048e884e836cbc" Jul 16 13:10:36.668185 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:36.668159 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4"] Jul 16 13:10:36.672462 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:36.672442 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-cb4eee-openshift-default-597b95499-87xc4"] Jul 16 13:10:37.976936 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:37.976904 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12e2f5b8-c482-43fa-a69e-57c639af1ff8" path="/var/lib/kubelet/pods/12e2f5b8-c482-43fa-a69e-57c639af1ff8/volumes" Jul 16 13:10:39.317435 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:39.317382 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:10:39.728619 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:39.728580 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:10:40.007322 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.007247 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv"] Jul 16 13:10:40.007567 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.007555 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12e2f5b8-c482-43fa-a69e-57c639af1ff8" containerName="istio-proxy" Jul 16 13:10:40.007622 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.007569 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="12e2f5b8-c482-43fa-a69e-57c639af1ff8" containerName="istio-proxy" Jul 16 13:10:40.007658 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.007641 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="12e2f5b8-c482-43fa-a69e-57c639af1ff8" containerName="istio-proxy" Jul 16 13:10:40.007658 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.007651 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="12e2f5b8-c482-43fa-a69e-57c639af1ff8" containerName="istio-proxy" Jul 16 13:10:40.011747 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.011730 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:40.014354 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.014328 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a22377-openshift-default-dockercfg-b9jj8\"" Jul 16 13:10:40.023113 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.023089 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv"] Jul 16 13:10:40.037976 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.037956 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:10:40.126100 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.126068 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9106e0bd-ef66-4211-bc4b-3305e41f991e-istio-podinfo\") pod \"e2e-api-b-a22377-openshift-default-67645c967c-wptcv\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:40.126265 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.126122 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9106e0bd-ef66-4211-bc4b-3305e41f991e-istio-envoy\") pod \"e2e-api-b-a22377-openshift-default-67645c967c-wptcv\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:40.126265 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.126217 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9106e0bd-ef66-4211-bc4b-3305e41f991e-istio-data\") pod \"e2e-api-b-a22377-openshift-default-67645c967c-wptcv\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:40.126346 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.126266 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9106e0bd-ef66-4211-bc4b-3305e41f991e-workload-certs\") pod \"e2e-api-b-a22377-openshift-default-67645c967c-wptcv\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:40.126346 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.126327 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9106e0bd-ef66-4211-bc4b-3305e41f991e-workload-socket\") pod \"e2e-api-b-a22377-openshift-default-67645c967c-wptcv\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:40.126434 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.126418 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9106e0bd-ef66-4211-bc4b-3305e41f991e-istio-token\") pod \"e2e-api-b-a22377-openshift-default-67645c967c-wptcv\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:40.126475 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.126456 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9106e0bd-ef66-4211-bc4b-3305e41f991e-istiod-ca-cert\") pod \"e2e-api-b-a22377-openshift-default-67645c967c-wptcv\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:40.126513 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.126499 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9106e0bd-ef66-4211-bc4b-3305e41f991e-credential-socket\") pod \"e2e-api-b-a22377-openshift-default-67645c967c-wptcv\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:40.126549 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.126527 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4dvl\" (UniqueName: \"kubernetes.io/projected/9106e0bd-ef66-4211-bc4b-3305e41f991e-kube-api-access-z4dvl\") pod \"e2e-api-b-a22377-openshift-default-67645c967c-wptcv\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:40.227354 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.227321 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9106e0bd-ef66-4211-bc4b-3305e41f991e-istio-podinfo\") pod \"e2e-api-b-a22377-openshift-default-67645c967c-wptcv\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:40.227354 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.227361 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9106e0bd-ef66-4211-bc4b-3305e41f991e-istio-envoy\") pod \"e2e-api-b-a22377-openshift-default-67645c967c-wptcv\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:40.227623 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.227384 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9106e0bd-ef66-4211-bc4b-3305e41f991e-istio-data\") pod \"e2e-api-b-a22377-openshift-default-67645c967c-wptcv\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:40.227623 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.227424 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9106e0bd-ef66-4211-bc4b-3305e41f991e-workload-certs\") pod \"e2e-api-b-a22377-openshift-default-67645c967c-wptcv\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:40.227623 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.227445 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9106e0bd-ef66-4211-bc4b-3305e41f991e-workload-socket\") pod \"e2e-api-b-a22377-openshift-default-67645c967c-wptcv\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:40.227623 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.227468 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9106e0bd-ef66-4211-bc4b-3305e41f991e-istio-token\") pod \"e2e-api-b-a22377-openshift-default-67645c967c-wptcv\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:40.227623 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.227488 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9106e0bd-ef66-4211-bc4b-3305e41f991e-istiod-ca-cert\") pod \"e2e-api-b-a22377-openshift-default-67645c967c-wptcv\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:40.227623 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.227513 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9106e0bd-ef66-4211-bc4b-3305e41f991e-credential-socket\") pod \"e2e-api-b-a22377-openshift-default-67645c967c-wptcv\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:40.227623 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.227536 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4dvl\" (UniqueName: \"kubernetes.io/projected/9106e0bd-ef66-4211-bc4b-3305e41f991e-kube-api-access-z4dvl\") pod \"e2e-api-b-a22377-openshift-default-67645c967c-wptcv\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:40.227992 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.227888 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9106e0bd-ef66-4211-bc4b-3305e41f991e-workload-certs\") pod \"e2e-api-b-a22377-openshift-default-67645c967c-wptcv\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:40.227992 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.227943 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9106e0bd-ef66-4211-bc4b-3305e41f991e-istio-data\") pod \"e2e-api-b-a22377-openshift-default-67645c967c-wptcv\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:40.228129 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.228111 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9106e0bd-ef66-4211-bc4b-3305e41f991e-workload-socket\") pod \"e2e-api-b-a22377-openshift-default-67645c967c-wptcv\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:40.228171 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.228116 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9106e0bd-ef66-4211-bc4b-3305e41f991e-credential-socket\") pod \"e2e-api-b-a22377-openshift-default-67645c967c-wptcv\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:40.228433 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.228409 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9106e0bd-ef66-4211-bc4b-3305e41f991e-istiod-ca-cert\") pod \"e2e-api-b-a22377-openshift-default-67645c967c-wptcv\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:40.229955 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.229934 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9106e0bd-ef66-4211-bc4b-3305e41f991e-istio-envoy\") pod \"e2e-api-b-a22377-openshift-default-67645c967c-wptcv\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:40.230025 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.229955 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9106e0bd-ef66-4211-bc4b-3305e41f991e-istio-podinfo\") pod \"e2e-api-b-a22377-openshift-default-67645c967c-wptcv\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:40.234884 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.234862 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9106e0bd-ef66-4211-bc4b-3305e41f991e-istio-token\") pod \"e2e-api-b-a22377-openshift-default-67645c967c-wptcv\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:40.234993 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.234976 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4dvl\" (UniqueName: \"kubernetes.io/projected/9106e0bd-ef66-4211-bc4b-3305e41f991e-kube-api-access-z4dvl\") pod \"e2e-api-b-a22377-openshift-default-67645c967c-wptcv\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:40.322046 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.321973 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:40.457248 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.457217 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv"] Jul 16 13:10:40.458402 ip-10-0-140-130 kubenswrapper[2582]: W0716 13:10:40.458366 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9106e0bd_ef66_4211_bc4b_3305e41f991e.slice/crio-d227b81c51bea5c5a9f21bb5da1765b72fc7d4d0bf1cffdd9cf8eda981db8bda WatchSource:0}: Error finding container d227b81c51bea5c5a9f21bb5da1765b72fc7d4d0bf1cffdd9cf8eda981db8bda: Status 404 returned error can't find the container with id d227b81c51bea5c5a9f21bb5da1765b72fc7d4d0bf1cffdd9cf8eda981db8bda Jul 16 13:10:40.460470 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.460438 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:10:40.460568 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.460517 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:10:40.460568 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.460560 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:10:40.464150 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.464122 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:10:40.667869 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.667833 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" event={"ID":"9106e0bd-ef66-4211-bc4b-3305e41f991e","Type":"ContainerStarted","Data":"66be56e92d47fce6dfc5bdefe0f5209d199e6298e6c7dd808b0779fdaeb9978f"} Jul 16 13:10:40.668069 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.667891 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" event={"ID":"9106e0bd-ef66-4211-bc4b-3305e41f991e","Type":"ContainerStarted","Data":"d227b81c51bea5c5a9f21bb5da1765b72fc7d4d0bf1cffdd9cf8eda981db8bda"} Jul 16 13:10:40.687137 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:40.687077 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" podStartSLOduration=1.6870586969999999 podStartE2EDuration="1.687058697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:10:40.686379122 +0000 UTC m=+1899.164124769" watchObservedRunningTime="2026-07-16 13:10:40.687058697 +0000 UTC m=+1899.164804314" Jul 16 13:10:41.322471 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:41.322438 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:41.327403 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:41.327362 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:41.584581 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:41.584468 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:10:41.678175 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:41.678134 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:41.679097 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:41.679073 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:41.710673 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:41.710621 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:10:41.824504 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:41.824474 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:10:41.990010 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:41.989974 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:10:42.176788 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:42.176751 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:10:42.296991 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:42.296900 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:10:47.268989 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:47.268956 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:10:49.669560 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:49.669509 2582 patch_prober.go:28] interesting pod/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 16 13:10:49.669949 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:49.669571 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" podUID="28ef90b2-69d9-4c1d-b7fa-cecdfca899dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 16 13:10:50.735271 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:50.735233 2582 generic.go:356] "Generic (PLEG): container finished" podID="28ef90b2-69d9-4c1d-b7fa-cecdfca899dc" containerID="fe976bfdc12f4065efe6e12e5a9322c34ecb68b17b6ebc67481d58148c1907a2" exitCode=0 Jul 16 13:10:50.735726 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:50.735310 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" event={"ID":"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc","Type":"ContainerDied","Data":"fe976bfdc12f4065efe6e12e5a9322c34ecb68b17b6ebc67481d58148c1907a2"} Jul 16 13:10:50.735726 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:50.735702 2582 scope.go:117] "RemoveContainer" containerID="fe976bfdc12f4065efe6e12e5a9322c34ecb68b17b6ebc67481d58148c1907a2" Jul 16 13:10:50.736637 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:50.736608 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:10:50.736748 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:50.736690 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:10:50.736748 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:50.736721 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:10:51.742671 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:51.742636 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" event={"ID":"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc","Type":"ContainerStarted","Data":"95865d7c9b523bef3c49d3809ff128cb1fbaa5152845a1189e50ebf9d17bba37"} Jul 16 13:10:52.690627 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:52.690587 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:52.690842 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:52.690821 2582 patch_prober.go:28] interesting pod/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 16 13:10:52.690923 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:52.690874 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" podUID="28ef90b2-69d9-4c1d-b7fa-cecdfca899dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 16 13:10:53.691493 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:53.691452 2582 patch_prober.go:28] interesting pod/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 16 13:10:53.692046 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:53.691513 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" podUID="28ef90b2-69d9-4c1d-b7fa-cecdfca899dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 16 13:10:54.691370 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:54.691335 2582 patch_prober.go:28] interesting pod/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 16 13:10:54.691604 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:54.691430 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" podUID="28ef90b2-69d9-4c1d-b7fa-cecdfca899dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 16 13:10:54.759255 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:54.759211 2582 generic.go:356] "Generic (PLEG): container finished" podID="9106e0bd-ef66-4211-bc4b-3305e41f991e" containerID="66be56e92d47fce6dfc5bdefe0f5209d199e6298e6c7dd808b0779fdaeb9978f" exitCode=0 Jul 16 13:10:54.759476 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:54.759284 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" event={"ID":"9106e0bd-ef66-4211-bc4b-3305e41f991e","Type":"ContainerDied","Data":"66be56e92d47fce6dfc5bdefe0f5209d199e6298e6c7dd808b0779fdaeb9978f"} Jul 16 13:10:54.759809 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:54.759786 2582 scope.go:117] "RemoveContainer" containerID="66be56e92d47fce6dfc5bdefe0f5209d199e6298e6c7dd808b0779fdaeb9978f" Jul 16 13:10:54.761038 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:54.761010 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:10:54.761145 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:54.761077 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:10:54.761145 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:54.761115 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:10:55.691637 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:55.691601 2582 patch_prober.go:28] interesting pod/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 16 13:10:55.692009 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:55.691657 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" podUID="28ef90b2-69d9-4c1d-b7fa-cecdfca899dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 16 13:10:55.716231 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:55.716153 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:10:55.765956 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:55.765922 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" event={"ID":"9106e0bd-ef66-4211-bc4b-3305e41f991e","Type":"ContainerStarted","Data":"b2279335922fd90d8a49db068251d0129ed3b9d465f9e27fc65a0925e84ace9b"} Jul 16 13:10:55.820985 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:55.820934 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:10:56.322558 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:56.322523 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:56.323594 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:56.323552 2582 patch_prober.go:28] interesting pod/e2e-api-b-a22377-openshift-default-67645c967c-wptcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 16 13:10:56.323741 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:56.323616 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" podUID="9106e0bd-ef66-4211-bc4b-3305e41f991e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 16 13:10:56.678373 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:56.678339 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:10:56.694492 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:56.694460 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:56.695594 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:56.695563 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:56.704822 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:56.704796 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:10:57.322892 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:57.322856 2582 patch_prober.go:28] interesting pod/e2e-api-b-a22377-openshift-default-67645c967c-wptcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 16 13:10:57.323097 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:57.322929 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" podUID="9106e0bd-ef66-4211-bc4b-3305e41f991e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 16 13:10:58.323289 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:58.323256 2582 patch_prober.go:28] interesting pod/e2e-api-b-a22377-openshift-default-67645c967c-wptcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 16 13:10:58.323709 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:58.323312 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" podUID="9106e0bd-ef66-4211-bc4b-3305e41f991e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 16 13:10:59.322485 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:59.322444 2582 patch_prober.go:28] interesting pod/e2e-api-b-a22377-openshift-default-67645c967c-wptcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 16 13:10:59.322691 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:10:59.322509 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" podUID="9106e0bd-ef66-4211-bc4b-3305e41f991e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 16 13:11:00.326684 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:00.326652 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:11:00.327742 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:00.327720 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:11:01.812904 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:01.812868 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:03.961560 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:03.961519 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:04.064436 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:04.064383 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:07.683556 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:07.683495 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:07.785699 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:07.785668 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:07.970846 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:07.970766 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq"] Jul 16 13:11:07.971038 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:07.971013 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" podUID="28ef90b2-69d9-4c1d-b7fa-cecdfca899dc" containerName="istio-proxy" containerID="cri-o://95865d7c9b523bef3c49d3809ff128cb1fbaa5152845a1189e50ebf9d17bba37" gracePeriod=30 Jul 16 13:11:07.991172 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:07.991139 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:08.246719 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:08.246637 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:08.704731 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:08.704700 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:11.697563 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:11.697530 2582 patch_prober.go:28] interesting pod/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 16 13:11:11.697924 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:11.697593 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" podUID="28ef90b2-69d9-4c1d-b7fa-cecdfca899dc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 13:11:13.215458 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.215434 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:11:13.344277 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.344176 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-istio-data\") pod \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " Jul 16 13:11:13.344277 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.344230 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-istio-token\") pod \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " Jul 16 13:11:13.344277 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.344248 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-istiod-ca-cert\") pod \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " Jul 16 13:11:13.344277 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.344268 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-workload-certs\") pod \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " Jul 16 13:11:13.344643 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.344436 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-workload-socket\") pod \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " Jul 16 13:11:13.344643 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.344482 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-credential-socket\") pod \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " Jul 16 13:11:13.344643 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.344550 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6znp2\" (UniqueName: \"kubernetes.io/projected/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-kube-api-access-6znp2\") pod \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " Jul 16 13:11:13.344643 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.344615 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "28ef90b2-69d9-4c1d-b7fa-cecdfca899dc" (UID: "28ef90b2-69d9-4c1d-b7fa-cecdfca899dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:11:13.344643 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.344625 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-istio-podinfo\") pod \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " Jul 16 13:11:13.344898 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.344654 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-istio-envoy\") pod \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\" (UID: \"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc\") " Jul 16 13:11:13.344898 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.344735 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "28ef90b2-69d9-4c1d-b7fa-cecdfca899dc" (UID: "28ef90b2-69d9-4c1d-b7fa-cecdfca899dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:11:13.344898 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.344764 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "28ef90b2-69d9-4c1d-b7fa-cecdfca899dc" (UID: "28ef90b2-69d9-4c1d-b7fa-cecdfca899dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:11:13.344898 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.344809 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "28ef90b2-69d9-4c1d-b7fa-cecdfca899dc" (UID: "28ef90b2-69d9-4c1d-b7fa-cecdfca899dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:11:13.345092 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.344973 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-istiod-ca-cert\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:11:13.345092 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.344994 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-workload-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:11:13.345092 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.345006 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-workload-socket\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:11:13.345092 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.345014 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-credential-socket\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:11:13.345368 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.345344 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-istio-data" (OuterVolumeSpecName: "istio-data") pod "28ef90b2-69d9-4c1d-b7fa-cecdfca899dc" (UID: "28ef90b2-69d9-4c1d-b7fa-cecdfca899dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:11:13.346680 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.346655 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-istio-token" (OuterVolumeSpecName: "istio-token") pod "28ef90b2-69d9-4c1d-b7fa-cecdfca899dc" (UID: "28ef90b2-69d9-4c1d-b7fa-cecdfca899dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:11:13.346917 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.346896 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "28ef90b2-69d9-4c1d-b7fa-cecdfca899dc" (UID: "28ef90b2-69d9-4c1d-b7fa-cecdfca899dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:11:13.347141 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.347124 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-kube-api-access-6znp2" (OuterVolumeSpecName: "kube-api-access-6znp2") pod "28ef90b2-69d9-4c1d-b7fa-cecdfca899dc" (UID: "28ef90b2-69d9-4c1d-b7fa-cecdfca899dc"). InnerVolumeSpecName "kube-api-access-6znp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:11:13.347197 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.347140 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "28ef90b2-69d9-4c1d-b7fa-cecdfca899dc" (UID: "28ef90b2-69d9-4c1d-b7fa-cecdfca899dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 13:11:13.445566 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.445523 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-istio-data\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:11:13.445566 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.445558 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-istio-token\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:11:13.445566 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.445568 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6znp2\" (UniqueName: \"kubernetes.io/projected/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-kube-api-access-6znp2\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:11:13.445566 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.445577 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-istio-podinfo\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:11:13.445851 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.445586 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc-istio-envoy\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:11:13.855105 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.855070 2582 generic.go:356] "Generic (PLEG): container finished" podID="28ef90b2-69d9-4c1d-b7fa-cecdfca899dc" containerID="95865d7c9b523bef3c49d3809ff128cb1fbaa5152845a1189e50ebf9d17bba37" exitCode=0 Jul 16 13:11:13.855256 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.855119 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" Jul 16 13:11:13.855256 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.855158 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" event={"ID":"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc","Type":"ContainerDied","Data":"95865d7c9b523bef3c49d3809ff128cb1fbaa5152845a1189e50ebf9d17bba37"} Jul 16 13:11:13.855256 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.855211 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq" event={"ID":"28ef90b2-69d9-4c1d-b7fa-cecdfca899dc","Type":"ContainerDied","Data":"c6f2ad95bc90023b5c789ee7b9b275a584afb87b5765b062d8f7ced423938269"} Jul 16 13:11:13.855256 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.855234 2582 scope.go:117] "RemoveContainer" containerID="95865d7c9b523bef3c49d3809ff128cb1fbaa5152845a1189e50ebf9d17bba37" Jul 16 13:11:13.864655 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.864636 2582 scope.go:117] "RemoveContainer" containerID="fe976bfdc12f4065efe6e12e5a9322c34ecb68b17b6ebc67481d58148c1907a2" Jul 16 13:11:13.872167 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.872151 2582 scope.go:117] "RemoveContainer" containerID="95865d7c9b523bef3c49d3809ff128cb1fbaa5152845a1189e50ebf9d17bba37" Jul 16 13:11:13.872448 ip-10-0-140-130 kubenswrapper[2582]: E0716 13:11:13.872426 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95865d7c9b523bef3c49d3809ff128cb1fbaa5152845a1189e50ebf9d17bba37\": container with ID starting with 95865d7c9b523bef3c49d3809ff128cb1fbaa5152845a1189e50ebf9d17bba37 not found: ID does not exist" containerID="95865d7c9b523bef3c49d3809ff128cb1fbaa5152845a1189e50ebf9d17bba37" Jul 16 13:11:13.872545 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.872456 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95865d7c9b523bef3c49d3809ff128cb1fbaa5152845a1189e50ebf9d17bba37"} err="failed to get container status \"95865d7c9b523bef3c49d3809ff128cb1fbaa5152845a1189e50ebf9d17bba37\": rpc error: code = NotFound desc = could not find container \"95865d7c9b523bef3c49d3809ff128cb1fbaa5152845a1189e50ebf9d17bba37\": container with ID starting with 95865d7c9b523bef3c49d3809ff128cb1fbaa5152845a1189e50ebf9d17bba37 not found: ID does not exist" Jul 16 13:11:13.872545 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.872478 2582 scope.go:117] "RemoveContainer" containerID="fe976bfdc12f4065efe6e12e5a9322c34ecb68b17b6ebc67481d58148c1907a2" Jul 16 13:11:13.872736 ip-10-0-140-130 kubenswrapper[2582]: E0716 13:11:13.872719 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe976bfdc12f4065efe6e12e5a9322c34ecb68b17b6ebc67481d58148c1907a2\": container with ID starting with fe976bfdc12f4065efe6e12e5a9322c34ecb68b17b6ebc67481d58148c1907a2 not found: ID does not exist" containerID="fe976bfdc12f4065efe6e12e5a9322c34ecb68b17b6ebc67481d58148c1907a2" Jul 16 13:11:13.872776 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.872742 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe976bfdc12f4065efe6e12e5a9322c34ecb68b17b6ebc67481d58148c1907a2"} err="failed to get container status \"fe976bfdc12f4065efe6e12e5a9322c34ecb68b17b6ebc67481d58148c1907a2\": rpc error: code = NotFound desc = could not find container \"fe976bfdc12f4065efe6e12e5a9322c34ecb68b17b6ebc67481d58148c1907a2\": container with ID starting with fe976bfdc12f4065efe6e12e5a9322c34ecb68b17b6ebc67481d58148c1907a2 not found: ID does not exist" Jul 16 13:11:13.876323 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.876304 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq"] Jul 16 13:11:13.880015 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.879997 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-213b02-openshift-default-7658f8664c-qgwjq"] Jul 16 13:11:13.977115 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:13.977083 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28ef90b2-69d9-4c1d-b7fa-cecdfca899dc" path="/var/lib/kubelet/pods/28ef90b2-69d9-4c1d-b7fa-cecdfca899dc/volumes" Jul 16 13:11:14.641909 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:14.641877 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:20.460809 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:20.460773 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:20.535733 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:20.535701 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:20.724402 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:20.724306 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv"] Jul 16 13:11:20.724580 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:20.724533 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" podUID="9106e0bd-ef66-4211-bc4b-3305e41f991e" containerName="istio-proxy" containerID="cri-o://b2279335922fd90d8a49db068251d0129ed3b9d465f9e27fc65a0925e84ace9b" gracePeriod=30 Jul 16 13:11:20.782634 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:20.782600 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:20.978968 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:20.978872 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:21.346452 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.346368 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:21.631483 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.631454 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk"] Jul 16 13:11:21.631932 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.631893 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12e2f5b8-c482-43fa-a69e-57c639af1ff8" containerName="istio-proxy" Jul 16 13:11:21.631932 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.631913 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="12e2f5b8-c482-43fa-a69e-57c639af1ff8" containerName="istio-proxy" Jul 16 13:11:21.631932 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.631931 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28ef90b2-69d9-4c1d-b7fa-cecdfca899dc" containerName="istio-proxy" Jul 16 13:11:21.632089 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.631939 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="28ef90b2-69d9-4c1d-b7fa-cecdfca899dc" containerName="istio-proxy" Jul 16 13:11:21.632089 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.631964 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28ef90b2-69d9-4c1d-b7fa-cecdfca899dc" containerName="istio-proxy" Jul 16 13:11:21.632089 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.631973 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="28ef90b2-69d9-4c1d-b7fa-cecdfca899dc" containerName="istio-proxy" Jul 16 13:11:21.632089 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.632052 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="28ef90b2-69d9-4c1d-b7fa-cecdfca899dc" containerName="istio-proxy" Jul 16 13:11:21.632089 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.632065 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="28ef90b2-69d9-4c1d-b7fa-cecdfca899dc" containerName="istio-proxy" Jul 16 13:11:21.636710 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.636691 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:21.638808 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.638792 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-b12c92-openshift-default-dockercfg-pd26t\"" Jul 16 13:11:21.647673 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.647652 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk"] Jul 16 13:11:21.671922 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.671889 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:21.703257 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.703231 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80415598-8795-489d-b5b1-a19908646f84-workload-socket\") pod \"e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:21.703352 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.703259 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80415598-8795-489d-b5b1-a19908646f84-workload-certs\") pod \"e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:21.703352 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.703282 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80415598-8795-489d-b5b1-a19908646f84-istio-envoy\") pod \"e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:21.703352 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.703300 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80415598-8795-489d-b5b1-a19908646f84-istio-token\") pod \"e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:21.703568 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.703413 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80415598-8795-489d-b5b1-a19908646f84-istio-data\") pod \"e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:21.703568 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.703458 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80415598-8795-489d-b5b1-a19908646f84-istio-podinfo\") pod \"e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:21.703568 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.703489 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80415598-8795-489d-b5b1-a19908646f84-istiod-ca-cert\") pod \"e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:21.703687 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.703575 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80415598-8795-489d-b5b1-a19908646f84-credential-socket\") pod \"e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:21.703687 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.703644 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wn6n\" (UniqueName: \"kubernetes.io/projected/80415598-8795-489d-b5b1-a19908646f84-kube-api-access-9wn6n\") pod \"e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:21.804569 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.804536 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80415598-8795-489d-b5b1-a19908646f84-workload-socket\") pod \"e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:21.804569 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.804571 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80415598-8795-489d-b5b1-a19908646f84-workload-certs\") pod \"e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:21.804818 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.804597 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80415598-8795-489d-b5b1-a19908646f84-istio-envoy\") pod \"e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:21.804818 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.804629 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80415598-8795-489d-b5b1-a19908646f84-istio-token\") pod \"e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:21.804818 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.804655 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80415598-8795-489d-b5b1-a19908646f84-istio-data\") pod \"e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:21.804818 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.804677 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80415598-8795-489d-b5b1-a19908646f84-istio-podinfo\") pod \"e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:21.804818 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.804695 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80415598-8795-489d-b5b1-a19908646f84-istiod-ca-cert\") pod \"e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:21.804818 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.804761 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80415598-8795-489d-b5b1-a19908646f84-credential-socket\") pod \"e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:21.804818 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.804817 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wn6n\" (UniqueName: \"kubernetes.io/projected/80415598-8795-489d-b5b1-a19908646f84-kube-api-access-9wn6n\") pod \"e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:21.805293 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.805263 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80415598-8795-489d-b5b1-a19908646f84-workload-certs\") pod \"e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:21.805477 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.805293 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80415598-8795-489d-b5b1-a19908646f84-istio-data\") pod \"e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:21.805606 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.805347 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80415598-8795-489d-b5b1-a19908646f84-workload-socket\") pod \"e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:21.805778 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.805755 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80415598-8795-489d-b5b1-a19908646f84-credential-socket\") pod \"e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:21.810650 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.806097 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80415598-8795-489d-b5b1-a19908646f84-istiod-ca-cert\") pod \"e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:21.813009 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.811329 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80415598-8795-489d-b5b1-a19908646f84-istio-envoy\") pod \"e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:21.813009 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.811997 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80415598-8795-489d-b5b1-a19908646f84-istio-podinfo\") pod \"e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:21.813877 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.813859 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80415598-8795-489d-b5b1-a19908646f84-istio-token\") pod \"e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:21.814029 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.814008 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wn6n\" (UniqueName: \"kubernetes.io/projected/80415598-8795-489d-b5b1-a19908646f84-kube-api-access-9wn6n\") pod \"e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:21.837271 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.837241 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:21.946897 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:21.946801 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:22.080469 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:22.080442 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk"] Jul 16 13:11:22.082313 ip-10-0-140-130 kubenswrapper[2582]: W0716 13:11:22.082283 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80415598_8795_489d_b5b1_a19908646f84.slice/crio-ad54ea7e062043ddb340bc77fde34c97b7535ff8b481c161ae34106c49806bcc WatchSource:0}: Error finding container ad54ea7e062043ddb340bc77fde34c97b7535ff8b481c161ae34106c49806bcc: Status 404 returned error can't find the container with id ad54ea7e062043ddb340bc77fde34c97b7535ff8b481c161ae34106c49806bcc Jul 16 13:11:22.084415 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:22.084367 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:11:22.084497 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:22.084445 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:11:22.084497 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:22.084472 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:11:22.900465 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:22.900427 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" event={"ID":"80415598-8795-489d-b5b1-a19908646f84","Type":"ContainerStarted","Data":"36bdeef81fd7c90215b64e4b1a37f874c8612f5a70fe78e43a2aaa7b808e4c7a"} Jul 16 13:11:22.900869 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:22.900485 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" event={"ID":"80415598-8795-489d-b5b1-a19908646f84","Type":"ContainerStarted","Data":"ad54ea7e062043ddb340bc77fde34c97b7535ff8b481c161ae34106c49806bcc"} Jul 16 13:11:22.922926 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:22.922879 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" podStartSLOduration=1.922864617 podStartE2EDuration="1.922864617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:11:22.921379589 +0000 UTC m=+1941.399125257" watchObservedRunningTime="2026-07-16 13:11:22.922864617 +0000 UTC m=+1941.400610233" Jul 16 13:11:22.947531 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:22.947503 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:23.849576 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:23.849543 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:23.944517 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:23.944489 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:23.952073 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:23.952050 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:24.009434 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:24.009379 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:24.427349 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:24.427314 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:24.648941 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:24.648909 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:24.828379 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:24.828294 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:24.910342 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:24.910318 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:24.911482 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:24.911464 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:25.917982 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:25.917941 2582 generic.go:356] "Generic (PLEG): container finished" podID="9106e0bd-ef66-4211-bc4b-3305e41f991e" containerID="b2279335922fd90d8a49db068251d0129ed3b9d465f9e27fc65a0925e84ace9b" exitCode=0 Jul 16 13:11:25.918369 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:25.917986 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" event={"ID":"9106e0bd-ef66-4211-bc4b-3305e41f991e","Type":"ContainerDied","Data":"b2279335922fd90d8a49db068251d0129ed3b9d465f9e27fc65a0925e84ace9b"} Jul 16 13:11:25.918369 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:25.918058 2582 scope.go:117] "RemoveContainer" containerID="66be56e92d47fce6dfc5bdefe0f5209d199e6298e6c7dd808b0779fdaeb9978f" Jul 16 13:11:25.968503 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:25.968479 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:11:26.036170 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:26.036097 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4dvl\" (UniqueName: \"kubernetes.io/projected/9106e0bd-ef66-4211-bc4b-3305e41f991e-kube-api-access-z4dvl\") pod \"9106e0bd-ef66-4211-bc4b-3305e41f991e\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " Jul 16 13:11:26.036170 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:26.036154 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9106e0bd-ef66-4211-bc4b-3305e41f991e-istio-podinfo\") pod \"9106e0bd-ef66-4211-bc4b-3305e41f991e\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " Jul 16 13:11:26.036365 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:26.036189 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9106e0bd-ef66-4211-bc4b-3305e41f991e-istiod-ca-cert\") pod \"9106e0bd-ef66-4211-bc4b-3305e41f991e\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " Jul 16 13:11:26.036365 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:26.036216 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9106e0bd-ef66-4211-bc4b-3305e41f991e-workload-socket\") pod \"9106e0bd-ef66-4211-bc4b-3305e41f991e\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " Jul 16 13:11:26.036365 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:26.036264 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9106e0bd-ef66-4211-bc4b-3305e41f991e-istio-envoy\") pod \"9106e0bd-ef66-4211-bc4b-3305e41f991e\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " Jul 16 13:11:26.036365 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:26.036308 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9106e0bd-ef66-4211-bc4b-3305e41f991e-credential-socket\") pod \"9106e0bd-ef66-4211-bc4b-3305e41f991e\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " Jul 16 13:11:26.036365 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:26.036346 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9106e0bd-ef66-4211-bc4b-3305e41f991e-istio-data\") pod \"9106e0bd-ef66-4211-bc4b-3305e41f991e\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " Jul 16 13:11:26.036653 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:26.036375 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9106e0bd-ef66-4211-bc4b-3305e41f991e-workload-certs\") pod \"9106e0bd-ef66-4211-bc4b-3305e41f991e\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " Jul 16 13:11:26.036653 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:26.036442 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9106e0bd-ef66-4211-bc4b-3305e41f991e-istio-token\") pod \"9106e0bd-ef66-4211-bc4b-3305e41f991e\" (UID: \"9106e0bd-ef66-4211-bc4b-3305e41f991e\") " Jul 16 13:11:26.036653 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:26.036538 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9106e0bd-ef66-4211-bc4b-3305e41f991e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9106e0bd-ef66-4211-bc4b-3305e41f991e" (UID: "9106e0bd-ef66-4211-bc4b-3305e41f991e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:11:26.036816 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:26.036626 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9106e0bd-ef66-4211-bc4b-3305e41f991e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9106e0bd-ef66-4211-bc4b-3305e41f991e" (UID: "9106e0bd-ef66-4211-bc4b-3305e41f991e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:11:26.036816 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:26.036630 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9106e0bd-ef66-4211-bc4b-3305e41f991e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9106e0bd-ef66-4211-bc4b-3305e41f991e" (UID: "9106e0bd-ef66-4211-bc4b-3305e41f991e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:11:26.036816 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:26.036748 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9106e0bd-ef66-4211-bc4b-3305e41f991e-credential-socket\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:11:26.036816 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:26.036768 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9106e0bd-ef66-4211-bc4b-3305e41f991e-workload-socket\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:11:26.036976 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:26.036823 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9106e0bd-ef66-4211-bc4b-3305e41f991e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9106e0bd-ef66-4211-bc4b-3305e41f991e" (UID: "9106e0bd-ef66-4211-bc4b-3305e41f991e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:11:26.037579 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:26.037497 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9106e0bd-ef66-4211-bc4b-3305e41f991e-istio-data" (OuterVolumeSpecName: "istio-data") pod "9106e0bd-ef66-4211-bc4b-3305e41f991e" (UID: "9106e0bd-ef66-4211-bc4b-3305e41f991e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:11:26.038552 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:26.038528 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9106e0bd-ef66-4211-bc4b-3305e41f991e-kube-api-access-z4dvl" (OuterVolumeSpecName: "kube-api-access-z4dvl") pod "9106e0bd-ef66-4211-bc4b-3305e41f991e" (UID: "9106e0bd-ef66-4211-bc4b-3305e41f991e"). InnerVolumeSpecName "kube-api-access-z4dvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:11:26.038552 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:26.038546 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9106e0bd-ef66-4211-bc4b-3305e41f991e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9106e0bd-ef66-4211-bc4b-3305e41f991e" (UID: "9106e0bd-ef66-4211-bc4b-3305e41f991e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 13:11:26.038689 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:26.038675 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9106e0bd-ef66-4211-bc4b-3305e41f991e-istio-token" (OuterVolumeSpecName: "istio-token") pod "9106e0bd-ef66-4211-bc4b-3305e41f991e" (UID: "9106e0bd-ef66-4211-bc4b-3305e41f991e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:11:26.038927 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:26.038909 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9106e0bd-ef66-4211-bc4b-3305e41f991e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9106e0bd-ef66-4211-bc4b-3305e41f991e" (UID: "9106e0bd-ef66-4211-bc4b-3305e41f991e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:11:26.137140 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:26.137111 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9106e0bd-ef66-4211-bc4b-3305e41f991e-istio-envoy\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:11:26.137140 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:26.137137 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9106e0bd-ef66-4211-bc4b-3305e41f991e-istio-data\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:11:26.137318 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:26.137150 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9106e0bd-ef66-4211-bc4b-3305e41f991e-workload-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:11:26.137318 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:26.137162 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9106e0bd-ef66-4211-bc4b-3305e41f991e-istio-token\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:11:26.137318 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:26.137173 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4dvl\" (UniqueName: \"kubernetes.io/projected/9106e0bd-ef66-4211-bc4b-3305e41f991e-kube-api-access-z4dvl\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:11:26.137318 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:26.137185 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9106e0bd-ef66-4211-bc4b-3305e41f991e-istio-podinfo\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:11:26.137318 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:26.137196 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9106e0bd-ef66-4211-bc4b-3305e41f991e-istiod-ca-cert\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:11:26.923715 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:26.923683 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" Jul 16 13:11:26.924135 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:26.923721 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv" event={"ID":"9106e0bd-ef66-4211-bc4b-3305e41f991e","Type":"ContainerDied","Data":"d227b81c51bea5c5a9f21bb5da1765b72fc7d4d0bf1cffdd9cf8eda981db8bda"} Jul 16 13:11:26.924135 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:26.923780 2582 scope.go:117] "RemoveContainer" containerID="b2279335922fd90d8a49db068251d0129ed3b9d465f9e27fc65a0925e84ace9b" Jul 16 13:11:26.945402 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:26.945363 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv"] Jul 16 13:11:26.954262 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:26.954238 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a22377-openshift-default-67645c967c-wptcv"] Jul 16 13:11:27.976875 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:27.976842 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9106e0bd-ef66-4211-bc4b-3305e41f991e" path="/var/lib/kubelet/pods/9106e0bd-ef66-4211-bc4b-3305e41f991e/volumes" Jul 16 13:11:29.429218 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:29.429184 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:29.796045 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:29.795959 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:30.095106 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.094998 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf"] Jul 16 13:11:30.095452 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.095432 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9106e0bd-ef66-4211-bc4b-3305e41f991e" containerName="istio-proxy" Jul 16 13:11:30.095569 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.095455 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9106e0bd-ef66-4211-bc4b-3305e41f991e" containerName="istio-proxy" Jul 16 13:11:30.095569 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.095483 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9106e0bd-ef66-4211-bc4b-3305e41f991e" containerName="istio-proxy" Jul 16 13:11:30.095569 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.095494 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9106e0bd-ef66-4211-bc4b-3305e41f991e" containerName="istio-proxy" Jul 16 13:11:30.095731 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.095603 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9106e0bd-ef66-4211-bc4b-3305e41f991e" containerName="istio-proxy" Jul 16 13:11:30.100183 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.100156 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:30.102529 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.102503 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-8d69f8-openshift-default-dockercfg-lhj2j\"" Jul 16 13:11:30.112239 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.112214 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf"] Jul 16 13:11:30.137611 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.137584 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:30.168854 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.168827 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/245797fa-8338-4d44-90fd-363f6b375cf7-workload-certs\") pod \"e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:30.168949 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.168862 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/245797fa-8338-4d44-90fd-363f6b375cf7-istio-data\") pod \"e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:30.168949 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.168885 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/245797fa-8338-4d44-90fd-363f6b375cf7-istiod-ca-cert\") pod \"e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:30.168949 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.168937 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/245797fa-8338-4d44-90fd-363f6b375cf7-istio-token\") pod \"e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:30.169059 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.168952 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwqjt\" (UniqueName: \"kubernetes.io/projected/245797fa-8338-4d44-90fd-363f6b375cf7-kube-api-access-vwqjt\") pod \"e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:30.169059 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.168982 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/245797fa-8338-4d44-90fd-363f6b375cf7-credential-socket\") pod \"e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:30.169059 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.169005 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/245797fa-8338-4d44-90fd-363f6b375cf7-istio-podinfo\") pod \"e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:30.169059 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.169034 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/245797fa-8338-4d44-90fd-363f6b375cf7-istio-envoy\") pod \"e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:30.169180 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.169100 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/245797fa-8338-4d44-90fd-363f6b375cf7-workload-socket\") pod \"e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:30.269938 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.269904 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/245797fa-8338-4d44-90fd-363f6b375cf7-workload-certs\") pod \"e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:30.269938 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.269939 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/245797fa-8338-4d44-90fd-363f6b375cf7-istio-data\") pod \"e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:30.270186 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.269957 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/245797fa-8338-4d44-90fd-363f6b375cf7-istiod-ca-cert\") pod \"e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:30.270186 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.270047 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/245797fa-8338-4d44-90fd-363f6b375cf7-istio-token\") pod \"e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:30.270186 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.270084 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwqjt\" (UniqueName: \"kubernetes.io/projected/245797fa-8338-4d44-90fd-363f6b375cf7-kube-api-access-vwqjt\") pod \"e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:30.270351 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.270219 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/245797fa-8338-4d44-90fd-363f6b375cf7-credential-socket\") pod \"e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:30.270351 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.270278 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/245797fa-8338-4d44-90fd-363f6b375cf7-istio-podinfo\") pod \"e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:30.270351 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.270322 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/245797fa-8338-4d44-90fd-363f6b375cf7-istio-envoy\") pod \"e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:30.270525 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.270352 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/245797fa-8338-4d44-90fd-363f6b375cf7-workload-certs\") pod \"e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:30.270525 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.270468 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/245797fa-8338-4d44-90fd-363f6b375cf7-workload-socket\") pod \"e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:30.270630 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.270571 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/245797fa-8338-4d44-90fd-363f6b375cf7-istio-data\") pod \"e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:30.270630 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.270584 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/245797fa-8338-4d44-90fd-363f6b375cf7-credential-socket\") pod \"e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:30.270773 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.270750 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/245797fa-8338-4d44-90fd-363f6b375cf7-istiod-ca-cert\") pod \"e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:30.270820 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.270753 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/245797fa-8338-4d44-90fd-363f6b375cf7-workload-socket\") pod \"e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:30.272745 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.272723 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/245797fa-8338-4d44-90fd-363f6b375cf7-istio-envoy\") pod \"e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:30.272858 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.272839 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/245797fa-8338-4d44-90fd-363f6b375cf7-istio-podinfo\") pod \"e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:30.278686 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.278659 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/245797fa-8338-4d44-90fd-363f6b375cf7-istio-token\") pod \"e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:30.279167 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.279146 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwqjt\" (UniqueName: \"kubernetes.io/projected/245797fa-8338-4d44-90fd-363f6b375cf7-kube-api-access-vwqjt\") pod \"e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:30.287019 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.286995 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:30.411311 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.411279 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:30.549635 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.549610 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf"] Jul 16 13:11:30.551513 ip-10-0-140-130 kubenswrapper[2582]: W0716 13:11:30.551488 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod245797fa_8338_4d44_90fd_363f6b375cf7.slice/crio-e2e44eb32bad12fc21c0d43c693b3839f9144ee83e74f7efc92294f7e640ebb1 WatchSource:0}: Error finding container e2e44eb32bad12fc21c0d43c693b3839f9144ee83e74f7efc92294f7e640ebb1: Status 404 returned error can't find the container with id e2e44eb32bad12fc21c0d43c693b3839f9144ee83e74f7efc92294f7e640ebb1 Jul 16 13:11:30.553566 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.553534 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:11:30.553665 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.553611 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:11:30.553665 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.553655 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:11:30.945204 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.945167 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" event={"ID":"245797fa-8338-4d44-90fd-363f6b375cf7","Type":"ContainerStarted","Data":"64ade314bd6166dac88752a773b3db80ddea819384f90de67922e43baa292df4"} Jul 16 13:11:30.945375 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.945228 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" event={"ID":"245797fa-8338-4d44-90fd-363f6b375cf7","Type":"ContainerStarted","Data":"e2e44eb32bad12fc21c0d43c693b3839f9144ee83e74f7efc92294f7e640ebb1"} Jul 16 13:11:30.980903 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:30.980858 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" podStartSLOduration=0.980843178 podStartE2EDuration="980.843178ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:11:30.97879241 +0000 UTC m=+1949.456538035" watchObservedRunningTime="2026-07-16 13:11:30.980843178 +0000 UTC m=+1949.458588794" Jul 16 13:11:31.412454 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:31.412420 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:31.417408 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:31.417367 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:31.949795 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:31.949762 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:31.950861 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:31.950840 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:31.960130 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:31.960099 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:32.060409 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:32.053931 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:32.138590 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:32.138555 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:32.236443 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:32.236335 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:32.747257 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:32.747223 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:32.847998 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:32.847962 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:37.606916 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:37.606873 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:40.951536 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:40.951497 2582 patch_prober.go:28] interesting pod/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 16 13:11:40.951914 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:40.951583 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" podUID="80415598-8795-489d-b5b1-a19908646f84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 16 13:11:43.006005 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:43.005957 2582 generic.go:356] "Generic (PLEG): container finished" podID="80415598-8795-489d-b5b1-a19908646f84" containerID="36bdeef81fd7c90215b64e4b1a37f874c8612f5a70fe78e43a2aaa7b808e4c7a" exitCode=0 Jul 16 13:11:43.006574 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:43.006013 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" event={"ID":"80415598-8795-489d-b5b1-a19908646f84","Type":"ContainerDied","Data":"36bdeef81fd7c90215b64e4b1a37f874c8612f5a70fe78e43a2aaa7b808e4c7a"} Jul 16 13:11:43.006574 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:43.006538 2582 scope.go:117] "RemoveContainer" containerID="36bdeef81fd7c90215b64e4b1a37f874c8612f5a70fe78e43a2aaa7b808e4c7a" Jul 16 13:11:43.007821 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:43.007777 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:11:43.007921 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:43.007871 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:11:43.007998 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:43.007917 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:11:44.016006 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:44.015958 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" event={"ID":"80415598-8795-489d-b5b1-a19908646f84","Type":"ContainerStarted","Data":"811e02ec77dca08542bbc329604ba21c7cbe23d84f2c1d305b0021d56165c628"} Jul 16 13:11:44.947939 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:44.947894 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:44.948145 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:44.948109 2582 patch_prober.go:28] interesting pod/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 16 13:11:44.948224 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:44.948180 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" podUID="80415598-8795-489d-b5b1-a19908646f84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 16 13:11:45.947380 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:45.947348 2582 patch_prober.go:28] interesting pod/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 16 13:11:45.947815 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:45.947746 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" podUID="80415598-8795-489d-b5b1-a19908646f84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 16 13:11:46.355783 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:46.355700 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:46.502459 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:46.502421 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:46.947209 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:46.947177 2582 patch_prober.go:28] interesting pod/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 16 13:11:46.947473 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:46.947434 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" podUID="80415598-8795-489d-b5b1-a19908646f84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 16 13:11:47.947313 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:47.947271 2582 patch_prober.go:28] interesting pod/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 16 13:11:47.947526 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:47.947340 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" podUID="80415598-8795-489d-b5b1-a19908646f84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 16 13:11:48.951272 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:48.951243 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:48.951695 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:48.951619 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:48.952182 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:48.952160 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:11:51.063521 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:51.063487 2582 generic.go:356] "Generic (PLEG): container finished" podID="245797fa-8338-4d44-90fd-363f6b375cf7" containerID="64ade314bd6166dac88752a773b3db80ddea819384f90de67922e43baa292df4" exitCode=0 Jul 16 13:11:51.064050 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:51.063566 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" event={"ID":"245797fa-8338-4d44-90fd-363f6b375cf7","Type":"ContainerDied","Data":"64ade314bd6166dac88752a773b3db80ddea819384f90de67922e43baa292df4"} Jul 16 13:11:51.064117 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:51.064067 2582 scope.go:117] "RemoveContainer" containerID="64ade314bd6166dac88752a773b3db80ddea819384f90de67922e43baa292df4" Jul 16 13:11:51.065057 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:51.065033 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:11:51.065175 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:51.065096 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:11:51.065175 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:51.065126 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 16 13:11:52.070207 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:52.070171 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" event={"ID":"245797fa-8338-4d44-90fd-363f6b375cf7","Type":"ContainerStarted","Data":"8767e6d745ebe43375c7c0701f055091a79f7e289b8be4fd0de06cfc058617c3"} Jul 16 13:11:52.411464 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:52.411435 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:52.411693 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:52.411673 2582 patch_prober.go:28] interesting pod/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 16 13:11:52.411766 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:52.411719 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" podUID="245797fa-8338-4d44-90fd-363f6b375cf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 16 13:11:53.411781 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:53.411742 2582 patch_prober.go:28] interesting pod/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 16 13:11:53.412160 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:53.411797 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" podUID="245797fa-8338-4d44-90fd-363f6b375cf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 16 13:11:54.412480 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:54.412443 2582 patch_prober.go:28] interesting pod/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 16 13:11:54.412957 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:54.412508 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" podUID="245797fa-8338-4d44-90fd-363f6b375cf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 16 13:11:54.582645 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:54.582609 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:54.682978 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:54.682868 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:55.412322 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:55.412284 2582 patch_prober.go:28] interesting pod/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 16 13:11:55.412626 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:55.412597 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" podUID="245797fa-8338-4d44-90fd-363f6b375cf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 16 13:11:56.412474 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:56.412430 2582 patch_prober.go:28] interesting pod/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 16 13:11:56.412690 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:56.412509 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" podUID="245797fa-8338-4d44-90fd-363f6b375cf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 16 13:11:57.412219 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:57.412179 2582 patch_prober.go:28] interesting pod/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 16 13:11:57.412638 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:57.412606 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" podUID="245797fa-8338-4d44-90fd-363f6b375cf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 16 13:11:58.411929 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:58.411877 2582 patch_prober.go:28] interesting pod/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 16 13:11:58.412442 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:58.411954 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" podUID="245797fa-8338-4d44-90fd-363f6b375cf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 16 13:11:59.415860 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:59.415827 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:59.416320 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:59.416102 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:59.416881 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:59.416866 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:11:59.669979 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:59.669893 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h"] Jul 16 13:11:59.670372 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:59.670353 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9106e0bd-ef66-4211-bc4b-3305e41f991e" containerName="istio-proxy" Jul 16 13:11:59.672239 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:59.672217 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h" Jul 16 13:11:59.674687 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:59.674666 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b12c92\"/\"kube-root-ca.crt\"" Jul 16 13:11:59.674792 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:59.674683 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b12c92\"/\"openshift-service-ca.crt\"" Jul 16 13:11:59.674792 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:59.674736 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b12c92\"/\"default-dockercfg-5n2sv\"" Jul 16 13:11:59.674792 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:59.674773 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b12c92\"/\"auth-test-model-b12c92-kserve-self-signed-certs\"" Jul 16 13:11:59.683360 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:59.683338 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h"] Jul 16 13:11:59.701827 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:59.701791 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:11:59.847578 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:59.847526 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqdxj\" (UniqueName: \"kubernetes.io/projected/59a2b825-2c52-491e-a25d-c54bc317a6bd-kube-api-access-fqdxj\") pod \"auth-test-model-b12c92-kserve-57f997dd5f-mx29h\" (UID: \"59a2b825-2c52-491e-a25d-c54bc317a6bd\") " pod="ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h" Jul 16 13:11:59.847578 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:59.847583 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59a2b825-2c52-491e-a25d-c54bc317a6bd-home\") pod \"auth-test-model-b12c92-kserve-57f997dd5f-mx29h\" (UID: \"59a2b825-2c52-491e-a25d-c54bc317a6bd\") " pod="ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h" Jul 16 13:11:59.847860 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:59.847696 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a2b825-2c52-491e-a25d-c54bc317a6bd-tls-certs\") pod \"auth-test-model-b12c92-kserve-57f997dd5f-mx29h\" (UID: \"59a2b825-2c52-491e-a25d-c54bc317a6bd\") " pod="ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h" Jul 16 13:11:59.847860 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:59.847742 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59a2b825-2c52-491e-a25d-c54bc317a6bd-tmp-dir\") pod \"auth-test-model-b12c92-kserve-57f997dd5f-mx29h\" (UID: \"59a2b825-2c52-491e-a25d-c54bc317a6bd\") " pod="ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h" Jul 16 13:11:59.847860 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:59.847819 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59a2b825-2c52-491e-a25d-c54bc317a6bd-dshm\") pod \"auth-test-model-b12c92-kserve-57f997dd5f-mx29h\" (UID: \"59a2b825-2c52-491e-a25d-c54bc317a6bd\") " pod="ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h" Jul 16 13:11:59.847974 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:59.847858 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59a2b825-2c52-491e-a25d-c54bc317a6bd-model-cache\") pod \"auth-test-model-b12c92-kserve-57f997dd5f-mx29h\" (UID: \"59a2b825-2c52-491e-a25d-c54bc317a6bd\") " pod="ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h" Jul 16 13:11:59.948734 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:59.948657 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a2b825-2c52-491e-a25d-c54bc317a6bd-tls-certs\") pod \"auth-test-model-b12c92-kserve-57f997dd5f-mx29h\" (UID: \"59a2b825-2c52-491e-a25d-c54bc317a6bd\") " pod="ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h" Jul 16 13:11:59.948734 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:59.948697 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59a2b825-2c52-491e-a25d-c54bc317a6bd-tmp-dir\") pod \"auth-test-model-b12c92-kserve-57f997dd5f-mx29h\" (UID: \"59a2b825-2c52-491e-a25d-c54bc317a6bd\") " pod="ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h" Jul 16 13:11:59.948734 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:59.948724 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59a2b825-2c52-491e-a25d-c54bc317a6bd-dshm\") pod \"auth-test-model-b12c92-kserve-57f997dd5f-mx29h\" (UID: \"59a2b825-2c52-491e-a25d-c54bc317a6bd\") " pod="ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h" Jul 16 13:11:59.948989 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:59.948840 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59a2b825-2c52-491e-a25d-c54bc317a6bd-model-cache\") pod \"auth-test-model-b12c92-kserve-57f997dd5f-mx29h\" (UID: \"59a2b825-2c52-491e-a25d-c54bc317a6bd\") " pod="ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h" Jul 16 13:11:59.948989 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:59.948904 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqdxj\" (UniqueName: \"kubernetes.io/projected/59a2b825-2c52-491e-a25d-c54bc317a6bd-kube-api-access-fqdxj\") pod \"auth-test-model-b12c92-kserve-57f997dd5f-mx29h\" (UID: \"59a2b825-2c52-491e-a25d-c54bc317a6bd\") " pod="ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h" Jul 16 13:11:59.948989 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:59.948933 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59a2b825-2c52-491e-a25d-c54bc317a6bd-home\") pod \"auth-test-model-b12c92-kserve-57f997dd5f-mx29h\" (UID: \"59a2b825-2c52-491e-a25d-c54bc317a6bd\") " pod="ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h" Jul 16 13:11:59.949183 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:59.949159 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59a2b825-2c52-491e-a25d-c54bc317a6bd-tmp-dir\") pod \"auth-test-model-b12c92-kserve-57f997dd5f-mx29h\" (UID: \"59a2b825-2c52-491e-a25d-c54bc317a6bd\") " pod="ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h" Jul 16 13:11:59.949253 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:59.949188 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59a2b825-2c52-491e-a25d-c54bc317a6bd-model-cache\") pod \"auth-test-model-b12c92-kserve-57f997dd5f-mx29h\" (UID: \"59a2b825-2c52-491e-a25d-c54bc317a6bd\") " pod="ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h" Jul 16 13:11:59.949304 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:59.949257 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59a2b825-2c52-491e-a25d-c54bc317a6bd-home\") pod \"auth-test-model-b12c92-kserve-57f997dd5f-mx29h\" (UID: \"59a2b825-2c52-491e-a25d-c54bc317a6bd\") " pod="ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h" Jul 16 13:11:59.951112 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:59.951091 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59a2b825-2c52-491e-a25d-c54bc317a6bd-dshm\") pod \"auth-test-model-b12c92-kserve-57f997dd5f-mx29h\" (UID: \"59a2b825-2c52-491e-a25d-c54bc317a6bd\") " pod="ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h" Jul 16 13:11:59.951355 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:59.951336 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a2b825-2c52-491e-a25d-c54bc317a6bd-tls-certs\") pod \"auth-test-model-b12c92-kserve-57f997dd5f-mx29h\" (UID: \"59a2b825-2c52-491e-a25d-c54bc317a6bd\") " pod="ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h" Jul 16 13:11:59.959278 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:59.959259 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqdxj\" (UniqueName: \"kubernetes.io/projected/59a2b825-2c52-491e-a25d-c54bc317a6bd-kube-api-access-fqdxj\") pod \"auth-test-model-b12c92-kserve-57f997dd5f-mx29h\" (UID: \"59a2b825-2c52-491e-a25d-c54bc317a6bd\") " pod="ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h" Jul 16 13:11:59.981535 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:11:59.981511 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h" Jul 16 13:12:00.121981 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:00.121953 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h"] Jul 16 13:12:00.124907 ip-10-0-140-130 kubenswrapper[2582]: W0716 13:12:00.124882 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59a2b825_2c52_491e_a25d_c54bc317a6bd.slice/crio-1b0187bfea2467d294270d0cf1dc234b8406d48fe7cf9705a425f4dbcd4b1f46 WatchSource:0}: Error finding container 1b0187bfea2467d294270d0cf1dc234b8406d48fe7cf9705a425f4dbcd4b1f46: Status 404 returned error can't find the container with id 1b0187bfea2467d294270d0cf1dc234b8406d48fe7cf9705a425f4dbcd4b1f46 Jul 16 13:12:01.116259 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:01.116214 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h" event={"ID":"59a2b825-2c52-491e-a25d-c54bc317a6bd","Type":"ContainerStarted","Data":"d804053bdce5d74f96db375240c37b8b6fe7db47272b35b860870671e84f9f34"} Jul 16 13:12:01.116774 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:01.116265 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h" event={"ID":"59a2b825-2c52-491e-a25d-c54bc317a6bd","Type":"ContainerStarted","Data":"1b0187bfea2467d294270d0cf1dc234b8406d48fe7cf9705a425f4dbcd4b1f46"} Jul 16 13:12:01.148384 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:01.148332 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h" podStartSLOduration=2.148317329 podStartE2EDuration="2.148317329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:12:01.146490333 +0000 UTC m=+1979.624235953" watchObservedRunningTime="2026-07-16 13:12:01.148317329 +0000 UTC m=+1979.626062945" Jul 16 13:12:09.982460 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:09.982427 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h" Jul 16 13:12:09.982876 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:09.982720 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h" Jul 16 13:12:09.995057 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:09.995035 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h" Jul 16 13:12:10.174300 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:10.174269 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h" Jul 16 13:12:30.692503 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:30.692469 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44"] Jul 16 13:12:30.694850 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:30.694832 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44" Jul 16 13:12:30.697345 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:30.697318 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-8d69f8\"/\"openshift-service-ca.crt\"" Jul 16 13:12:30.698475 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:30.698444 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-8d69f8\"/\"default-dockercfg-7c7kk\"" Jul 16 13:12:30.698602 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:30.698473 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-8d69f8\"/\"kube-root-ca.crt\"" Jul 16 13:12:30.698602 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:30.698586 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-8d69f8\"/\"auth-test-model-8d69f8-kserve-self-signed-certs\"" Jul 16 13:12:30.705731 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:30.705712 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44"] Jul 16 13:12:30.721752 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:30.721715 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:12:30.792647 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:30.792615 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af044d90-73cb-4fe4-ae35-7929f6000adb-dshm\") pod \"auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44\" (UID: \"af044d90-73cb-4fe4-ae35-7929f6000adb\") " pod="ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44" Jul 16 13:12:30.792647 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:30.792657 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8brpf\" (UniqueName: \"kubernetes.io/projected/af044d90-73cb-4fe4-ae35-7929f6000adb-kube-api-access-8brpf\") pod \"auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44\" (UID: \"af044d90-73cb-4fe4-ae35-7929f6000adb\") " pod="ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44" Jul 16 13:12:30.792909 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:30.792697 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af044d90-73cb-4fe4-ae35-7929f6000adb-tls-certs\") pod \"auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44\" (UID: \"af044d90-73cb-4fe4-ae35-7929f6000adb\") " pod="ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44" Jul 16 13:12:30.792909 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:30.792727 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af044d90-73cb-4fe4-ae35-7929f6000adb-model-cache\") pod \"auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44\" (UID: \"af044d90-73cb-4fe4-ae35-7929f6000adb\") " pod="ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44" Jul 16 13:12:30.792909 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:30.792767 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af044d90-73cb-4fe4-ae35-7929f6000adb-home\") pod \"auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44\" (UID: \"af044d90-73cb-4fe4-ae35-7929f6000adb\") " pod="ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44" Jul 16 13:12:30.792909 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:30.792819 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af044d90-73cb-4fe4-ae35-7929f6000adb-tmp-dir\") pod \"auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44\" (UID: \"af044d90-73cb-4fe4-ae35-7929f6000adb\") " pod="ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44" Jul 16 13:12:30.893630 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:30.893596 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af044d90-73cb-4fe4-ae35-7929f6000adb-tls-certs\") pod \"auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44\" (UID: \"af044d90-73cb-4fe4-ae35-7929f6000adb\") " pod="ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44" Jul 16 13:12:30.893630 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:30.893634 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af044d90-73cb-4fe4-ae35-7929f6000adb-model-cache\") pod \"auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44\" (UID: \"af044d90-73cb-4fe4-ae35-7929f6000adb\") " pod="ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44" Jul 16 13:12:30.893849 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:30.893665 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af044d90-73cb-4fe4-ae35-7929f6000adb-home\") pod \"auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44\" (UID: \"af044d90-73cb-4fe4-ae35-7929f6000adb\") " pod="ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44" Jul 16 13:12:30.893849 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:30.893701 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af044d90-73cb-4fe4-ae35-7929f6000adb-tmp-dir\") pod \"auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44\" (UID: \"af044d90-73cb-4fe4-ae35-7929f6000adb\") " pod="ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44" Jul 16 13:12:30.893849 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:30.893745 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af044d90-73cb-4fe4-ae35-7929f6000adb-dshm\") pod \"auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44\" (UID: \"af044d90-73cb-4fe4-ae35-7929f6000adb\") " pod="ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44" Jul 16 13:12:30.893849 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:30.893764 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8brpf\" (UniqueName: \"kubernetes.io/projected/af044d90-73cb-4fe4-ae35-7929f6000adb-kube-api-access-8brpf\") pod \"auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44\" (UID: \"af044d90-73cb-4fe4-ae35-7929f6000adb\") " pod="ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44" Jul 16 13:12:30.894092 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:30.894067 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af044d90-73cb-4fe4-ae35-7929f6000adb-model-cache\") pod \"auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44\" (UID: \"af044d90-73cb-4fe4-ae35-7929f6000adb\") " pod="ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44" Jul 16 13:12:30.894092 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:30.894086 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af044d90-73cb-4fe4-ae35-7929f6000adb-home\") pod \"auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44\" (UID: \"af044d90-73cb-4fe4-ae35-7929f6000adb\") " pod="ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44" Jul 16 13:12:30.894196 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:30.894128 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af044d90-73cb-4fe4-ae35-7929f6000adb-tmp-dir\") pod \"auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44\" (UID: \"af044d90-73cb-4fe4-ae35-7929f6000adb\") " pod="ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44" Jul 16 13:12:30.896093 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:30.896069 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af044d90-73cb-4fe4-ae35-7929f6000adb-dshm\") pod \"auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44\" (UID: \"af044d90-73cb-4fe4-ae35-7929f6000adb\") " pod="ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44" Jul 16 13:12:30.896316 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:30.896301 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af044d90-73cb-4fe4-ae35-7929f6000adb-tls-certs\") pod \"auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44\" (UID: \"af044d90-73cb-4fe4-ae35-7929f6000adb\") " pod="ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44" Jul 16 13:12:30.902355 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:30.902329 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8brpf\" (UniqueName: \"kubernetes.io/projected/af044d90-73cb-4fe4-ae35-7929f6000adb-kube-api-access-8brpf\") pod \"auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44\" (UID: \"af044d90-73cb-4fe4-ae35-7929f6000adb\") " pod="ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44" Jul 16 13:12:31.003828 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:31.003729 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44" Jul 16 13:12:31.146269 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:31.146243 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44"] Jul 16 13:12:31.148263 ip-10-0-140-130 kubenswrapper[2582]: W0716 13:12:31.148237 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf044d90_73cb_4fe4_ae35_7929f6000adb.slice/crio-c2978bfe60d8b4aadde0629263fd6e3f2e930f8cdb9469ea1610864750bb2e27 WatchSource:0}: Error finding container c2978bfe60d8b4aadde0629263fd6e3f2e930f8cdb9469ea1610864750bb2e27: Status 404 returned error can't find the container with id c2978bfe60d8b4aadde0629263fd6e3f2e930f8cdb9469ea1610864750bb2e27 Jul 16 13:12:31.266095 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:31.266003 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44" event={"ID":"af044d90-73cb-4fe4-ae35-7929f6000adb","Type":"ContainerStarted","Data":"49c505394b5fab0c7a3ed81a365d935c8ee0c0de832481221ee18bb44a2183b4"} Jul 16 13:12:31.266095 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:31.266049 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44" event={"ID":"af044d90-73cb-4fe4-ae35-7929f6000adb","Type":"ContainerStarted","Data":"c2978bfe60d8b4aadde0629263fd6e3f2e930f8cdb9469ea1610864750bb2e27"} Jul 16 13:12:31.285576 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:31.285523 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44" podStartSLOduration=1.285508704 podStartE2EDuration="1.285508704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:12:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:12:31.283476893 +0000 UTC m=+2009.761222508" watchObservedRunningTime="2026-07-16 13:12:31.285508704 +0000 UTC m=+2009.763254319" Jul 16 13:12:40.259650 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:40.259617 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:12:40.566541 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:40.566454 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:12:41.004851 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:41.004812 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44" Jul 16 13:12:41.005035 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:41.004873 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44" Jul 16 13:12:41.018216 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:41.018189 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44" Jul 16 13:12:41.328298 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:12:41.328206 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44" Jul 16 13:13:06.925596 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:13:06.925539 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:13:07.024684 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:13:07.024635 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:14:51.914903 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:14:51.914864 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h"] Jul 16 13:14:52.038278 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:14:52.038238 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:14:52.765999 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:14:52.765962 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44"] Jul 16 13:14:54.767185 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:14:54.767147 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:14:58.284683 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:14:58.284652 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9"] Jul 16 13:14:58.287158 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:14:58.287133 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9" Jul 16 13:14:58.289632 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:14:58.289612 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b12c92\"/\"sub-test-model-b12c92-kserve-self-signed-certs\"" Jul 16 13:14:58.300994 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:14:58.300973 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9"] Jul 16 13:14:58.328810 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:14:58.328786 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:14:58.422244 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:14:58.422209 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1546c1fc-d64c-4a2c-9d52-013ed6478944-tmp-dir\") pod \"sub-test-model-b12c92-kserve-8664d6b8fc-67jb9\" (UID: \"1546c1fc-d64c-4a2c-9d52-013ed6478944\") " pod="ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9" Jul 16 13:14:58.422423 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:14:58.422258 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1546c1fc-d64c-4a2c-9d52-013ed6478944-dshm\") pod \"sub-test-model-b12c92-kserve-8664d6b8fc-67jb9\" (UID: \"1546c1fc-d64c-4a2c-9d52-013ed6478944\") " pod="ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9" Jul 16 13:14:58.422423 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:14:58.422336 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1546c1fc-d64c-4a2c-9d52-013ed6478944-tls-certs\") pod \"sub-test-model-b12c92-kserve-8664d6b8fc-67jb9\" (UID: \"1546c1fc-d64c-4a2c-9d52-013ed6478944\") " pod="ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9" Jul 16 13:14:58.422423 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:14:58.422366 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1546c1fc-d64c-4a2c-9d52-013ed6478944-home\") pod \"sub-test-model-b12c92-kserve-8664d6b8fc-67jb9\" (UID: \"1546c1fc-d64c-4a2c-9d52-013ed6478944\") " pod="ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9" Jul 16 13:14:58.422423 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:14:58.422413 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1546c1fc-d64c-4a2c-9d52-013ed6478944-model-cache\") pod \"sub-test-model-b12c92-kserve-8664d6b8fc-67jb9\" (UID: \"1546c1fc-d64c-4a2c-9d52-013ed6478944\") " pod="ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9" Jul 16 13:14:58.422588 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:14:58.422447 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsh5z\" (UniqueName: \"kubernetes.io/projected/1546c1fc-d64c-4a2c-9d52-013ed6478944-kube-api-access-fsh5z\") pod \"sub-test-model-b12c92-kserve-8664d6b8fc-67jb9\" (UID: \"1546c1fc-d64c-4a2c-9d52-013ed6478944\") " pod="ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9" Jul 16 13:14:58.523521 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:14:58.523489 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1546c1fc-d64c-4a2c-9d52-013ed6478944-tls-certs\") pod \"sub-test-model-b12c92-kserve-8664d6b8fc-67jb9\" (UID: \"1546c1fc-d64c-4a2c-9d52-013ed6478944\") " pod="ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9" Jul 16 13:14:58.523667 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:14:58.523525 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1546c1fc-d64c-4a2c-9d52-013ed6478944-home\") pod \"sub-test-model-b12c92-kserve-8664d6b8fc-67jb9\" (UID: \"1546c1fc-d64c-4a2c-9d52-013ed6478944\") " pod="ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9" Jul 16 13:14:58.523667 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:14:58.523544 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1546c1fc-d64c-4a2c-9d52-013ed6478944-model-cache\") pod \"sub-test-model-b12c92-kserve-8664d6b8fc-67jb9\" (UID: \"1546c1fc-d64c-4a2c-9d52-013ed6478944\") " pod="ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9" Jul 16 13:14:58.523667 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:14:58.523583 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsh5z\" (UniqueName: \"kubernetes.io/projected/1546c1fc-d64c-4a2c-9d52-013ed6478944-kube-api-access-fsh5z\") pod \"sub-test-model-b12c92-kserve-8664d6b8fc-67jb9\" (UID: \"1546c1fc-d64c-4a2c-9d52-013ed6478944\") " pod="ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9" Jul 16 13:14:58.523667 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:14:58.523660 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1546c1fc-d64c-4a2c-9d52-013ed6478944-tmp-dir\") pod \"sub-test-model-b12c92-kserve-8664d6b8fc-67jb9\" (UID: \"1546c1fc-d64c-4a2c-9d52-013ed6478944\") " pod="ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9" Jul 16 13:14:58.523890 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:14:58.523696 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1546c1fc-d64c-4a2c-9d52-013ed6478944-dshm\") pod \"sub-test-model-b12c92-kserve-8664d6b8fc-67jb9\" (UID: \"1546c1fc-d64c-4a2c-9d52-013ed6478944\") " pod="ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9" Jul 16 13:14:58.523999 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:14:58.523977 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1546c1fc-d64c-4a2c-9d52-013ed6478944-model-cache\") pod \"sub-test-model-b12c92-kserve-8664d6b8fc-67jb9\" (UID: \"1546c1fc-d64c-4a2c-9d52-013ed6478944\") " pod="ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9" Jul 16 13:14:58.524078 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:14:58.524005 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1546c1fc-d64c-4a2c-9d52-013ed6478944-home\") pod \"sub-test-model-b12c92-kserve-8664d6b8fc-67jb9\" (UID: \"1546c1fc-d64c-4a2c-9d52-013ed6478944\") " pod="ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9" Jul 16 13:14:58.524078 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:14:58.524052 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1546c1fc-d64c-4a2c-9d52-013ed6478944-tmp-dir\") pod \"sub-test-model-b12c92-kserve-8664d6b8fc-67jb9\" (UID: \"1546c1fc-d64c-4a2c-9d52-013ed6478944\") " pod="ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9" Jul 16 13:14:58.526083 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:14:58.526063 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1546c1fc-d64c-4a2c-9d52-013ed6478944-dshm\") pod \"sub-test-model-b12c92-kserve-8664d6b8fc-67jb9\" (UID: \"1546c1fc-d64c-4a2c-9d52-013ed6478944\") " pod="ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9" Jul 16 13:14:58.526256 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:14:58.526241 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1546c1fc-d64c-4a2c-9d52-013ed6478944-tls-certs\") pod \"sub-test-model-b12c92-kserve-8664d6b8fc-67jb9\" (UID: \"1546c1fc-d64c-4a2c-9d52-013ed6478944\") " pod="ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9" Jul 16 13:14:58.533572 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:14:58.533550 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsh5z\" (UniqueName: \"kubernetes.io/projected/1546c1fc-d64c-4a2c-9d52-013ed6478944-kube-api-access-fsh5z\") pod \"sub-test-model-b12c92-kserve-8664d6b8fc-67jb9\" (UID: \"1546c1fc-d64c-4a2c-9d52-013ed6478944\") " pod="ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9" Jul 16 13:14:58.596593 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:14:58.596511 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9" Jul 16 13:14:58.735285 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:14:58.735164 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9"] Jul 16 13:14:58.737966 ip-10-0-140-130 kubenswrapper[2582]: W0716 13:14:58.737938 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1546c1fc_d64c_4a2c_9d52_013ed6478944.slice/crio-d051317676deb23913662326926ffc6e860a7a1c8482fcb1d0e92723c35c5b6e WatchSource:0}: Error finding container d051317676deb23913662326926ffc6e860a7a1c8482fcb1d0e92723c35c5b6e: Status 404 returned error can't find the container with id d051317676deb23913662326926ffc6e860a7a1c8482fcb1d0e92723c35c5b6e Jul 16 13:14:58.976884 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:14:58.976844 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9" event={"ID":"1546c1fc-d64c-4a2c-9d52-013ed6478944","Type":"ContainerStarted","Data":"7f53615dd414c62e168b48e5d28303de76005c90d1c2e622cfb85f797d9532b0"} Jul 16 13:14:58.976884 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:14:58.976885 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9" event={"ID":"1546c1fc-d64c-4a2c-9d52-013ed6478944","Type":"ContainerStarted","Data":"d051317676deb23913662326926ffc6e860a7a1c8482fcb1d0e92723c35c5b6e"} Jul 16 13:14:59.010398 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:14:59.010349 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9" podStartSLOduration=1.010332704 podStartE2EDuration="1.010332704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:14:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:14:59.010067857 +0000 UTC m=+2157.487813469" watchObservedRunningTime="2026-07-16 13:14:59.010332704 +0000 UTC m=+2157.488078320" Jul 16 13:14:59.251525 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:14:59.251427 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:14:59.338156 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:14:59.338121 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:15:00.199444 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:00.199416 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8d69f8-29736795-6s4ll"] Jul 16 13:15:00.201994 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:00.201971 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8d69f8-29736795-6s4ll" Jul 16 13:15:00.204270 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:00.204252 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zhdlk\"" Jul 16 13:15:00.211201 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:00.211150 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8d69f8-29736795-6s4ll"] Jul 16 13:15:00.213785 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:00.213766 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8d69f8-29736795-6s4ll" Jul 16 13:15:00.353929 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:00.353864 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8d69f8-29736795-6s4ll"] Jul 16 13:15:00.356128 ip-10-0-140-130 kubenswrapper[2582]: W0716 13:15:00.356100 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3d3cdf4_c99f_4d12_a2a9_ffc8a9b1e418.slice/crio-8eba26b1e36f934f29d4919c019ee89a6aab77c4b3d280e565342a66e16f5c5f WatchSource:0}: Error finding container 8eba26b1e36f934f29d4919c019ee89a6aab77c4b3d280e565342a66e16f5c5f: Status 404 returned error can't find the container with id 8eba26b1e36f934f29d4919c019ee89a6aab77c4b3d280e565342a66e16f5c5f Jul 16 13:15:00.358028 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:00.358010 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 13:15:00.988453 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:00.988418 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8d69f8-29736795-6s4ll" event={"ID":"c3d3cdf4-c99f-4d12-a2a9-ffc8a9b1e418","Type":"ContainerStarted","Data":"8eba26b1e36f934f29d4919c019ee89a6aab77c4b3d280e565342a66e16f5c5f"} Jul 16 13:15:03.001050 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:03.001008 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8d69f8-29736795-6s4ll" event={"ID":"c3d3cdf4-c99f-4d12-a2a9-ffc8a9b1e418","Type":"ContainerStarted","Data":"ab3e3cd69a5c745c63177eec459fc3fc8eec0a3a8782f6dd5c92e65674dce8be"} Jul 16 13:15:03.019317 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:03.019267 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8d69f8-29736795-6s4ll" podStartSLOduration=0.741304538 podStartE2EDuration="3.019250605s" totalImagesPullingTime="2.277946067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:15:03.017687271 +0000 UTC m=+2161.495432897" watchObservedRunningTime="2026-07-16 13:15:03.019250605 +0000 UTC m=+2161.496996221" Jul 16 13:15:04.006743 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:04.006712 2582 generic.go:356] "Generic (PLEG): container finished" podID="c3d3cdf4-c99f-4d12-a2a9-ffc8a9b1e418" containerID="ab3e3cd69a5c745c63177eec459fc3fc8eec0a3a8782f6dd5c92e65674dce8be" exitCode=0 Jul 16 13:15:04.006743 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:04.006746 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8d69f8-29736795-6s4ll" event={"ID":"c3d3cdf4-c99f-4d12-a2a9-ffc8a9b1e418","Type":"ContainerDied","Data":"ab3e3cd69a5c745c63177eec459fc3fc8eec0a3a8782f6dd5c92e65674dce8be"} Jul 16 13:15:05.137220 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:05.137197 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8d69f8-29736795-6s4ll" Jul 16 13:15:06.017285 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:06.017252 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8d69f8-29736795-6s4ll" Jul 16 13:15:06.017453 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:06.017277 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-8d69f8-29736795-6s4ll" event={"ID":"c3d3cdf4-c99f-4d12-a2a9-ffc8a9b1e418","Type":"ContainerDied","Data":"8eba26b1e36f934f29d4919c019ee89a6aab77c4b3d280e565342a66e16f5c5f"} Jul 16 13:15:06.017453 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:06.017324 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8eba26b1e36f934f29d4919c019ee89a6aab77c4b3d280e565342a66e16f5c5f" Jul 16 13:15:06.935594 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:06.935516 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h" podUID="59a2b825-2c52-491e-a25d-c54bc317a6bd" containerName="main" containerID="cri-o://d804053bdce5d74f96db375240c37b8b6fe7db47272b35b860870671e84f9f34" gracePeriod=45 Jul 16 13:15:07.191296 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:07.191238 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h" Jul 16 13:15:07.299414 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:07.299353 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59a2b825-2c52-491e-a25d-c54bc317a6bd-tmp-dir\") pod \"59a2b825-2c52-491e-a25d-c54bc317a6bd\" (UID: \"59a2b825-2c52-491e-a25d-c54bc317a6bd\") " Jul 16 13:15:07.299601 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:07.299433 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqdxj\" (UniqueName: \"kubernetes.io/projected/59a2b825-2c52-491e-a25d-c54bc317a6bd-kube-api-access-fqdxj\") pod \"59a2b825-2c52-491e-a25d-c54bc317a6bd\" (UID: \"59a2b825-2c52-491e-a25d-c54bc317a6bd\") " Jul 16 13:15:07.299601 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:07.299470 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59a2b825-2c52-491e-a25d-c54bc317a6bd-model-cache\") pod \"59a2b825-2c52-491e-a25d-c54bc317a6bd\" (UID: \"59a2b825-2c52-491e-a25d-c54bc317a6bd\") " Jul 16 13:15:07.299601 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:07.299495 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a2b825-2c52-491e-a25d-c54bc317a6bd-tls-certs\") pod \"59a2b825-2c52-491e-a25d-c54bc317a6bd\" (UID: \"59a2b825-2c52-491e-a25d-c54bc317a6bd\") " Jul 16 13:15:07.299601 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:07.299587 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59a2b825-2c52-491e-a25d-c54bc317a6bd-home\") pod \"59a2b825-2c52-491e-a25d-c54bc317a6bd\" (UID: \"59a2b825-2c52-491e-a25d-c54bc317a6bd\") " Jul 16 13:15:07.299819 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:07.299615 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59a2b825-2c52-491e-a25d-c54bc317a6bd-dshm\") pod \"59a2b825-2c52-491e-a25d-c54bc317a6bd\" (UID: \"59a2b825-2c52-491e-a25d-c54bc317a6bd\") " Jul 16 13:15:07.299819 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:07.299644 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a2b825-2c52-491e-a25d-c54bc317a6bd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "59a2b825-2c52-491e-a25d-c54bc317a6bd" (UID: "59a2b825-2c52-491e-a25d-c54bc317a6bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:15:07.299819 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:07.299738 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a2b825-2c52-491e-a25d-c54bc317a6bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "59a2b825-2c52-491e-a25d-c54bc317a6bd" (UID: "59a2b825-2c52-491e-a25d-c54bc317a6bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:15:07.299963 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:07.299842 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a2b825-2c52-491e-a25d-c54bc317a6bd-home" (OuterVolumeSpecName: "home") pod "59a2b825-2c52-491e-a25d-c54bc317a6bd" (UID: "59a2b825-2c52-491e-a25d-c54bc317a6bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:15:07.299963 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:07.299929 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59a2b825-2c52-491e-a25d-c54bc317a6bd-home\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:15:07.299963 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:07.299960 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59a2b825-2c52-491e-a25d-c54bc317a6bd-tmp-dir\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:15:07.300111 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:07.299970 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59a2b825-2c52-491e-a25d-c54bc317a6bd-model-cache\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:15:07.301811 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:07.301787 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a2b825-2c52-491e-a25d-c54bc317a6bd-dshm" (OuterVolumeSpecName: "dshm") pod "59a2b825-2c52-491e-a25d-c54bc317a6bd" (UID: "59a2b825-2c52-491e-a25d-c54bc317a6bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:15:07.301811 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:07.301788 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59a2b825-2c52-491e-a25d-c54bc317a6bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "59a2b825-2c52-491e-a25d-c54bc317a6bd" (UID: "59a2b825-2c52-491e-a25d-c54bc317a6bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 13:15:07.301962 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:07.301841 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59a2b825-2c52-491e-a25d-c54bc317a6bd-kube-api-access-fqdxj" (OuterVolumeSpecName: "kube-api-access-fqdxj") pod "59a2b825-2c52-491e-a25d-c54bc317a6bd" (UID: "59a2b825-2c52-491e-a25d-c54bc317a6bd"). InnerVolumeSpecName "kube-api-access-fqdxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:15:07.401220 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:07.401188 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a2b825-2c52-491e-a25d-c54bc317a6bd-tls-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:15:07.401220 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:07.401215 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59a2b825-2c52-491e-a25d-c54bc317a6bd-dshm\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:15:07.401220 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:07.401224 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqdxj\" (UniqueName: \"kubernetes.io/projected/59a2b825-2c52-491e-a25d-c54bc317a6bd-kube-api-access-fqdxj\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:15:07.786970 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:07.786925 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44" podUID="af044d90-73cb-4fe4-ae35-7929f6000adb" containerName="main" containerID="cri-o://49c505394b5fab0c7a3ed81a365d935c8ee0c0de832481221ee18bb44a2183b4" gracePeriod=45 Jul 16 13:15:08.028624 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.028590 2582 generic.go:356] "Generic (PLEG): container finished" podID="59a2b825-2c52-491e-a25d-c54bc317a6bd" containerID="d804053bdce5d74f96db375240c37b8b6fe7db47272b35b860870671e84f9f34" exitCode=0 Jul 16 13:15:08.028997 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.028637 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h" Jul 16 13:15:08.028997 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.028676 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h" event={"ID":"59a2b825-2c52-491e-a25d-c54bc317a6bd","Type":"ContainerDied","Data":"d804053bdce5d74f96db375240c37b8b6fe7db47272b35b860870671e84f9f34"} Jul 16 13:15:08.028997 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.028724 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h" event={"ID":"59a2b825-2c52-491e-a25d-c54bc317a6bd","Type":"ContainerDied","Data":"1b0187bfea2467d294270d0cf1dc234b8406d48fe7cf9705a425f4dbcd4b1f46"} Jul 16 13:15:08.028997 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.028749 2582 scope.go:117] "RemoveContainer" containerID="d804053bdce5d74f96db375240c37b8b6fe7db47272b35b860870671e84f9f34" Jul 16 13:15:08.030132 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.030104 2582 generic.go:356] "Generic (PLEG): container finished" podID="af044d90-73cb-4fe4-ae35-7929f6000adb" containerID="49c505394b5fab0c7a3ed81a365d935c8ee0c0de832481221ee18bb44a2183b4" exitCode=0 Jul 16 13:15:08.030421 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.030158 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44" event={"ID":"af044d90-73cb-4fe4-ae35-7929f6000adb","Type":"ContainerDied","Data":"49c505394b5fab0c7a3ed81a365d935c8ee0c0de832481221ee18bb44a2183b4"} Jul 16 13:15:08.030421 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.030180 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44" event={"ID":"af044d90-73cb-4fe4-ae35-7929f6000adb","Type":"ContainerDied","Data":"c2978bfe60d8b4aadde0629263fd6e3f2e930f8cdb9469ea1610864750bb2e27"} Jul 16 13:15:08.030421 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.030191 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2978bfe60d8b4aadde0629263fd6e3f2e930f8cdb9469ea1610864750bb2e27" Jul 16 13:15:08.037561 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.037507 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44" Jul 16 13:15:08.037762 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.037689 2582 scope.go:117] "RemoveContainer" containerID="d804053bdce5d74f96db375240c37b8b6fe7db47272b35b860870671e84f9f34" Jul 16 13:15:08.037921 ip-10-0-140-130 kubenswrapper[2582]: E0716 13:15:08.037904 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d804053bdce5d74f96db375240c37b8b6fe7db47272b35b860870671e84f9f34\": container with ID starting with d804053bdce5d74f96db375240c37b8b6fe7db47272b35b860870671e84f9f34 not found: ID does not exist" containerID="d804053bdce5d74f96db375240c37b8b6fe7db47272b35b860870671e84f9f34" Jul 16 13:15:08.037981 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.037927 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d804053bdce5d74f96db375240c37b8b6fe7db47272b35b860870671e84f9f34"} err="failed to get container status \"d804053bdce5d74f96db375240c37b8b6fe7db47272b35b860870671e84f9f34\": rpc error: code = NotFound desc = could not find container \"d804053bdce5d74f96db375240c37b8b6fe7db47272b35b860870671e84f9f34\": container with ID starting with d804053bdce5d74f96db375240c37b8b6fe7db47272b35b860870671e84f9f34 not found: ID does not exist" Jul 16 13:15:08.049792 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.049770 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h"] Jul 16 13:15:08.055904 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.055886 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-b12c92/auth-test-model-b12c92-kserve-57f997dd5f-mx29h"] Jul 16 13:15:08.209776 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.209745 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af044d90-73cb-4fe4-ae35-7929f6000adb-tls-certs\") pod \"af044d90-73cb-4fe4-ae35-7929f6000adb\" (UID: \"af044d90-73cb-4fe4-ae35-7929f6000adb\") " Jul 16 13:15:08.209776 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.209782 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8brpf\" (UniqueName: \"kubernetes.io/projected/af044d90-73cb-4fe4-ae35-7929f6000adb-kube-api-access-8brpf\") pod \"af044d90-73cb-4fe4-ae35-7929f6000adb\" (UID: \"af044d90-73cb-4fe4-ae35-7929f6000adb\") " Jul 16 13:15:08.209979 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.209835 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af044d90-73cb-4fe4-ae35-7929f6000adb-dshm\") pod \"af044d90-73cb-4fe4-ae35-7929f6000adb\" (UID: \"af044d90-73cb-4fe4-ae35-7929f6000adb\") " Jul 16 13:15:08.209979 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.209882 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af044d90-73cb-4fe4-ae35-7929f6000adb-model-cache\") pod \"af044d90-73cb-4fe4-ae35-7929f6000adb\" (UID: \"af044d90-73cb-4fe4-ae35-7929f6000adb\") " Jul 16 13:15:08.209979 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.209932 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af044d90-73cb-4fe4-ae35-7929f6000adb-home\") pod \"af044d90-73cb-4fe4-ae35-7929f6000adb\" (UID: \"af044d90-73cb-4fe4-ae35-7929f6000adb\") " Jul 16 13:15:08.209979 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.209963 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af044d90-73cb-4fe4-ae35-7929f6000adb-tmp-dir\") pod \"af044d90-73cb-4fe4-ae35-7929f6000adb\" (UID: \"af044d90-73cb-4fe4-ae35-7929f6000adb\") " Jul 16 13:15:08.210291 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.210233 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af044d90-73cb-4fe4-ae35-7929f6000adb-home" (OuterVolumeSpecName: "home") pod "af044d90-73cb-4fe4-ae35-7929f6000adb" (UID: "af044d90-73cb-4fe4-ae35-7929f6000adb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:15:08.210291 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.210268 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af044d90-73cb-4fe4-ae35-7929f6000adb-model-cache" (OuterVolumeSpecName: "model-cache") pod "af044d90-73cb-4fe4-ae35-7929f6000adb" (UID: "af044d90-73cb-4fe4-ae35-7929f6000adb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:15:08.210291 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.210284 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af044d90-73cb-4fe4-ae35-7929f6000adb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "af044d90-73cb-4fe4-ae35-7929f6000adb" (UID: "af044d90-73cb-4fe4-ae35-7929f6000adb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:15:08.212202 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.212176 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af044d90-73cb-4fe4-ae35-7929f6000adb-dshm" (OuterVolumeSpecName: "dshm") pod "af044d90-73cb-4fe4-ae35-7929f6000adb" (UID: "af044d90-73cb-4fe4-ae35-7929f6000adb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:15:08.212312 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.212243 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af044d90-73cb-4fe4-ae35-7929f6000adb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "af044d90-73cb-4fe4-ae35-7929f6000adb" (UID: "af044d90-73cb-4fe4-ae35-7929f6000adb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 13:15:08.212312 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.212268 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af044d90-73cb-4fe4-ae35-7929f6000adb-kube-api-access-8brpf" (OuterVolumeSpecName: "kube-api-access-8brpf") pod "af044d90-73cb-4fe4-ae35-7929f6000adb" (UID: "af044d90-73cb-4fe4-ae35-7929f6000adb"). InnerVolumeSpecName "kube-api-access-8brpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:15:08.311423 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.311332 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af044d90-73cb-4fe4-ae35-7929f6000adb-tls-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:15:08.311423 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.311357 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8brpf\" (UniqueName: \"kubernetes.io/projected/af044d90-73cb-4fe4-ae35-7929f6000adb-kube-api-access-8brpf\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:15:08.311423 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.311365 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af044d90-73cb-4fe4-ae35-7929f6000adb-dshm\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:15:08.311423 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.311375 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af044d90-73cb-4fe4-ae35-7929f6000adb-model-cache\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:15:08.311423 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.311383 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af044d90-73cb-4fe4-ae35-7929f6000adb-home\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:15:08.311423 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.311407 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af044d90-73cb-4fe4-ae35-7929f6000adb-tmp-dir\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:15:08.597101 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.597011 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9" Jul 16 13:15:08.597248 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.597206 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9" Jul 16 13:15:08.609290 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:08.609265 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9" Jul 16 13:15:09.035765 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:09.035734 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44" Jul 16 13:15:09.047184 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:09.047163 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9" Jul 16 13:15:09.055927 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:09.055905 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44"] Jul 16 13:15:09.059583 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:09.059558 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-8d69f8/auth-test-model-8d69f8-kserve-6cd6fb64c-4sw44"] Jul 16 13:15:09.977955 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:09.977921 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59a2b825-2c52-491e-a25d-c54bc317a6bd" path="/var/lib/kubelet/pods/59a2b825-2c52-491e-a25d-c54bc317a6bd/volumes" Jul 16 13:15:09.978322 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:09.978309 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af044d90-73cb-4fe4-ae35-7929f6000adb" path="/var/lib/kubelet/pods/af044d90-73cb-4fe4-ae35-7929f6000adb/volumes" Jul 16 13:15:43.700970 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.700938 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6"] Jul 16 13:15:43.701433 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.701245 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3d3cdf4-c99f-4d12-a2a9-ffc8a9b1e418" containerName="cleanup" Jul 16 13:15:43.701433 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.701256 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3d3cdf4-c99f-4d12-a2a9-ffc8a9b1e418" containerName="cleanup" Jul 16 13:15:43.701433 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.701267 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59a2b825-2c52-491e-a25d-c54bc317a6bd" containerName="main" Jul 16 13:15:43.701433 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.701272 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="59a2b825-2c52-491e-a25d-c54bc317a6bd" containerName="main" Jul 16 13:15:43.701433 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.701279 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af044d90-73cb-4fe4-ae35-7929f6000adb" containerName="main" Jul 16 13:15:43.701433 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.701284 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="af044d90-73cb-4fe4-ae35-7929f6000adb" containerName="main" Jul 16 13:15:43.701433 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.701337 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3d3cdf4-c99f-4d12-a2a9-ffc8a9b1e418" containerName="cleanup" Jul 16 13:15:43.701433 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.701346 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="59a2b825-2c52-491e-a25d-c54bc317a6bd" containerName="main" Jul 16 13:15:43.701433 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.701354 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="af044d90-73cb-4fe4-ae35-7929f6000adb" containerName="main" Jul 16 13:15:43.703105 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.703087 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6" Jul 16 13:15:43.706620 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.706593 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-8d69f8\"/\"sub-test-model-8d69f8-kserve-self-signed-certs\"" Jul 16 13:15:43.706756 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.706615 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-8d69f8\"/\"openshift-service-ca.crt\"" Jul 16 13:15:43.706756 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.706644 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-8d69f8\"/\"default-dockercfg-7c7kk\"" Jul 16 13:15:43.706756 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.706694 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-8d69f8\"/\"kube-root-ca.crt\"" Jul 16 13:15:43.713150 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.713131 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6"] Jul 16 13:15:43.741009 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.740983 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:15:43.802472 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.802444 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a02b9eea-c744-4cba-b5df-b1b297e04dfd-tmp-dir\") pod \"sub-test-model-8d69f8-kserve-86687578c7-nt4m6\" (UID: \"a02b9eea-c744-4cba-b5df-b1b297e04dfd\") " pod="ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6" Jul 16 13:15:43.802601 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.802480 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a02b9eea-c744-4cba-b5df-b1b297e04dfd-dshm\") pod \"sub-test-model-8d69f8-kserve-86687578c7-nt4m6\" (UID: \"a02b9eea-c744-4cba-b5df-b1b297e04dfd\") " pod="ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6" Jul 16 13:15:43.802601 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.802498 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a02b9eea-c744-4cba-b5df-b1b297e04dfd-tls-certs\") pod \"sub-test-model-8d69f8-kserve-86687578c7-nt4m6\" (UID: \"a02b9eea-c744-4cba-b5df-b1b297e04dfd\") " pod="ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6" Jul 16 13:15:43.802601 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.802571 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8sfm\" (UniqueName: \"kubernetes.io/projected/a02b9eea-c744-4cba-b5df-b1b297e04dfd-kube-api-access-n8sfm\") pod \"sub-test-model-8d69f8-kserve-86687578c7-nt4m6\" (UID: \"a02b9eea-c744-4cba-b5df-b1b297e04dfd\") " pod="ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6" Jul 16 13:15:43.802727 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.802606 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a02b9eea-c744-4cba-b5df-b1b297e04dfd-model-cache\") pod \"sub-test-model-8d69f8-kserve-86687578c7-nt4m6\" (UID: \"a02b9eea-c744-4cba-b5df-b1b297e04dfd\") " pod="ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6" Jul 16 13:15:43.802727 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.802697 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a02b9eea-c744-4cba-b5df-b1b297e04dfd-home\") pod \"sub-test-model-8d69f8-kserve-86687578c7-nt4m6\" (UID: \"a02b9eea-c744-4cba-b5df-b1b297e04dfd\") " pod="ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6" Jul 16 13:15:43.903534 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.903497 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a02b9eea-c744-4cba-b5df-b1b297e04dfd-home\") pod \"sub-test-model-8d69f8-kserve-86687578c7-nt4m6\" (UID: \"a02b9eea-c744-4cba-b5df-b1b297e04dfd\") " pod="ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6" Jul 16 13:15:43.903534 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.903538 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a02b9eea-c744-4cba-b5df-b1b297e04dfd-tmp-dir\") pod \"sub-test-model-8d69f8-kserve-86687578c7-nt4m6\" (UID: \"a02b9eea-c744-4cba-b5df-b1b297e04dfd\") " pod="ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6" Jul 16 13:15:43.903770 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.903565 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a02b9eea-c744-4cba-b5df-b1b297e04dfd-dshm\") pod \"sub-test-model-8d69f8-kserve-86687578c7-nt4m6\" (UID: \"a02b9eea-c744-4cba-b5df-b1b297e04dfd\") " pod="ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6" Jul 16 13:15:43.903770 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.903681 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a02b9eea-c744-4cba-b5df-b1b297e04dfd-tls-certs\") pod \"sub-test-model-8d69f8-kserve-86687578c7-nt4m6\" (UID: \"a02b9eea-c744-4cba-b5df-b1b297e04dfd\") " pod="ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6" Jul 16 13:15:43.903770 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.903728 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8sfm\" (UniqueName: \"kubernetes.io/projected/a02b9eea-c744-4cba-b5df-b1b297e04dfd-kube-api-access-n8sfm\") pod \"sub-test-model-8d69f8-kserve-86687578c7-nt4m6\" (UID: \"a02b9eea-c744-4cba-b5df-b1b297e04dfd\") " pod="ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6" Jul 16 13:15:43.903770 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.903755 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a02b9eea-c744-4cba-b5df-b1b297e04dfd-model-cache\") pod \"sub-test-model-8d69f8-kserve-86687578c7-nt4m6\" (UID: \"a02b9eea-c744-4cba-b5df-b1b297e04dfd\") " pod="ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6" Jul 16 13:15:43.903974 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.903924 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a02b9eea-c744-4cba-b5df-b1b297e04dfd-tmp-dir\") pod \"sub-test-model-8d69f8-kserve-86687578c7-nt4m6\" (UID: \"a02b9eea-c744-4cba-b5df-b1b297e04dfd\") " pod="ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6" Jul 16 13:15:43.904044 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.903975 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a02b9eea-c744-4cba-b5df-b1b297e04dfd-home\") pod \"sub-test-model-8d69f8-kserve-86687578c7-nt4m6\" (UID: \"a02b9eea-c744-4cba-b5df-b1b297e04dfd\") " pod="ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6" Jul 16 13:15:43.904117 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.904100 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a02b9eea-c744-4cba-b5df-b1b297e04dfd-model-cache\") pod \"sub-test-model-8d69f8-kserve-86687578c7-nt4m6\" (UID: \"a02b9eea-c744-4cba-b5df-b1b297e04dfd\") " pod="ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6" Jul 16 13:15:43.906110 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.906078 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a02b9eea-c744-4cba-b5df-b1b297e04dfd-dshm\") pod \"sub-test-model-8d69f8-kserve-86687578c7-nt4m6\" (UID: \"a02b9eea-c744-4cba-b5df-b1b297e04dfd\") " pod="ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6" Jul 16 13:15:43.906212 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.906193 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a02b9eea-c744-4cba-b5df-b1b297e04dfd-tls-certs\") pod \"sub-test-model-8d69f8-kserve-86687578c7-nt4m6\" (UID: \"a02b9eea-c744-4cba-b5df-b1b297e04dfd\") " pod="ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6" Jul 16 13:15:43.912571 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:43.912543 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8sfm\" (UniqueName: \"kubernetes.io/projected/a02b9eea-c744-4cba-b5df-b1b297e04dfd-kube-api-access-n8sfm\") pod \"sub-test-model-8d69f8-kserve-86687578c7-nt4m6\" (UID: \"a02b9eea-c744-4cba-b5df-b1b297e04dfd\") " pod="ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6" Jul 16 13:15:44.012521 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:44.012424 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6" Jul 16 13:15:44.148680 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:44.148651 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6"] Jul 16 13:15:44.150127 ip-10-0-140-130 kubenswrapper[2582]: W0716 13:15:44.150102 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda02b9eea_c744_4cba_b5df_b1b297e04dfd.slice/crio-060f98d8a527b733949134de23f28a4ee2ff6c80b052fe452b6fe7253d099a9b WatchSource:0}: Error finding container 060f98d8a527b733949134de23f28a4ee2ff6c80b052fe452b6fe7253d099a9b: Status 404 returned error can't find the container with id 060f98d8a527b733949134de23f28a4ee2ff6c80b052fe452b6fe7253d099a9b Jul 16 13:15:44.208602 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:44.208573 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6" event={"ID":"a02b9eea-c744-4cba-b5df-b1b297e04dfd","Type":"ContainerStarted","Data":"060f98d8a527b733949134de23f28a4ee2ff6c80b052fe452b6fe7253d099a9b"} Jul 16 13:15:44.586767 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:44.586727 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:15:45.215143 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:45.215109 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6" event={"ID":"a02b9eea-c744-4cba-b5df-b1b297e04dfd","Type":"ContainerStarted","Data":"d3bcf0e386d9f9148e1dc42d171c578bc4603e9d69d816f54df3f0c38375eea5"} Jul 16 13:15:45.235275 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:45.235224 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6" podStartSLOduration=2.235207564 podStartE2EDuration="2.235207564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:15:45.233778564 +0000 UTC m=+2203.711524180" watchObservedRunningTime="2026-07-16 13:15:45.235207564 +0000 UTC m=+2203.712953189" Jul 16 13:15:52.360466 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:52.360429 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:15:52.461068 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:52.461033 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:15:54.012960 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:54.012927 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6" Jul 16 13:15:54.013367 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:54.012973 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6" Jul 16 13:15:54.025278 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:54.025257 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6" Jul 16 13:15:54.271461 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:15:54.271368 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6" Jul 16 13:16:24.407625 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:24.407530 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9"] Jul 16 13:16:24.425659 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:24.425607 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:16:25.273304 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:25.273269 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6"] Jul 16 13:16:25.294437 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:25.294383 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:16:33.303851 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:33.303814 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:16:39.427808 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:39.427758 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9" podUID="1546c1fc-d64c-4a2c-9d52-013ed6478944" containerName="main" containerID="cri-o://7f53615dd414c62e168b48e5d28303de76005c90d1c2e622cfb85f797d9532b0" gracePeriod=45 Jul 16 13:16:39.673282 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:39.673253 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9" Jul 16 13:16:39.761155 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:39.761077 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1546c1fc-d64c-4a2c-9d52-013ed6478944-home\") pod \"1546c1fc-d64c-4a2c-9d52-013ed6478944\" (UID: \"1546c1fc-d64c-4a2c-9d52-013ed6478944\") " Jul 16 13:16:39.761311 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:39.761160 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsh5z\" (UniqueName: \"kubernetes.io/projected/1546c1fc-d64c-4a2c-9d52-013ed6478944-kube-api-access-fsh5z\") pod \"1546c1fc-d64c-4a2c-9d52-013ed6478944\" (UID: \"1546c1fc-d64c-4a2c-9d52-013ed6478944\") " Jul 16 13:16:39.761311 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:39.761197 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1546c1fc-d64c-4a2c-9d52-013ed6478944-model-cache\") pod \"1546c1fc-d64c-4a2c-9d52-013ed6478944\" (UID: \"1546c1fc-d64c-4a2c-9d52-013ed6478944\") " Jul 16 13:16:39.761311 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:39.761243 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1546c1fc-d64c-4a2c-9d52-013ed6478944-tmp-dir\") pod \"1546c1fc-d64c-4a2c-9d52-013ed6478944\" (UID: \"1546c1fc-d64c-4a2c-9d52-013ed6478944\") " Jul 16 13:16:39.761311 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:39.761291 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1546c1fc-d64c-4a2c-9d52-013ed6478944-dshm\") pod \"1546c1fc-d64c-4a2c-9d52-013ed6478944\" (UID: \"1546c1fc-d64c-4a2c-9d52-013ed6478944\") " Jul 16 13:16:39.761571 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:39.761324 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1546c1fc-d64c-4a2c-9d52-013ed6478944-tls-certs\") pod \"1546c1fc-d64c-4a2c-9d52-013ed6478944\" (UID: \"1546c1fc-d64c-4a2c-9d52-013ed6478944\") " Jul 16 13:16:39.761571 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:39.761360 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1546c1fc-d64c-4a2c-9d52-013ed6478944-home" (OuterVolumeSpecName: "home") pod "1546c1fc-d64c-4a2c-9d52-013ed6478944" (UID: "1546c1fc-d64c-4a2c-9d52-013ed6478944"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:16:39.761685 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:39.761578 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1546c1fc-d64c-4a2c-9d52-013ed6478944-home\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:16:39.761685 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:39.761590 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1546c1fc-d64c-4a2c-9d52-013ed6478944-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1546c1fc-d64c-4a2c-9d52-013ed6478944" (UID: "1546c1fc-d64c-4a2c-9d52-013ed6478944"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:16:39.764069 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:39.764016 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1546c1fc-d64c-4a2c-9d52-013ed6478944-model-cache" (OuterVolumeSpecName: "model-cache") pod "1546c1fc-d64c-4a2c-9d52-013ed6478944" (UID: "1546c1fc-d64c-4a2c-9d52-013ed6478944"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:16:39.764230 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:39.764125 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1546c1fc-d64c-4a2c-9d52-013ed6478944-kube-api-access-fsh5z" (OuterVolumeSpecName: "kube-api-access-fsh5z") pod "1546c1fc-d64c-4a2c-9d52-013ed6478944" (UID: "1546c1fc-d64c-4a2c-9d52-013ed6478944"). InnerVolumeSpecName "kube-api-access-fsh5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:16:39.764230 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:39.764179 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1546c1fc-d64c-4a2c-9d52-013ed6478944-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1546c1fc-d64c-4a2c-9d52-013ed6478944" (UID: "1546c1fc-d64c-4a2c-9d52-013ed6478944"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 13:16:39.764959 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:39.764820 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1546c1fc-d64c-4a2c-9d52-013ed6478944-dshm" (OuterVolumeSpecName: "dshm") pod "1546c1fc-d64c-4a2c-9d52-013ed6478944" (UID: "1546c1fc-d64c-4a2c-9d52-013ed6478944"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:16:39.862571 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:39.862528 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsh5z\" (UniqueName: \"kubernetes.io/projected/1546c1fc-d64c-4a2c-9d52-013ed6478944-kube-api-access-fsh5z\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:16:39.862571 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:39.862571 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1546c1fc-d64c-4a2c-9d52-013ed6478944-model-cache\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:16:39.862834 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:39.862587 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1546c1fc-d64c-4a2c-9d52-013ed6478944-tmp-dir\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:16:39.862834 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:39.862598 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1546c1fc-d64c-4a2c-9d52-013ed6478944-dshm\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:16:39.862834 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:39.862608 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1546c1fc-d64c-4a2c-9d52-013ed6478944-tls-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:16:40.293423 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:40.293349 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6" podUID="a02b9eea-c744-4cba-b5df-b1b297e04dfd" containerName="main" containerID="cri-o://d3bcf0e386d9f9148e1dc42d171c578bc4603e9d69d816f54df3f0c38375eea5" gracePeriod=45 Jul 16 13:16:40.488951 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:40.488914 2582 generic.go:356] "Generic (PLEG): container finished" podID="1546c1fc-d64c-4a2c-9d52-013ed6478944" containerID="7f53615dd414c62e168b48e5d28303de76005c90d1c2e622cfb85f797d9532b0" exitCode=0 Jul 16 13:16:40.489326 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:40.489007 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9" Jul 16 13:16:40.489326 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:40.489049 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9" event={"ID":"1546c1fc-d64c-4a2c-9d52-013ed6478944","Type":"ContainerDied","Data":"7f53615dd414c62e168b48e5d28303de76005c90d1c2e622cfb85f797d9532b0"} Jul 16 13:16:40.489326 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:40.489087 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9" event={"ID":"1546c1fc-d64c-4a2c-9d52-013ed6478944","Type":"ContainerDied","Data":"d051317676deb23913662326926ffc6e860a7a1c8482fcb1d0e92723c35c5b6e"} Jul 16 13:16:40.489326 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:40.489108 2582 scope.go:117] "RemoveContainer" containerID="7f53615dd414c62e168b48e5d28303de76005c90d1c2e622cfb85f797d9532b0" Jul 16 13:16:40.490996 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:40.490969 2582 generic.go:356] "Generic (PLEG): container finished" podID="a02b9eea-c744-4cba-b5df-b1b297e04dfd" containerID="d3bcf0e386d9f9148e1dc42d171c578bc4603e9d69d816f54df3f0c38375eea5" exitCode=0 Jul 16 13:16:40.491172 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:40.491053 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6" event={"ID":"a02b9eea-c744-4cba-b5df-b1b297e04dfd","Type":"ContainerDied","Data":"d3bcf0e386d9f9148e1dc42d171c578bc4603e9d69d816f54df3f0c38375eea5"} Jul 16 13:16:40.498312 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:40.498295 2582 scope.go:117] "RemoveContainer" containerID="7f53615dd414c62e168b48e5d28303de76005c90d1c2e622cfb85f797d9532b0" Jul 16 13:16:40.498600 ip-10-0-140-130 kubenswrapper[2582]: E0716 13:16:40.498574 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f53615dd414c62e168b48e5d28303de76005c90d1c2e622cfb85f797d9532b0\": container with ID starting with 7f53615dd414c62e168b48e5d28303de76005c90d1c2e622cfb85f797d9532b0 not found: ID does not exist" containerID="7f53615dd414c62e168b48e5d28303de76005c90d1c2e622cfb85f797d9532b0" Jul 16 13:16:40.498669 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:40.498614 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f53615dd414c62e168b48e5d28303de76005c90d1c2e622cfb85f797d9532b0"} err="failed to get container status \"7f53615dd414c62e168b48e5d28303de76005c90d1c2e622cfb85f797d9532b0\": rpc error: code = NotFound desc = could not find container \"7f53615dd414c62e168b48e5d28303de76005c90d1c2e622cfb85f797d9532b0\": container with ID starting with 7f53615dd414c62e168b48e5d28303de76005c90d1c2e622cfb85f797d9532b0 not found: ID does not exist" Jul 16 13:16:40.507773 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:40.507753 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9"] Jul 16 13:16:40.510986 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:40.510966 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-b12c92/sub-test-model-b12c92-kserve-8664d6b8fc-67jb9"] Jul 16 13:16:40.537514 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:40.537495 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6" Jul 16 13:16:40.569040 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:40.568970 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a02b9eea-c744-4cba-b5df-b1b297e04dfd-home\") pod \"a02b9eea-c744-4cba-b5df-b1b297e04dfd\" (UID: \"a02b9eea-c744-4cba-b5df-b1b297e04dfd\") " Jul 16 13:16:40.569040 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:40.569008 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a02b9eea-c744-4cba-b5df-b1b297e04dfd-model-cache\") pod \"a02b9eea-c744-4cba-b5df-b1b297e04dfd\" (UID: \"a02b9eea-c744-4cba-b5df-b1b297e04dfd\") " Jul 16 13:16:40.569040 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:40.569028 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a02b9eea-c744-4cba-b5df-b1b297e04dfd-tmp-dir\") pod \"a02b9eea-c744-4cba-b5df-b1b297e04dfd\" (UID: \"a02b9eea-c744-4cba-b5df-b1b297e04dfd\") " Jul 16 13:16:40.569259 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:40.569049 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a02b9eea-c744-4cba-b5df-b1b297e04dfd-tls-certs\") pod \"a02b9eea-c744-4cba-b5df-b1b297e04dfd\" (UID: \"a02b9eea-c744-4cba-b5df-b1b297e04dfd\") " Jul 16 13:16:40.569259 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:40.569081 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a02b9eea-c744-4cba-b5df-b1b297e04dfd-dshm\") pod \"a02b9eea-c744-4cba-b5df-b1b297e04dfd\" (UID: \"a02b9eea-c744-4cba-b5df-b1b297e04dfd\") " Jul 16 13:16:40.569259 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:40.569117 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8sfm\" (UniqueName: \"kubernetes.io/projected/a02b9eea-c744-4cba-b5df-b1b297e04dfd-kube-api-access-n8sfm\") pod \"a02b9eea-c744-4cba-b5df-b1b297e04dfd\" (UID: \"a02b9eea-c744-4cba-b5df-b1b297e04dfd\") " Jul 16 13:16:40.569259 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:40.569250 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a02b9eea-c744-4cba-b5df-b1b297e04dfd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a02b9eea-c744-4cba-b5df-b1b297e04dfd" (UID: "a02b9eea-c744-4cba-b5df-b1b297e04dfd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:16:40.569503 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:40.569278 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a02b9eea-c744-4cba-b5df-b1b297e04dfd-home" (OuterVolumeSpecName: "home") pod "a02b9eea-c744-4cba-b5df-b1b297e04dfd" (UID: "a02b9eea-c744-4cba-b5df-b1b297e04dfd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:16:40.569503 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:40.569283 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a02b9eea-c744-4cba-b5df-b1b297e04dfd-model-cache" (OuterVolumeSpecName: "model-cache") pod "a02b9eea-c744-4cba-b5df-b1b297e04dfd" (UID: "a02b9eea-c744-4cba-b5df-b1b297e04dfd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:16:40.569503 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:40.569449 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a02b9eea-c744-4cba-b5df-b1b297e04dfd-home\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:16:40.569503 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:40.569466 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a02b9eea-c744-4cba-b5df-b1b297e04dfd-model-cache\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:16:40.569503 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:40.569480 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a02b9eea-c744-4cba-b5df-b1b297e04dfd-tmp-dir\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:16:40.571315 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:40.571288 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a02b9eea-c744-4cba-b5df-b1b297e04dfd-dshm" (OuterVolumeSpecName: "dshm") pod "a02b9eea-c744-4cba-b5df-b1b297e04dfd" (UID: "a02b9eea-c744-4cba-b5df-b1b297e04dfd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:16:40.571436 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:40.571329 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a02b9eea-c744-4cba-b5df-b1b297e04dfd-kube-api-access-n8sfm" (OuterVolumeSpecName: "kube-api-access-n8sfm") pod "a02b9eea-c744-4cba-b5df-b1b297e04dfd" (UID: "a02b9eea-c744-4cba-b5df-b1b297e04dfd"). InnerVolumeSpecName "kube-api-access-n8sfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:16:40.571482 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:40.571445 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a02b9eea-c744-4cba-b5df-b1b297e04dfd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a02b9eea-c744-4cba-b5df-b1b297e04dfd" (UID: "a02b9eea-c744-4cba-b5df-b1b297e04dfd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 13:16:40.669830 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:40.669804 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a02b9eea-c744-4cba-b5df-b1b297e04dfd-tls-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:16:40.669830 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:40.669826 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a02b9eea-c744-4cba-b5df-b1b297e04dfd-dshm\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:16:40.669830 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:40.669835 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8sfm\" (UniqueName: \"kubernetes.io/projected/a02b9eea-c744-4cba-b5df-b1b297e04dfd-kube-api-access-n8sfm\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:16:41.496842 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:41.496813 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6" Jul 16 13:16:41.497320 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:41.496819 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6" event={"ID":"a02b9eea-c744-4cba-b5df-b1b297e04dfd","Type":"ContainerDied","Data":"060f98d8a527b733949134de23f28a4ee2ff6c80b052fe452b6fe7253d099a9b"} Jul 16 13:16:41.497320 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:41.496957 2582 scope.go:117] "RemoveContainer" containerID="d3bcf0e386d9f9148e1dc42d171c578bc4603e9d69d816f54df3f0c38375eea5" Jul 16 13:16:41.530512 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:41.530486 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6"] Jul 16 13:16:41.537344 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:41.537320 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-8d69f8/sub-test-model-8d69f8-kserve-86687578c7-nt4m6"] Jul 16 13:16:41.976678 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:41.976645 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1546c1fc-d64c-4a2c-9d52-013ed6478944" path="/var/lib/kubelet/pods/1546c1fc-d64c-4a2c-9d52-013ed6478944/volumes" Jul 16 13:16:41.977024 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:41.977011 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a02b9eea-c744-4cba-b5df-b1b297e04dfd" path="/var/lib/kubelet/pods/a02b9eea-c744-4cba-b5df-b1b297e04dfd/volumes" Jul 16 13:16:47.689339 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:47.689301 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:16:47.794099 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:47.794058 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:16:55.790172 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:55.790134 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:16:55.892922 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:16:55.892880 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:17:09.291537 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:09.291502 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l"] Jul 16 13:17:09.291912 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:09.291824 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a02b9eea-c744-4cba-b5df-b1b297e04dfd" containerName="main" Jul 16 13:17:09.291912 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:09.291836 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a02b9eea-c744-4cba-b5df-b1b297e04dfd" containerName="main" Jul 16 13:17:09.291912 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:09.291854 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1546c1fc-d64c-4a2c-9d52-013ed6478944" containerName="main" Jul 16 13:17:09.291912 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:09.291859 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1546c1fc-d64c-4a2c-9d52-013ed6478944" containerName="main" Jul 16 13:17:09.292048 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:09.291930 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1546c1fc-d64c-4a2c-9d52-013ed6478944" containerName="main" Jul 16 13:17:09.292048 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:09.291939 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a02b9eea-c744-4cba-b5df-b1b297e04dfd" containerName="main" Jul 16 13:17:09.294051 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:09.294035 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l" Jul 16 13:17:09.297698 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:09.297670 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-8d69f8\"/\"kube-root-ca.crt\"" Jul 16 13:17:09.297891 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:09.297713 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-8d69f8\"/\"openshift-service-ca.crt\"" Jul 16 13:17:09.297891 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:09.297729 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-8d69f8\"/\"rate-test-model-8d69f8-kserve-self-signed-certs\"" Jul 16 13:17:09.297891 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:09.297720 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-8d69f8\"/\"default-dockercfg-7c7kk\"" Jul 16 13:17:09.303568 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:09.303547 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l"] Jul 16 13:17:09.385658 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:09.385622 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:17:09.399375 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:09.399346 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42cfeb90-9935-424f-b24b-49fa1bddd52c-home\") pod \"rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l\" (UID: \"42cfeb90-9935-424f-b24b-49fa1bddd52c\") " pod="ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l" Jul 16 13:17:09.399375 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:09.399376 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42cfeb90-9935-424f-b24b-49fa1bddd52c-tmp-dir\") pod \"rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l\" (UID: \"42cfeb90-9935-424f-b24b-49fa1bddd52c\") " pod="ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l" Jul 16 13:17:09.399586 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:09.399458 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42cfeb90-9935-424f-b24b-49fa1bddd52c-dshm\") pod \"rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l\" (UID: \"42cfeb90-9935-424f-b24b-49fa1bddd52c\") " pod="ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l" Jul 16 13:17:09.399586 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:09.399488 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2jv5\" (UniqueName: \"kubernetes.io/projected/42cfeb90-9935-424f-b24b-49fa1bddd52c-kube-api-access-x2jv5\") pod \"rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l\" (UID: \"42cfeb90-9935-424f-b24b-49fa1bddd52c\") " pod="ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l" Jul 16 13:17:09.399586 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:09.399555 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42cfeb90-9935-424f-b24b-49fa1bddd52c-model-cache\") pod \"rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l\" (UID: \"42cfeb90-9935-424f-b24b-49fa1bddd52c\") " pod="ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l" Jul 16 13:17:09.399714 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:09.399588 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42cfeb90-9935-424f-b24b-49fa1bddd52c-tls-certs\") pod \"rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l\" (UID: \"42cfeb90-9935-424f-b24b-49fa1bddd52c\") " pod="ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l" Jul 16 13:17:09.500290 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:09.500247 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42cfeb90-9935-424f-b24b-49fa1bddd52c-home\") pod \"rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l\" (UID: \"42cfeb90-9935-424f-b24b-49fa1bddd52c\") " pod="ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l" Jul 16 13:17:09.500290 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:09.500288 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42cfeb90-9935-424f-b24b-49fa1bddd52c-tmp-dir\") pod \"rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l\" (UID: \"42cfeb90-9935-424f-b24b-49fa1bddd52c\") " pod="ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l" Jul 16 13:17:09.500581 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:09.500310 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42cfeb90-9935-424f-b24b-49fa1bddd52c-dshm\") pod \"rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l\" (UID: \"42cfeb90-9935-424f-b24b-49fa1bddd52c\") " pod="ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l" Jul 16 13:17:09.500581 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:09.500329 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2jv5\" (UniqueName: \"kubernetes.io/projected/42cfeb90-9935-424f-b24b-49fa1bddd52c-kube-api-access-x2jv5\") pod \"rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l\" (UID: \"42cfeb90-9935-424f-b24b-49fa1bddd52c\") " pod="ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l" Jul 16 13:17:09.500581 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:09.500368 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42cfeb90-9935-424f-b24b-49fa1bddd52c-model-cache\") pod \"rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l\" (UID: \"42cfeb90-9935-424f-b24b-49fa1bddd52c\") " pod="ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l" Jul 16 13:17:09.500581 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:09.500383 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42cfeb90-9935-424f-b24b-49fa1bddd52c-tls-certs\") pod \"rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l\" (UID: \"42cfeb90-9935-424f-b24b-49fa1bddd52c\") " pod="ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l" Jul 16 13:17:09.500779 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:09.500734 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42cfeb90-9935-424f-b24b-49fa1bddd52c-tmp-dir\") pod \"rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l\" (UID: \"42cfeb90-9935-424f-b24b-49fa1bddd52c\") " pod="ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l" Jul 16 13:17:09.500779 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:09.500742 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42cfeb90-9935-424f-b24b-49fa1bddd52c-home\") pod \"rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l\" (UID: \"42cfeb90-9935-424f-b24b-49fa1bddd52c\") " pod="ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l" Jul 16 13:17:09.500896 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:09.500824 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42cfeb90-9935-424f-b24b-49fa1bddd52c-model-cache\") pod \"rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l\" (UID: \"42cfeb90-9935-424f-b24b-49fa1bddd52c\") " pod="ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l" Jul 16 13:17:09.502780 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:09.502750 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42cfeb90-9935-424f-b24b-49fa1bddd52c-dshm\") pod \"rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l\" (UID: \"42cfeb90-9935-424f-b24b-49fa1bddd52c\") " pod="ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l" Jul 16 13:17:09.502940 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:09.502922 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42cfeb90-9935-424f-b24b-49fa1bddd52c-tls-certs\") pod \"rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l\" (UID: \"42cfeb90-9935-424f-b24b-49fa1bddd52c\") " pod="ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l" Jul 16 13:17:09.508547 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:09.508527 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2jv5\" (UniqueName: \"kubernetes.io/projected/42cfeb90-9935-424f-b24b-49fa1bddd52c-kube-api-access-x2jv5\") pod \"rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l\" (UID: \"42cfeb90-9935-424f-b24b-49fa1bddd52c\") " pod="ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l" Jul 16 13:17:09.602993 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:09.602887 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l" Jul 16 13:17:09.742448 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:09.742421 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l"] Jul 16 13:17:09.743651 ip-10-0-140-130 kubenswrapper[2582]: W0716 13:17:09.743626 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42cfeb90_9935_424f_b24b_49fa1bddd52c.slice/crio-1e15ad06aae1c9be9c2108859a2a4a2af8f6a0007e486dc581b172d0f82509a5 WatchSource:0}: Error finding container 1e15ad06aae1c9be9c2108859a2a4a2af8f6a0007e486dc581b172d0f82509a5: Status 404 returned error can't find the container with id 1e15ad06aae1c9be9c2108859a2a4a2af8f6a0007e486dc581b172d0f82509a5 Jul 16 13:17:10.642215 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:10.642181 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l" event={"ID":"42cfeb90-9935-424f-b24b-49fa1bddd52c","Type":"ContainerStarted","Data":"9dda8df661eda6bff8fb2d7679d82b44953deeea9992d1390e49cf442089f00c"} Jul 16 13:17:10.642215 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:10.642221 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l" event={"ID":"42cfeb90-9935-424f-b24b-49fa1bddd52c","Type":"ContainerStarted","Data":"1e15ad06aae1c9be9c2108859a2a4a2af8f6a0007e486dc581b172d0f82509a5"} Jul 16 13:17:10.660405 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:10.660311 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l" podStartSLOduration=1.660295764 podStartE2EDuration="1.660295764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:17:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:17:10.659172981 +0000 UTC m=+2289.136918598" watchObservedRunningTime="2026-07-16 13:17:10.660295764 +0000 UTC m=+2289.138041381" Jul 16 13:17:19.603640 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:19.603598 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l" Jul 16 13:17:19.604037 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:19.603689 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l" Jul 16 13:17:19.616447 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:19.616423 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l" Jul 16 13:17:19.699234 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:19.699208 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l" Jul 16 13:17:33.629240 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:33.629205 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:17:44.583465 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:44.583424 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:17:57.740140 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:57.740060 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:17:58.661328 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:58.661296 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l"] Jul 16 13:17:58.680617 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:58.680592 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:17:59.704731 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:59.704692 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7563-139693656785840"] Jul 16 13:17:59.707927 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:59.707895 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7563-139693656785840" Jul 16 13:17:59.710508 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:59.710492 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7b9p2\"" Jul 16 13:17:59.715105 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:59.715079 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7563-139693656785840"] Jul 16 13:17:59.811326 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:59.811293 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6cft\" (UniqueName: \"kubernetes.io/projected/40b9d57f-cbdc-46e3-84b8-1378735690a8-kube-api-access-p6cft\") pod \"test-curl-7563-139693656785840\" (UID: \"40b9d57f-cbdc-46e3-84b8-1378735690a8\") " pod="opendatahub/test-curl-7563-139693656785840" Jul 16 13:17:59.911870 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:59.911833 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6cft\" (UniqueName: \"kubernetes.io/projected/40b9d57f-cbdc-46e3-84b8-1378735690a8-kube-api-access-p6cft\") pod \"test-curl-7563-139693656785840\" (UID: \"40b9d57f-cbdc-46e3-84b8-1378735690a8\") " pod="opendatahub/test-curl-7563-139693656785840" Jul 16 13:17:59.920788 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:17:59.920766 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6cft\" (UniqueName: \"kubernetes.io/projected/40b9d57f-cbdc-46e3-84b8-1378735690a8-kube-api-access-p6cft\") pod \"test-curl-7563-139693656785840\" (UID: \"40b9d57f-cbdc-46e3-84b8-1378735690a8\") " pod="opendatahub/test-curl-7563-139693656785840" Jul 16 13:18:00.018175 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:00.018096 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7563-139693656785840" Jul 16 13:18:00.152502 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:00.152478 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7563-139693656785840"] Jul 16 13:18:00.154216 ip-10-0-140-130 kubenswrapper[2582]: W0716 13:18:00.154190 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40b9d57f_cbdc_46e3_84b8_1378735690a8.slice/crio-e207fa3fa9326246e16eab364a242ff4ef0d9b7c25776f23ce59fe4aac721e23 WatchSource:0}: Error finding container e207fa3fa9326246e16eab364a242ff4ef0d9b7c25776f23ce59fe4aac721e23: Status 404 returned error can't find the container with id e207fa3fa9326246e16eab364a242ff4ef0d9b7c25776f23ce59fe4aac721e23 Jul 16 13:18:00.891183 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:00.891149 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7563-139693656785840" event={"ID":"40b9d57f-cbdc-46e3-84b8-1378735690a8","Type":"ContainerStarted","Data":"e207fa3fa9326246e16eab364a242ff4ef0d9b7c25776f23ce59fe4aac721e23"} Jul 16 13:18:01.897280 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:01.897249 2582 generic.go:356] "Generic (PLEG): container finished" podID="40b9d57f-cbdc-46e3-84b8-1378735690a8" containerID="ed06c2269dd5c1914a3247e143907c11719b10575337e190afe9070cda206128" exitCode=0 Jul 16 13:18:01.897630 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:01.897332 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7563-139693656785840" event={"ID":"40b9d57f-cbdc-46e3-84b8-1378735690a8","Type":"ContainerDied","Data":"ed06c2269dd5c1914a3247e143907c11719b10575337e190afe9070cda206128"} Jul 16 13:18:03.026919 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:03.026894 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7563-139693656785840" Jul 16 13:18:03.048281 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:03.048257 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7563-139693656785840_40b9d57f-cbdc-46e3-84b8-1378735690a8/test-curl-7563-139693656785840/0.log" Jul 16 13:18:03.072598 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:03.072573 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7563-139693656785840"] Jul 16 13:18:03.078970 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:03.078950 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7563-139693656785840"] Jul 16 13:18:03.140337 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:03.140318 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6cft\" (UniqueName: \"kubernetes.io/projected/40b9d57f-cbdc-46e3-84b8-1378735690a8-kube-api-access-p6cft\") pod \"40b9d57f-cbdc-46e3-84b8-1378735690a8\" (UID: \"40b9d57f-cbdc-46e3-84b8-1378735690a8\") " Jul 16 13:18:03.142556 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:03.142530 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b9d57f-cbdc-46e3-84b8-1378735690a8-kube-api-access-p6cft" (OuterVolumeSpecName: "kube-api-access-p6cft") pod "40b9d57f-cbdc-46e3-84b8-1378735690a8" (UID: "40b9d57f-cbdc-46e3-84b8-1378735690a8"). InnerVolumeSpecName "kube-api-access-p6cft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:18:03.207644 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:03.207587 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7563-139693656911152"] Jul 16 13:18:03.207887 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:03.207874 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40b9d57f-cbdc-46e3-84b8-1378735690a8" containerName="test-curl-7563-139693656785840" Jul 16 13:18:03.207933 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:03.207888 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="40b9d57f-cbdc-46e3-84b8-1378735690a8" containerName="test-curl-7563-139693656785840" Jul 16 13:18:03.207965 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:03.207946 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="40b9d57f-cbdc-46e3-84b8-1378735690a8" containerName="test-curl-7563-139693656785840" Jul 16 13:18:03.210741 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:03.210726 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7563-139693656911152" Jul 16 13:18:03.218237 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:03.218217 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7563-139693656911152"] Jul 16 13:18:03.241810 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:03.241787 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6cft\" (UniqueName: \"kubernetes.io/projected/40b9d57f-cbdc-46e3-84b8-1378735690a8-kube-api-access-p6cft\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:18:03.343101 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:03.343070 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mblv5\" (UniqueName: \"kubernetes.io/projected/33e1355e-ae22-4c2a-9723-4ff0f946f764-kube-api-access-mblv5\") pod \"test-curl-7563-139693656911152\" (UID: \"33e1355e-ae22-4c2a-9723-4ff0f946f764\") " pod="opendatahub/test-curl-7563-139693656911152" Jul 16 13:18:03.443636 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:03.443603 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mblv5\" (UniqueName: \"kubernetes.io/projected/33e1355e-ae22-4c2a-9723-4ff0f946f764-kube-api-access-mblv5\") pod \"test-curl-7563-139693656911152\" (UID: \"33e1355e-ae22-4c2a-9723-4ff0f946f764\") " pod="opendatahub/test-curl-7563-139693656911152" Jul 16 13:18:03.452135 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:03.452112 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mblv5\" (UniqueName: \"kubernetes.io/projected/33e1355e-ae22-4c2a-9723-4ff0f946f764-kube-api-access-mblv5\") pod \"test-curl-7563-139693656911152\" (UID: \"33e1355e-ae22-4c2a-9723-4ff0f946f764\") " pod="opendatahub/test-curl-7563-139693656911152" Jul 16 13:18:03.520374 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:03.520286 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7563-139693656911152" Jul 16 13:18:03.647016 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:03.646993 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7563-139693656911152"] Jul 16 13:18:03.648442 ip-10-0-140-130 kubenswrapper[2582]: W0716 13:18:03.648418 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33e1355e_ae22_4c2a_9723_4ff0f946f764.slice/crio-8a25c81384f2e63c640e1187df847c08a5cfda40ddc905da5a4da2393177965e WatchSource:0}: Error finding container 8a25c81384f2e63c640e1187df847c08a5cfda40ddc905da5a4da2393177965e: Status 404 returned error can't find the container with id 8a25c81384f2e63c640e1187df847c08a5cfda40ddc905da5a4da2393177965e Jul 16 13:18:03.908535 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:03.908506 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7563-139693656785840" Jul 16 13:18:03.908709 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:03.908512 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e207fa3fa9326246e16eab364a242ff4ef0d9b7c25776f23ce59fe4aac721e23" Jul 16 13:18:03.909783 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:03.909761 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7563-139693656911152" event={"ID":"33e1355e-ae22-4c2a-9723-4ff0f946f764","Type":"ContainerStarted","Data":"8a25c81384f2e63c640e1187df847c08a5cfda40ddc905da5a4da2393177965e"} Jul 16 13:18:03.918575 ip-10-0-140-130 kubenswrapper[2582]: E0716 13:18:03.918551 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7563-139693656785840\" is forbidden: User \"system:node:ip-10-0-140-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-130.ec2.internal' and this object" podUID="40b9d57f-cbdc-46e3-84b8-1378735690a8" pod="opendatahub/test-curl-7563-139693656785840" Jul 16 13:18:03.978430 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:03.978379 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40b9d57f-cbdc-46e3-84b8-1378735690a8" path="/var/lib/kubelet/pods/40b9d57f-cbdc-46e3-84b8-1378735690a8/volumes" Jul 16 13:18:04.916535 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:04.916495 2582 generic.go:356] "Generic (PLEG): container finished" podID="33e1355e-ae22-4c2a-9723-4ff0f946f764" containerID="4f27a3d3549584646a1964854c8cceb4ba8f4ab88971998e125e7da6e290bf1f" exitCode=0 Jul 16 13:18:04.916927 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:04.916578 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7563-139693656911152" event={"ID":"33e1355e-ae22-4c2a-9723-4ff0f946f764","Type":"ContainerDied","Data":"4f27a3d3549584646a1964854c8cceb4ba8f4ab88971998e125e7da6e290bf1f"} Jul 16 13:18:06.061607 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:06.061580 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7563-139693656911152" Jul 16 13:18:06.083926 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:06.083890 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7563-139693656911152_33e1355e-ae22-4c2a-9723-4ff0f946f764/test-curl-7563-139693656911152/0.log" Jul 16 13:18:06.108825 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:06.108801 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7563-139693656911152"] Jul 16 13:18:06.112711 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:06.112678 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7563-139693656911152"] Jul 16 13:18:06.175040 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:06.174940 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mblv5\" (UniqueName: \"kubernetes.io/projected/33e1355e-ae22-4c2a-9723-4ff0f946f764-kube-api-access-mblv5\") pod \"33e1355e-ae22-4c2a-9723-4ff0f946f764\" (UID: \"33e1355e-ae22-4c2a-9723-4ff0f946f764\") " Jul 16 13:18:06.177197 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:06.177172 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e1355e-ae22-4c2a-9723-4ff0f946f764-kube-api-access-mblv5" (OuterVolumeSpecName: "kube-api-access-mblv5") pod "33e1355e-ae22-4c2a-9723-4ff0f946f764" (UID: "33e1355e-ae22-4c2a-9723-4ff0f946f764"). InnerVolumeSpecName "kube-api-access-mblv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:18:06.272063 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:06.272029 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7563-139693652434352"] Jul 16 13:18:06.272486 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:06.272453 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33e1355e-ae22-4c2a-9723-4ff0f946f764" containerName="test-curl-7563-139693656911152" Jul 16 13:18:06.272636 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:06.272493 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="33e1355e-ae22-4c2a-9723-4ff0f946f764" containerName="test-curl-7563-139693656911152" Jul 16 13:18:06.272636 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:06.272573 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="33e1355e-ae22-4c2a-9723-4ff0f946f764" containerName="test-curl-7563-139693656911152" Jul 16 13:18:06.275475 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:06.275448 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7563-139693652434352" Jul 16 13:18:06.275796 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:06.275772 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mblv5\" (UniqueName: \"kubernetes.io/projected/33e1355e-ae22-4c2a-9723-4ff0f946f764-kube-api-access-mblv5\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:18:06.279469 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:06.279446 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7563-139693652434352"] Jul 16 13:18:06.376476 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:06.376446 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtrtn\" (UniqueName: \"kubernetes.io/projected/46fb0fdc-1010-41f2-892d-933e05fb1d4c-kube-api-access-vtrtn\") pod \"test-curl-7563-139693652434352\" (UID: \"46fb0fdc-1010-41f2-892d-933e05fb1d4c\") " pod="opendatahub/test-curl-7563-139693652434352" Jul 16 13:18:06.477303 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:06.477219 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtrtn\" (UniqueName: \"kubernetes.io/projected/46fb0fdc-1010-41f2-892d-933e05fb1d4c-kube-api-access-vtrtn\") pod \"test-curl-7563-139693652434352\" (UID: \"46fb0fdc-1010-41f2-892d-933e05fb1d4c\") " pod="opendatahub/test-curl-7563-139693652434352" Jul 16 13:18:06.486365 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:06.486338 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtrtn\" (UniqueName: \"kubernetes.io/projected/46fb0fdc-1010-41f2-892d-933e05fb1d4c-kube-api-access-vtrtn\") pod \"test-curl-7563-139693652434352\" (UID: \"46fb0fdc-1010-41f2-892d-933e05fb1d4c\") " pod="opendatahub/test-curl-7563-139693652434352" Jul 16 13:18:06.585297 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:06.585269 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7563-139693652434352" Jul 16 13:18:06.712980 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:06.712957 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7563-139693652434352"] Jul 16 13:18:06.714044 ip-10-0-140-130 kubenswrapper[2582]: W0716 13:18:06.714015 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46fb0fdc_1010_41f2_892d_933e05fb1d4c.slice/crio-76d0acda9dfc042e707ead8db8e8da00a9f49bfcb41c58ce68da369b28a8b2c1 WatchSource:0}: Error finding container 76d0acda9dfc042e707ead8db8e8da00a9f49bfcb41c58ce68da369b28a8b2c1: Status 404 returned error can't find the container with id 76d0acda9dfc042e707ead8db8e8da00a9f49bfcb41c58ce68da369b28a8b2c1 Jul 16 13:18:06.927708 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:06.927669 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7563-139693656911152" Jul 16 13:18:06.927708 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:06.927686 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a25c81384f2e63c640e1187df847c08a5cfda40ddc905da5a4da2393177965e" Jul 16 13:18:06.928964 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:06.928939 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7563-139693652434352" event={"ID":"46fb0fdc-1010-41f2-892d-933e05fb1d4c","Type":"ContainerStarted","Data":"76d0acda9dfc042e707ead8db8e8da00a9f49bfcb41c58ce68da369b28a8b2c1"} Jul 16 13:18:06.937592 ip-10-0-140-130 kubenswrapper[2582]: E0716 13:18:06.937568 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7563-139693656911152\" is forbidden: User \"system:node:ip-10-0-140-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-130.ec2.internal' and this object" podUID="33e1355e-ae22-4c2a-9723-4ff0f946f764" pod="opendatahub/test-curl-7563-139693656911152" Jul 16 13:18:07.709062 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:07.709022 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:18:07.809128 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:07.809088 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:18:07.935076 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:07.935043 2582 generic.go:356] "Generic (PLEG): container finished" podID="46fb0fdc-1010-41f2-892d-933e05fb1d4c" containerID="66d75641d5badd21e25cf2176dee1db46590d94913136dc90e4f37e26cd38176" exitCode=0 Jul 16 13:18:07.935242 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:07.935135 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7563-139693652434352" event={"ID":"46fb0fdc-1010-41f2-892d-933e05fb1d4c","Type":"ContainerDied","Data":"66d75641d5badd21e25cf2176dee1db46590d94913136dc90e4f37e26cd38176"} Jul 16 13:18:07.936977 ip-10-0-140-130 kubenswrapper[2582]: E0716 13:18:07.936955 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7563-139693656911152\" is forbidden: User \"system:node:ip-10-0-140-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-130.ec2.internal' and this object" podUID="33e1355e-ae22-4c2a-9723-4ff0f946f764" pod="opendatahub/test-curl-7563-139693656911152" Jul 16 13:18:07.976951 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:07.976885 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33e1355e-ae22-4c2a-9723-4ff0f946f764" path="/var/lib/kubelet/pods/33e1355e-ae22-4c2a-9723-4ff0f946f764/volumes" Jul 16 13:18:09.065025 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:09.065004 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7563-139693652434352" Jul 16 13:18:09.086046 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:09.086020 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7563-139693652434352_46fb0fdc-1010-41f2-892d-933e05fb1d4c/test-curl-7563-139693652434352/0.log" Jul 16 13:18:09.111505 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:09.111480 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7563-139693652434352"] Jul 16 13:18:09.115348 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:09.115327 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7563-139693652434352"] Jul 16 13:18:09.201285 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:09.201205 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtrtn\" (UniqueName: \"kubernetes.io/projected/46fb0fdc-1010-41f2-892d-933e05fb1d4c-kube-api-access-vtrtn\") pod \"46fb0fdc-1010-41f2-892d-933e05fb1d4c\" (UID: \"46fb0fdc-1010-41f2-892d-933e05fb1d4c\") " Jul 16 13:18:09.203251 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:09.203223 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46fb0fdc-1010-41f2-892d-933e05fb1d4c-kube-api-access-vtrtn" (OuterVolumeSpecName: "kube-api-access-vtrtn") pod "46fb0fdc-1010-41f2-892d-933e05fb1d4c" (UID: "46fb0fdc-1010-41f2-892d-933e05fb1d4c"). InnerVolumeSpecName "kube-api-access-vtrtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:18:09.260426 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:09.260381 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7563-139693652434352"] Jul 16 13:18:09.260730 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:09.260716 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46fb0fdc-1010-41f2-892d-933e05fb1d4c" containerName="test-curl-7563-139693652434352" Jul 16 13:18:09.260775 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:09.260732 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="46fb0fdc-1010-41f2-892d-933e05fb1d4c" containerName="test-curl-7563-139693652434352" Jul 16 13:18:09.260811 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:09.260796 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="46fb0fdc-1010-41f2-892d-933e05fb1d4c" containerName="test-curl-7563-139693652434352" Jul 16 13:18:09.263549 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:09.263533 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7563-139693652434352" Jul 16 13:18:09.270689 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:09.270669 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7563-139693652434352"] Jul 16 13:18:09.301816 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:09.301788 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtrtn\" (UniqueName: \"kubernetes.io/projected/46fb0fdc-1010-41f2-892d-933e05fb1d4c-kube-api-access-vtrtn\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:18:09.403014 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:09.402982 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghm2d\" (UniqueName: \"kubernetes.io/projected/4c0517b4-3ab6-4920-9a9f-f08bd5b0d8b5-kube-api-access-ghm2d\") pod \"test-curl-7563-139693652434352\" (UID: \"4c0517b4-3ab6-4920-9a9f-f08bd5b0d8b5\") " pod="opendatahub/test-curl-7563-139693652434352" Jul 16 13:18:09.504472 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:09.504377 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghm2d\" (UniqueName: \"kubernetes.io/projected/4c0517b4-3ab6-4920-9a9f-f08bd5b0d8b5-kube-api-access-ghm2d\") pod \"test-curl-7563-139693652434352\" (UID: \"4c0517b4-3ab6-4920-9a9f-f08bd5b0d8b5\") " pod="opendatahub/test-curl-7563-139693652434352" Jul 16 13:18:09.515169 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:09.515140 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghm2d\" (UniqueName: \"kubernetes.io/projected/4c0517b4-3ab6-4920-9a9f-f08bd5b0d8b5-kube-api-access-ghm2d\") pod \"test-curl-7563-139693652434352\" (UID: \"4c0517b4-3ab6-4920-9a9f-f08bd5b0d8b5\") " pod="opendatahub/test-curl-7563-139693652434352" Jul 16 13:18:09.572345 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:09.572303 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7563-139693652434352" Jul 16 13:18:09.700018 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:09.699995 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7563-139693652434352"] Jul 16 13:18:09.701913 ip-10-0-140-130 kubenswrapper[2582]: W0716 13:18:09.701889 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c0517b4_3ab6_4920_9a9f_f08bd5b0d8b5.slice/crio-8b58496bd805f5f53f7ad1632eef4be0bdd045d3b5e82e4095c6a8c4549dd121 WatchSource:0}: Error finding container 8b58496bd805f5f53f7ad1632eef4be0bdd045d3b5e82e4095c6a8c4549dd121: Status 404 returned error can't find the container with id 8b58496bd805f5f53f7ad1632eef4be0bdd045d3b5e82e4095c6a8c4549dd121 Jul 16 13:18:09.948035 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:09.948007 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76d0acda9dfc042e707ead8db8e8da00a9f49bfcb41c58ce68da369b28a8b2c1" Jul 16 13:18:09.948035 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:09.948013 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7563-139693652434352" Jul 16 13:18:09.949135 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:09.949105 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7563-139693652434352" event={"ID":"4c0517b4-3ab6-4920-9a9f-f08bd5b0d8b5","Type":"ContainerStarted","Data":"8b58496bd805f5f53f7ad1632eef4be0bdd045d3b5e82e4095c6a8c4549dd121"} Jul 16 13:18:09.959796 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:09.959768 2582 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7563-139693652434352" oldPodUID="46fb0fdc-1010-41f2-892d-933e05fb1d4c" podUID="4c0517b4-3ab6-4920-9a9f-f08bd5b0d8b5" Jul 16 13:18:09.977049 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:09.977026 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46fb0fdc-1010-41f2-892d-933e05fb1d4c" path="/var/lib/kubelet/pods/46fb0fdc-1010-41f2-892d-933e05fb1d4c/volumes" Jul 16 13:18:10.955444 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:10.955409 2582 generic.go:356] "Generic (PLEG): container finished" podID="4c0517b4-3ab6-4920-9a9f-f08bd5b0d8b5" containerID="d25e80165d7ca34e9a3caa6e8ede20760a7b36c198960e60b3119bf4f8550efa" exitCode=0 Jul 16 13:18:10.955843 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:10.955473 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7563-139693652434352" event={"ID":"4c0517b4-3ab6-4920-9a9f-f08bd5b0d8b5","Type":"ContainerDied","Data":"d25e80165d7ca34e9a3caa6e8ede20760a7b36c198960e60b3119bf4f8550efa"} Jul 16 13:18:12.088104 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:12.088078 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7563-139693652434352" Jul 16 13:18:12.114036 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:12.114015 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7563-139693652434352_4c0517b4-3ab6-4920-9a9f-f08bd5b0d8b5/test-curl-7563-139693652434352/0.log" Jul 16 13:18:12.162059 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:12.162025 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7563-139693652434352"] Jul 16 13:18:12.171993 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:12.171970 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7563-139693652434352"] Jul 16 13:18:12.226163 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:12.226105 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghm2d\" (UniqueName: \"kubernetes.io/projected/4c0517b4-3ab6-4920-9a9f-f08bd5b0d8b5-kube-api-access-ghm2d\") pod \"4c0517b4-3ab6-4920-9a9f-f08bd5b0d8b5\" (UID: \"4c0517b4-3ab6-4920-9a9f-f08bd5b0d8b5\") " Jul 16 13:18:12.228228 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:12.228199 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c0517b4-3ab6-4920-9a9f-f08bd5b0d8b5-kube-api-access-ghm2d" (OuterVolumeSpecName: "kube-api-access-ghm2d") pod "4c0517b4-3ab6-4920-9a9f-f08bd5b0d8b5" (UID: "4c0517b4-3ab6-4920-9a9f-f08bd5b0d8b5"). InnerVolumeSpecName "kube-api-access-ghm2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:18:12.298953 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:12.298922 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7563-139693653627504"] Jul 16 13:18:12.299222 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:12.299210 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c0517b4-3ab6-4920-9a9f-f08bd5b0d8b5" containerName="test-curl-7563-139693652434352" Jul 16 13:18:12.299264 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:12.299224 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c0517b4-3ab6-4920-9a9f-f08bd5b0d8b5" containerName="test-curl-7563-139693652434352" Jul 16 13:18:12.299297 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:12.299290 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c0517b4-3ab6-4920-9a9f-f08bd5b0d8b5" containerName="test-curl-7563-139693652434352" Jul 16 13:18:12.302043 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:12.302025 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7563-139693653627504" Jul 16 13:18:12.308172 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:12.308148 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7563-139693653627504"] Jul 16 13:18:12.327128 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:12.327104 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghm2d\" (UniqueName: \"kubernetes.io/projected/4c0517b4-3ab6-4920-9a9f-f08bd5b0d8b5-kube-api-access-ghm2d\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:18:12.427888 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:12.427863 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnr5b\" (UniqueName: \"kubernetes.io/projected/16c57579-06fe-4233-9fce-634b4f49bf95-kube-api-access-cnr5b\") pod \"test-curl-7563-139693653627504\" (UID: \"16c57579-06fe-4233-9fce-634b4f49bf95\") " pod="opendatahub/test-curl-7563-139693653627504" Jul 16 13:18:12.528326 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:12.528250 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnr5b\" (UniqueName: \"kubernetes.io/projected/16c57579-06fe-4233-9fce-634b4f49bf95-kube-api-access-cnr5b\") pod \"test-curl-7563-139693653627504\" (UID: \"16c57579-06fe-4233-9fce-634b4f49bf95\") " pod="opendatahub/test-curl-7563-139693653627504" Jul 16 13:18:12.536675 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:12.536650 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnr5b\" (UniqueName: \"kubernetes.io/projected/16c57579-06fe-4233-9fce-634b4f49bf95-kube-api-access-cnr5b\") pod \"test-curl-7563-139693653627504\" (UID: \"16c57579-06fe-4233-9fce-634b4f49bf95\") " pod="opendatahub/test-curl-7563-139693653627504" Jul 16 13:18:12.611905 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:12.611882 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7563-139693653627504" Jul 16 13:18:12.740498 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:12.740472 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7563-139693653627504"] Jul 16 13:18:12.742072 ip-10-0-140-130 kubenswrapper[2582]: W0716 13:18:12.742049 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16c57579_06fe_4233_9fce_634b4f49bf95.slice/crio-06116582feef79aa3bf529e5fbf190c36d9cca24c7052a78ec5a0030818fa943 WatchSource:0}: Error finding container 06116582feef79aa3bf529e5fbf190c36d9cca24c7052a78ec5a0030818fa943: Status 404 returned error can't find the container with id 06116582feef79aa3bf529e5fbf190c36d9cca24c7052a78ec5a0030818fa943 Jul 16 13:18:12.966812 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:12.966778 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7563-139693653627504" event={"ID":"16c57579-06fe-4233-9fce-634b4f49bf95","Type":"ContainerStarted","Data":"06116582feef79aa3bf529e5fbf190c36d9cca24c7052a78ec5a0030818fa943"} Jul 16 13:18:12.967930 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:12.967907 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7563-139693652434352" Jul 16 13:18:12.967930 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:12.967922 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b58496bd805f5f53f7ad1632eef4be0bdd045d3b5e82e4095c6a8c4549dd121" Jul 16 13:18:12.977739 ip-10-0-140-130 kubenswrapper[2582]: E0716 13:18:12.977716 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7563-139693652434352\" is forbidden: User \"system:node:ip-10-0-140-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-130.ec2.internal' and this object" podUID="4c0517b4-3ab6-4920-9a9f-f08bd5b0d8b5" pod="opendatahub/test-curl-7563-139693652434352" Jul 16 13:18:13.681670 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:13.681623 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l" podUID="42cfeb90-9935-424f-b24b-49fa1bddd52c" containerName="main" containerID="cri-o://9dda8df661eda6bff8fb2d7679d82b44953deeea9992d1390e49cf442089f00c" gracePeriod=45 Jul 16 13:18:13.926296 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:13.926275 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l" Jul 16 13:18:13.945989 ip-10-0-140-130 kubenswrapper[2582]: E0716 13:18:13.945925 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7563-139693652434352\" is forbidden: User \"system:node:ip-10-0-140-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-130.ec2.internal' and this object" podUID="4c0517b4-3ab6-4920-9a9f-f08bd5b0d8b5" pod="opendatahub/test-curl-7563-139693652434352" Jul 16 13:18:13.974123 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:13.974100 2582 generic.go:356] "Generic (PLEG): container finished" podID="16c57579-06fe-4233-9fce-634b4f49bf95" containerID="cdbdf2cf5305103f4b925226e0c20ed45b657a56907cdabe344c5881918b77d4" exitCode=0 Jul 16 13:18:13.975513 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:13.975490 2582 generic.go:356] "Generic (PLEG): container finished" podID="42cfeb90-9935-424f-b24b-49fa1bddd52c" containerID="9dda8df661eda6bff8fb2d7679d82b44953deeea9992d1390e49cf442089f00c" exitCode=0 Jul 16 13:18:13.975623 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:13.975554 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l" Jul 16 13:18:13.976899 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:13.976877 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c0517b4-3ab6-4920-9a9f-f08bd5b0d8b5" path="/var/lib/kubelet/pods/4c0517b4-3ab6-4920-9a9f-f08bd5b0d8b5/volumes" Jul 16 13:18:13.978601 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:13.978576 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7563-139693653627504" event={"ID":"16c57579-06fe-4233-9fce-634b4f49bf95","Type":"ContainerDied","Data":"cdbdf2cf5305103f4b925226e0c20ed45b657a56907cdabe344c5881918b77d4"} Jul 16 13:18:13.978678 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:13.978610 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l" event={"ID":"42cfeb90-9935-424f-b24b-49fa1bddd52c","Type":"ContainerDied","Data":"9dda8df661eda6bff8fb2d7679d82b44953deeea9992d1390e49cf442089f00c"} Jul 16 13:18:13.978678 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:13.978628 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l" event={"ID":"42cfeb90-9935-424f-b24b-49fa1bddd52c","Type":"ContainerDied","Data":"1e15ad06aae1c9be9c2108859a2a4a2af8f6a0007e486dc581b172d0f82509a5"} Jul 16 13:18:13.978678 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:13.978648 2582 scope.go:117] "RemoveContainer" containerID="9dda8df661eda6bff8fb2d7679d82b44953deeea9992d1390e49cf442089f00c" Jul 16 13:18:13.987140 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:13.987124 2582 scope.go:117] "RemoveContainer" containerID="9dda8df661eda6bff8fb2d7679d82b44953deeea9992d1390e49cf442089f00c" Jul 16 13:18:13.987405 ip-10-0-140-130 kubenswrapper[2582]: E0716 13:18:13.987370 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dda8df661eda6bff8fb2d7679d82b44953deeea9992d1390e49cf442089f00c\": container with ID starting with 9dda8df661eda6bff8fb2d7679d82b44953deeea9992d1390e49cf442089f00c not found: ID does not exist" containerID="9dda8df661eda6bff8fb2d7679d82b44953deeea9992d1390e49cf442089f00c" Jul 16 13:18:13.987484 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:13.987418 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dda8df661eda6bff8fb2d7679d82b44953deeea9992d1390e49cf442089f00c"} err="failed to get container status \"9dda8df661eda6bff8fb2d7679d82b44953deeea9992d1390e49cf442089f00c\": rpc error: code = NotFound desc = could not find container \"9dda8df661eda6bff8fb2d7679d82b44953deeea9992d1390e49cf442089f00c\": container with ID starting with 9dda8df661eda6bff8fb2d7679d82b44953deeea9992d1390e49cf442089f00c not found: ID does not exist" Jul 16 13:18:13.993952 ip-10-0-140-130 kubenswrapper[2582]: E0716 13:18:13.993925 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7563-139693652434352\" is forbidden: User \"system:node:ip-10-0-140-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-130.ec2.internal' and this object" podUID="4c0517b4-3ab6-4920-9a9f-f08bd5b0d8b5" pod="opendatahub/test-curl-7563-139693652434352" Jul 16 13:18:14.042848 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:14.042822 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42cfeb90-9935-424f-b24b-49fa1bddd52c-home\") pod \"42cfeb90-9935-424f-b24b-49fa1bddd52c\" (UID: \"42cfeb90-9935-424f-b24b-49fa1bddd52c\") " Jul 16 13:18:14.042940 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:14.042860 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42cfeb90-9935-424f-b24b-49fa1bddd52c-model-cache\") pod \"42cfeb90-9935-424f-b24b-49fa1bddd52c\" (UID: \"42cfeb90-9935-424f-b24b-49fa1bddd52c\") " Jul 16 13:18:14.042940 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:14.042899 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42cfeb90-9935-424f-b24b-49fa1bddd52c-dshm\") pod \"42cfeb90-9935-424f-b24b-49fa1bddd52c\" (UID: \"42cfeb90-9935-424f-b24b-49fa1bddd52c\") " Jul 16 13:18:14.042940 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:14.042924 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2jv5\" (UniqueName: \"kubernetes.io/projected/42cfeb90-9935-424f-b24b-49fa1bddd52c-kube-api-access-x2jv5\") pod \"42cfeb90-9935-424f-b24b-49fa1bddd52c\" (UID: \"42cfeb90-9935-424f-b24b-49fa1bddd52c\") " Jul 16 13:18:14.043108 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:14.043009 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42cfeb90-9935-424f-b24b-49fa1bddd52c-tmp-dir\") pod \"42cfeb90-9935-424f-b24b-49fa1bddd52c\" (UID: \"42cfeb90-9935-424f-b24b-49fa1bddd52c\") " Jul 16 13:18:14.043108 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:14.043039 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42cfeb90-9935-424f-b24b-49fa1bddd52c-tls-certs\") pod \"42cfeb90-9935-424f-b24b-49fa1bddd52c\" (UID: \"42cfeb90-9935-424f-b24b-49fa1bddd52c\") " Jul 16 13:18:14.043108 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:14.043057 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42cfeb90-9935-424f-b24b-49fa1bddd52c-home" (OuterVolumeSpecName: "home") pod "42cfeb90-9935-424f-b24b-49fa1bddd52c" (UID: "42cfeb90-9935-424f-b24b-49fa1bddd52c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:18:14.043246 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:14.043144 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42cfeb90-9935-424f-b24b-49fa1bddd52c-model-cache" (OuterVolumeSpecName: "model-cache") pod "42cfeb90-9935-424f-b24b-49fa1bddd52c" (UID: "42cfeb90-9935-424f-b24b-49fa1bddd52c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:18:14.043884 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:14.043848 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42cfeb90-9935-424f-b24b-49fa1bddd52c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "42cfeb90-9935-424f-b24b-49fa1bddd52c" (UID: "42cfeb90-9935-424f-b24b-49fa1bddd52c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:18:14.046061 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:14.044001 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42cfeb90-9935-424f-b24b-49fa1bddd52c-home\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:18:14.046061 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:14.044032 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42cfeb90-9935-424f-b24b-49fa1bddd52c-model-cache\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:18:14.050380 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:14.050360 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42cfeb90-9935-424f-b24b-49fa1bddd52c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "42cfeb90-9935-424f-b24b-49fa1bddd52c" (UID: "42cfeb90-9935-424f-b24b-49fa1bddd52c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 13:18:14.050867 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:14.050848 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42cfeb90-9935-424f-b24b-49fa1bddd52c-kube-api-access-x2jv5" (OuterVolumeSpecName: "kube-api-access-x2jv5") pod "42cfeb90-9935-424f-b24b-49fa1bddd52c" (UID: "42cfeb90-9935-424f-b24b-49fa1bddd52c"). InnerVolumeSpecName "kube-api-access-x2jv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:18:14.050969 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:14.050949 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42cfeb90-9935-424f-b24b-49fa1bddd52c-dshm" (OuterVolumeSpecName: "dshm") pod "42cfeb90-9935-424f-b24b-49fa1bddd52c" (UID: "42cfeb90-9935-424f-b24b-49fa1bddd52c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:18:14.144739 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:14.144700 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42cfeb90-9935-424f-b24b-49fa1bddd52c-dshm\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:18:14.144739 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:14.144739 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2jv5\" (UniqueName: \"kubernetes.io/projected/42cfeb90-9935-424f-b24b-49fa1bddd52c-kube-api-access-x2jv5\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:18:14.144897 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:14.144755 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42cfeb90-9935-424f-b24b-49fa1bddd52c-tmp-dir\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:18:14.144897 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:14.144768 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42cfeb90-9935-424f-b24b-49fa1bddd52c-tls-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:18:14.297295 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:14.297269 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l"] Jul 16 13:18:14.302006 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:14.301981 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-8d69f8/rate-test-model-8d69f8-kserve-7869cdfb95-zdh4l"] Jul 16 13:18:15.105292 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:15.105261 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7563-139693653627504" Jul 16 13:18:15.128027 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:15.128005 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7563-139693653627504_16c57579-06fe-4233-9fce-634b4f49bf95/test-curl-7563-139693653627504/0.log" Jul 16 13:18:15.153040 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:15.153015 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7563-139693653627504"] Jul 16 13:18:15.156304 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:15.156277 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7563-139693653627504"] Jul 16 13:18:15.253751 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:15.253657 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnr5b\" (UniqueName: \"kubernetes.io/projected/16c57579-06fe-4233-9fce-634b4f49bf95-kube-api-access-cnr5b\") pod \"16c57579-06fe-4233-9fce-634b4f49bf95\" (UID: \"16c57579-06fe-4233-9fce-634b4f49bf95\") " Jul 16 13:18:15.255805 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:15.255786 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16c57579-06fe-4233-9fce-634b4f49bf95-kube-api-access-cnr5b" (OuterVolumeSpecName: "kube-api-access-cnr5b") pod "16c57579-06fe-4233-9fce-634b4f49bf95" (UID: "16c57579-06fe-4233-9fce-634b4f49bf95"). InnerVolumeSpecName "kube-api-access-cnr5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:18:15.287017 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:15.286991 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7563-139693653627504"] Jul 16 13:18:15.287284 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:15.287272 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16c57579-06fe-4233-9fce-634b4f49bf95" containerName="test-curl-7563-139693653627504" Jul 16 13:18:15.287323 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:15.287286 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="16c57579-06fe-4233-9fce-634b4f49bf95" containerName="test-curl-7563-139693653627504" Jul 16 13:18:15.287323 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:15.287303 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42cfeb90-9935-424f-b24b-49fa1bddd52c" containerName="main" Jul 16 13:18:15.287323 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:15.287309 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="42cfeb90-9935-424f-b24b-49fa1bddd52c" containerName="main" Jul 16 13:18:15.287438 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:15.287368 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="16c57579-06fe-4233-9fce-634b4f49bf95" containerName="test-curl-7563-139693653627504" Jul 16 13:18:15.287438 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:15.287375 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="42cfeb90-9935-424f-b24b-49fa1bddd52c" containerName="main" Jul 16 13:18:15.290151 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:15.290136 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7563-139693653627504" Jul 16 13:18:15.295717 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:15.295693 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7563-139693653627504"] Jul 16 13:18:15.355245 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:15.355218 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnr5b\" (UniqueName: \"kubernetes.io/projected/16c57579-06fe-4233-9fce-634b4f49bf95-kube-api-access-cnr5b\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:18:15.456253 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:15.456217 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gszd7\" (UniqueName: \"kubernetes.io/projected/0dcc2f41-b8f1-48fa-8b3e-be7672201817-kube-api-access-gszd7\") pod \"test-curl-7563-139693653627504\" (UID: \"0dcc2f41-b8f1-48fa-8b3e-be7672201817\") " pod="opendatahub/test-curl-7563-139693653627504" Jul 16 13:18:15.557621 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:15.557544 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gszd7\" (UniqueName: \"kubernetes.io/projected/0dcc2f41-b8f1-48fa-8b3e-be7672201817-kube-api-access-gszd7\") pod \"test-curl-7563-139693653627504\" (UID: \"0dcc2f41-b8f1-48fa-8b3e-be7672201817\") " pod="opendatahub/test-curl-7563-139693653627504" Jul 16 13:18:15.565648 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:15.565620 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gszd7\" (UniqueName: \"kubernetes.io/projected/0dcc2f41-b8f1-48fa-8b3e-be7672201817-kube-api-access-gszd7\") pod \"test-curl-7563-139693653627504\" (UID: \"0dcc2f41-b8f1-48fa-8b3e-be7672201817\") " pod="opendatahub/test-curl-7563-139693653627504" Jul 16 13:18:15.599554 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:15.599529 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7563-139693653627504" Jul 16 13:18:15.747446 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:15.747418 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7563-139693653627504"] Jul 16 13:18:15.748870 ip-10-0-140-130 kubenswrapper[2582]: W0716 13:18:15.748850 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0dcc2f41_b8f1_48fa_8b3e_be7672201817.slice/crio-a2e417857ed4d7ef9ec2f3a4f52fafe48c1c6b831307ab818e2ba4d0e00f887b WatchSource:0}: Error finding container a2e417857ed4d7ef9ec2f3a4f52fafe48c1c6b831307ab818e2ba4d0e00f887b: Status 404 returned error can't find the container with id a2e417857ed4d7ef9ec2f3a4f52fafe48c1c6b831307ab818e2ba4d0e00f887b Jul 16 13:18:15.977772 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:15.977741 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16c57579-06fe-4233-9fce-634b4f49bf95" path="/var/lib/kubelet/pods/16c57579-06fe-4233-9fce-634b4f49bf95/volumes" Jul 16 13:18:15.978119 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:15.978105 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42cfeb90-9935-424f-b24b-49fa1bddd52c" path="/var/lib/kubelet/pods/42cfeb90-9935-424f-b24b-49fa1bddd52c/volumes" Jul 16 13:18:15.987796 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:15.987771 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7563-139693653627504" event={"ID":"0dcc2f41-b8f1-48fa-8b3e-be7672201817","Type":"ContainerStarted","Data":"a2e417857ed4d7ef9ec2f3a4f52fafe48c1c6b831307ab818e2ba4d0e00f887b"} Jul 16 13:18:15.988922 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:15.988907 2582 scope.go:117] "RemoveContainer" containerID="cdbdf2cf5305103f4b925226e0c20ed45b657a56907cdabe344c5881918b77d4" Jul 16 13:18:15.989010 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:15.988940 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7563-139693653627504" Jul 16 13:18:16.994772 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:16.994739 2582 generic.go:356] "Generic (PLEG): container finished" podID="0dcc2f41-b8f1-48fa-8b3e-be7672201817" containerID="58bead971a126881048a8e13f9ab9164c82727f234ecb4aaea6e075cadfa847a" exitCode=0 Jul 16 13:18:16.995236 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:16.994831 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7563-139693653627504" event={"ID":"0dcc2f41-b8f1-48fa-8b3e-be7672201817","Type":"ContainerDied","Data":"58bead971a126881048a8e13f9ab9164c82727f234ecb4aaea6e075cadfa847a"} Jul 16 13:18:18.134181 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:18.134157 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7563-139693653627504" Jul 16 13:18:18.156406 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:18.156355 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7563-139693653627504_0dcc2f41-b8f1-48fa-8b3e-be7672201817/test-curl-7563-139693653627504/0.log" Jul 16 13:18:18.182840 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:18.182806 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7563-139693653627504"] Jul 16 13:18:18.186614 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:18.186591 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7563-139693653627504"] Jul 16 13:18:18.280813 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:18.280706 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gszd7\" (UniqueName: \"kubernetes.io/projected/0dcc2f41-b8f1-48fa-8b3e-be7672201817-kube-api-access-gszd7\") pod \"0dcc2f41-b8f1-48fa-8b3e-be7672201817\" (UID: \"0dcc2f41-b8f1-48fa-8b3e-be7672201817\") " Jul 16 13:18:18.282965 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:18.282929 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dcc2f41-b8f1-48fa-8b3e-be7672201817-kube-api-access-gszd7" (OuterVolumeSpecName: "kube-api-access-gszd7") pod "0dcc2f41-b8f1-48fa-8b3e-be7672201817" (UID: "0dcc2f41-b8f1-48fa-8b3e-be7672201817"). InnerVolumeSpecName "kube-api-access-gszd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:18:18.382352 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:18.382312 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gszd7\" (UniqueName: \"kubernetes.io/projected/0dcc2f41-b8f1-48fa-8b3e-be7672201817-kube-api-access-gszd7\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:18:18.848495 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:18.848460 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:18:19.007150 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:19.007125 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2e417857ed4d7ef9ec2f3a4f52fafe48c1c6b831307ab818e2ba4d0e00f887b" Jul 16 13:18:19.007314 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:19.007153 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7563-139693653627504" Jul 16 13:18:19.016819 ip-10-0-140-130 kubenswrapper[2582]: E0716 13:18:19.016790 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7563-139693653627504\" is forbidden: User \"system:node:ip-10-0-140-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-130.ec2.internal' and this object" podUID="0dcc2f41-b8f1-48fa-8b3e-be7672201817" pod="opendatahub/test-curl-7563-139693653627504" Jul 16 13:18:19.977676 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:19.977643 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dcc2f41-b8f1-48fa-8b3e-be7672201817" path="/var/lib/kubelet/pods/0dcc2f41-b8f1-48fa-8b3e-be7672201817/volumes" Jul 16 13:18:24.911884 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:24.911844 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:18:25.013381 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:25.013347 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:18:25.173500 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:25.173424 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk"] Jul 16 13:18:25.173680 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:25.173658 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" podUID="80415598-8795-489d-b5b1-a19908646f84" containerName="istio-proxy" containerID="cri-o://811e02ec77dca08542bbc329604ba21c7cbe23d84f2c1d305b0021d56165c628" gracePeriod=30 Jul 16 13:18:25.190660 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:25.190631 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:18:25.427021 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:25.426927 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:18:25.925538 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:25.925503 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:18:30.427012 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:30.426984 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:18:30.478019 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:30.477953 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wn6n\" (UniqueName: \"kubernetes.io/projected/80415598-8795-489d-b5b1-a19908646f84-kube-api-access-9wn6n\") pod \"80415598-8795-489d-b5b1-a19908646f84\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " Jul 16 13:18:30.478019 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:30.477990 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80415598-8795-489d-b5b1-a19908646f84-istio-envoy\") pod \"80415598-8795-489d-b5b1-a19908646f84\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " Jul 16 13:18:30.478019 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:30.478013 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80415598-8795-489d-b5b1-a19908646f84-workload-certs\") pod \"80415598-8795-489d-b5b1-a19908646f84\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " Jul 16 13:18:30.478254 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:30.478035 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80415598-8795-489d-b5b1-a19908646f84-istiod-ca-cert\") pod \"80415598-8795-489d-b5b1-a19908646f84\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " Jul 16 13:18:30.478254 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:30.478051 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80415598-8795-489d-b5b1-a19908646f84-credential-socket\") pod \"80415598-8795-489d-b5b1-a19908646f84\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " Jul 16 13:18:30.478254 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:30.478078 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80415598-8795-489d-b5b1-a19908646f84-istio-podinfo\") pod \"80415598-8795-489d-b5b1-a19908646f84\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " Jul 16 13:18:30.478254 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:30.478115 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80415598-8795-489d-b5b1-a19908646f84-istio-data\") pod \"80415598-8795-489d-b5b1-a19908646f84\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " Jul 16 13:18:30.478254 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:30.478136 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80415598-8795-489d-b5b1-a19908646f84-workload-socket\") pod \"80415598-8795-489d-b5b1-a19908646f84\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " Jul 16 13:18:30.478254 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:30.478162 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80415598-8795-489d-b5b1-a19908646f84-istio-token\") pod \"80415598-8795-489d-b5b1-a19908646f84\" (UID: \"80415598-8795-489d-b5b1-a19908646f84\") " Jul 16 13:18:30.478591 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:30.478317 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80415598-8795-489d-b5b1-a19908646f84-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "80415598-8795-489d-b5b1-a19908646f84" (UID: "80415598-8795-489d-b5b1-a19908646f84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:18:30.478591 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:30.478336 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80415598-8795-489d-b5b1-a19908646f84-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "80415598-8795-489d-b5b1-a19908646f84" (UID: "80415598-8795-489d-b5b1-a19908646f84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:18:30.478591 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:30.478488 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80415598-8795-489d-b5b1-a19908646f84-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "80415598-8795-489d-b5b1-a19908646f84" (UID: "80415598-8795-489d-b5b1-a19908646f84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:18:30.478591 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:30.478515 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80415598-8795-489d-b5b1-a19908646f84-workload-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:18:30.478591 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:30.478535 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80415598-8795-489d-b5b1-a19908646f84-credential-socket\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:18:30.478787 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:30.478604 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80415598-8795-489d-b5b1-a19908646f84-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "80415598-8795-489d-b5b1-a19908646f84" (UID: "80415598-8795-489d-b5b1-a19908646f84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:18:30.479365 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:30.479333 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80415598-8795-489d-b5b1-a19908646f84-istio-data" (OuterVolumeSpecName: "istio-data") pod "80415598-8795-489d-b5b1-a19908646f84" (UID: "80415598-8795-489d-b5b1-a19908646f84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:18:30.480332 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:30.480309 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80415598-8795-489d-b5b1-a19908646f84-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "80415598-8795-489d-b5b1-a19908646f84" (UID: "80415598-8795-489d-b5b1-a19908646f84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:18:30.480480 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:30.480379 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80415598-8795-489d-b5b1-a19908646f84-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "80415598-8795-489d-b5b1-a19908646f84" (UID: "80415598-8795-489d-b5b1-a19908646f84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 13:18:30.480480 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:30.480472 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80415598-8795-489d-b5b1-a19908646f84-istio-token" (OuterVolumeSpecName: "istio-token") pod "80415598-8795-489d-b5b1-a19908646f84" (UID: "80415598-8795-489d-b5b1-a19908646f84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:18:30.480738 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:30.480723 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80415598-8795-489d-b5b1-a19908646f84-kube-api-access-9wn6n" (OuterVolumeSpecName: "kube-api-access-9wn6n") pod "80415598-8795-489d-b5b1-a19908646f84" (UID: "80415598-8795-489d-b5b1-a19908646f84"). InnerVolumeSpecName "kube-api-access-9wn6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:18:30.578982 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:30.578958 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80415598-8795-489d-b5b1-a19908646f84-istiod-ca-cert\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:18:30.578982 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:30.578979 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80415598-8795-489d-b5b1-a19908646f84-istio-podinfo\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:18:30.578982 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:30.578987 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80415598-8795-489d-b5b1-a19908646f84-istio-data\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:18:30.579165 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:30.578997 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80415598-8795-489d-b5b1-a19908646f84-workload-socket\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:18:30.579165 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:30.579005 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80415598-8795-489d-b5b1-a19908646f84-istio-token\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:18:30.579165 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:30.579014 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wn6n\" (UniqueName: \"kubernetes.io/projected/80415598-8795-489d-b5b1-a19908646f84-kube-api-access-9wn6n\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:18:30.579165 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:30.579022 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80415598-8795-489d-b5b1-a19908646f84-istio-envoy\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:18:31.069177 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:31.069146 2582 generic.go:356] "Generic (PLEG): container finished" podID="80415598-8795-489d-b5b1-a19908646f84" containerID="811e02ec77dca08542bbc329604ba21c7cbe23d84f2c1d305b0021d56165c628" exitCode=0 Jul 16 13:18:31.069335 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:31.069192 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" Jul 16 13:18:31.069335 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:31.069208 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" event={"ID":"80415598-8795-489d-b5b1-a19908646f84","Type":"ContainerDied","Data":"811e02ec77dca08542bbc329604ba21c7cbe23d84f2c1d305b0021d56165c628"} Jul 16 13:18:31.069335 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:31.069251 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk" event={"ID":"80415598-8795-489d-b5b1-a19908646f84","Type":"ContainerDied","Data":"ad54ea7e062043ddb340bc77fde34c97b7535ff8b481c161ae34106c49806bcc"} Jul 16 13:18:31.069335 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:31.069272 2582 scope.go:117] "RemoveContainer" containerID="811e02ec77dca08542bbc329604ba21c7cbe23d84f2c1d305b0021d56165c628" Jul 16 13:18:31.085183 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:31.085158 2582 scope.go:117] "RemoveContainer" containerID="36bdeef81fd7c90215b64e4b1a37f874c8612f5a70fe78e43a2aaa7b808e4c7a" Jul 16 13:18:31.093518 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:31.093498 2582 scope.go:117] "RemoveContainer" containerID="811e02ec77dca08542bbc329604ba21c7cbe23d84f2c1d305b0021d56165c628" Jul 16 13:18:31.093756 ip-10-0-140-130 kubenswrapper[2582]: E0716 13:18:31.093739 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"811e02ec77dca08542bbc329604ba21c7cbe23d84f2c1d305b0021d56165c628\": container with ID starting with 811e02ec77dca08542bbc329604ba21c7cbe23d84f2c1d305b0021d56165c628 not found: ID does not exist" containerID="811e02ec77dca08542bbc329604ba21c7cbe23d84f2c1d305b0021d56165c628" Jul 16 13:18:31.093802 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:31.093762 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"811e02ec77dca08542bbc329604ba21c7cbe23d84f2c1d305b0021d56165c628"} err="failed to get container status \"811e02ec77dca08542bbc329604ba21c7cbe23d84f2c1d305b0021d56165c628\": rpc error: code = NotFound desc = could not find container \"811e02ec77dca08542bbc329604ba21c7cbe23d84f2c1d305b0021d56165c628\": container with ID starting with 811e02ec77dca08542bbc329604ba21c7cbe23d84f2c1d305b0021d56165c628 not found: ID does not exist" Jul 16 13:18:31.093802 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:31.093777 2582 scope.go:117] "RemoveContainer" containerID="36bdeef81fd7c90215b64e4b1a37f874c8612f5a70fe78e43a2aaa7b808e4c7a" Jul 16 13:18:31.093973 ip-10-0-140-130 kubenswrapper[2582]: E0716 13:18:31.093956 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36bdeef81fd7c90215b64e4b1a37f874c8612f5a70fe78e43a2aaa7b808e4c7a\": container with ID starting with 36bdeef81fd7c90215b64e4b1a37f874c8612f5a70fe78e43a2aaa7b808e4c7a not found: ID does not exist" containerID="36bdeef81fd7c90215b64e4b1a37f874c8612f5a70fe78e43a2aaa7b808e4c7a" Jul 16 13:18:31.094013 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:31.093977 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36bdeef81fd7c90215b64e4b1a37f874c8612f5a70fe78e43a2aaa7b808e4c7a"} err="failed to get container status \"36bdeef81fd7c90215b64e4b1a37f874c8612f5a70fe78e43a2aaa7b808e4c7a\": rpc error: code = NotFound desc = could not find container \"36bdeef81fd7c90215b64e4b1a37f874c8612f5a70fe78e43a2aaa7b808e4c7a\": container with ID starting with 36bdeef81fd7c90215b64e4b1a37f874c8612f5a70fe78e43a2aaa7b808e4c7a not found: ID does not exist" Jul 16 13:18:31.101527 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:31.101501 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk"] Jul 16 13:18:31.104599 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:31.104579 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-b12c92-openshift-default-599488db5f-s4wdk"] Jul 16 13:18:31.977348 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:31.977314 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80415598-8795-489d-b5b1-a19908646f84" path="/var/lib/kubelet/pods/80415598-8795-489d-b5b1-a19908646f84/volumes" Jul 16 13:18:32.331311 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:32.331173 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:18:32.397941 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:32.397875 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:18:32.600415 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:32.600321 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf"] Jul 16 13:18:32.600566 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:32.600539 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" podUID="245797fa-8338-4d44-90fd-363f6b375cf7" containerName="istio-proxy" containerID="cri-o://8767e6d745ebe43375c7c0701f055091a79f7e289b8be4fd0de06cfc058617c3" gracePeriod=30 Jul 16 13:18:32.622908 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:32.622876 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:18:32.872365 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:32.872270 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:18:33.355407 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:33.355357 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-74fd59b54b-6cnkq_d7b42a40-f9e4-41a0-920f-5fdba34878a8/authorino/0.log" Jul 16 13:18:37.846485 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:37.846463 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:18:37.937404 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:37.937357 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/245797fa-8338-4d44-90fd-363f6b375cf7-credential-socket\") pod \"245797fa-8338-4d44-90fd-363f6b375cf7\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " Jul 16 13:18:37.937548 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:37.937407 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/245797fa-8338-4d44-90fd-363f6b375cf7-istio-podinfo\") pod \"245797fa-8338-4d44-90fd-363f6b375cf7\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " Jul 16 13:18:37.937548 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:37.937438 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/245797fa-8338-4d44-90fd-363f6b375cf7-workload-certs\") pod \"245797fa-8338-4d44-90fd-363f6b375cf7\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " Jul 16 13:18:37.937548 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:37.937469 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/245797fa-8338-4d44-90fd-363f6b375cf7-istio-envoy\") pod \"245797fa-8338-4d44-90fd-363f6b375cf7\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " Jul 16 13:18:37.937548 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:37.937493 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwqjt\" (UniqueName: \"kubernetes.io/projected/245797fa-8338-4d44-90fd-363f6b375cf7-kube-api-access-vwqjt\") pod \"245797fa-8338-4d44-90fd-363f6b375cf7\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " Jul 16 13:18:37.937548 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:37.937510 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/245797fa-8338-4d44-90fd-363f6b375cf7-workload-socket\") pod \"245797fa-8338-4d44-90fd-363f6b375cf7\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " Jul 16 13:18:37.937548 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:37.937541 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/245797fa-8338-4d44-90fd-363f6b375cf7-istio-token\") pod \"245797fa-8338-4d44-90fd-363f6b375cf7\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " Jul 16 13:18:37.937857 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:37.937596 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/245797fa-8338-4d44-90fd-363f6b375cf7-istio-data\") pod \"245797fa-8338-4d44-90fd-363f6b375cf7\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " Jul 16 13:18:37.937857 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:37.937637 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/245797fa-8338-4d44-90fd-363f6b375cf7-istiod-ca-cert\") pod \"245797fa-8338-4d44-90fd-363f6b375cf7\" (UID: \"245797fa-8338-4d44-90fd-363f6b375cf7\") " Jul 16 13:18:37.937857 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:37.937646 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/245797fa-8338-4d44-90fd-363f6b375cf7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "245797fa-8338-4d44-90fd-363f6b375cf7" (UID: "245797fa-8338-4d44-90fd-363f6b375cf7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:18:37.937857 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:37.937800 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/245797fa-8338-4d44-90fd-363f6b375cf7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "245797fa-8338-4d44-90fd-363f6b375cf7" (UID: "245797fa-8338-4d44-90fd-363f6b375cf7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:18:37.937857 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:37.937856 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/245797fa-8338-4d44-90fd-363f6b375cf7-credential-socket\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:18:37.938093 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:37.937859 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/245797fa-8338-4d44-90fd-363f6b375cf7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "245797fa-8338-4d44-90fd-363f6b375cf7" (UID: "245797fa-8338-4d44-90fd-363f6b375cf7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:18:37.938165 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:37.938142 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/245797fa-8338-4d44-90fd-363f6b375cf7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "245797fa-8338-4d44-90fd-363f6b375cf7" (UID: "245797fa-8338-4d44-90fd-363f6b375cf7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 13:18:37.938540 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:37.938463 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/245797fa-8338-4d44-90fd-363f6b375cf7-istio-data" (OuterVolumeSpecName: "istio-data") pod "245797fa-8338-4d44-90fd-363f6b375cf7" (UID: "245797fa-8338-4d44-90fd-363f6b375cf7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:18:37.939952 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:37.939924 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/245797fa-8338-4d44-90fd-363f6b375cf7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "245797fa-8338-4d44-90fd-363f6b375cf7" (UID: "245797fa-8338-4d44-90fd-363f6b375cf7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:18:37.940057 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:37.939987 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/245797fa-8338-4d44-90fd-363f6b375cf7-kube-api-access-vwqjt" (OuterVolumeSpecName: "kube-api-access-vwqjt") pod "245797fa-8338-4d44-90fd-363f6b375cf7" (UID: "245797fa-8338-4d44-90fd-363f6b375cf7"). InnerVolumeSpecName "kube-api-access-vwqjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:18:37.940246 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:37.940217 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/245797fa-8338-4d44-90fd-363f6b375cf7-istio-token" (OuterVolumeSpecName: "istio-token") pod "245797fa-8338-4d44-90fd-363f6b375cf7" (UID: "245797fa-8338-4d44-90fd-363f6b375cf7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:18:37.940246 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:37.940221 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/245797fa-8338-4d44-90fd-363f6b375cf7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "245797fa-8338-4d44-90fd-363f6b375cf7" (UID: "245797fa-8338-4d44-90fd-363f6b375cf7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 13:18:38.038429 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:38.038378 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/245797fa-8338-4d44-90fd-363f6b375cf7-workload-certs\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:18:38.038429 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:38.038430 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/245797fa-8338-4d44-90fd-363f6b375cf7-istio-envoy\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:18:38.038576 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:38.038439 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwqjt\" (UniqueName: \"kubernetes.io/projected/245797fa-8338-4d44-90fd-363f6b375cf7-kube-api-access-vwqjt\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:18:38.038576 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:38.038449 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/245797fa-8338-4d44-90fd-363f6b375cf7-workload-socket\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:18:38.038576 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:38.038457 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/245797fa-8338-4d44-90fd-363f6b375cf7-istio-token\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:18:38.038576 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:38.038465 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/245797fa-8338-4d44-90fd-363f6b375cf7-istio-data\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:18:38.038576 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:38.038472 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/245797fa-8338-4d44-90fd-363f6b375cf7-istiod-ca-cert\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:18:38.038576 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:38.038483 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/245797fa-8338-4d44-90fd-363f6b375cf7-istio-podinfo\") on node \"ip-10-0-140-130.ec2.internal\" DevicePath \"\"" Jul 16 13:18:38.105795 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:38.105764 2582 generic.go:356] "Generic (PLEG): container finished" podID="245797fa-8338-4d44-90fd-363f6b375cf7" containerID="8767e6d745ebe43375c7c0701f055091a79f7e289b8be4fd0de06cfc058617c3" exitCode=0 Jul 16 13:18:38.105950 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:38.105802 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" event={"ID":"245797fa-8338-4d44-90fd-363f6b375cf7","Type":"ContainerDied","Data":"8767e6d745ebe43375c7c0701f055091a79f7e289b8be4fd0de06cfc058617c3"} Jul 16 13:18:38.105950 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:38.105812 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" Jul 16 13:18:38.105950 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:38.105844 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf" event={"ID":"245797fa-8338-4d44-90fd-363f6b375cf7","Type":"ContainerDied","Data":"e2e44eb32bad12fc21c0d43c693b3839f9144ee83e74f7efc92294f7e640ebb1"} Jul 16 13:18:38.105950 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:38.105865 2582 scope.go:117] "RemoveContainer" containerID="8767e6d745ebe43375c7c0701f055091a79f7e289b8be4fd0de06cfc058617c3" Jul 16 13:18:38.114900 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:38.114880 2582 scope.go:117] "RemoveContainer" containerID="64ade314bd6166dac88752a773b3db80ddea819384f90de67922e43baa292df4" Jul 16 13:18:38.123138 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:38.123114 2582 scope.go:117] "RemoveContainer" containerID="8767e6d745ebe43375c7c0701f055091a79f7e289b8be4fd0de06cfc058617c3" Jul 16 13:18:38.123380 ip-10-0-140-130 kubenswrapper[2582]: E0716 13:18:38.123361 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8767e6d745ebe43375c7c0701f055091a79f7e289b8be4fd0de06cfc058617c3\": container with ID starting with 8767e6d745ebe43375c7c0701f055091a79f7e289b8be4fd0de06cfc058617c3 not found: ID does not exist" containerID="8767e6d745ebe43375c7c0701f055091a79f7e289b8be4fd0de06cfc058617c3" Jul 16 13:18:38.123502 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:38.123482 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8767e6d745ebe43375c7c0701f055091a79f7e289b8be4fd0de06cfc058617c3"} err="failed to get container status \"8767e6d745ebe43375c7c0701f055091a79f7e289b8be4fd0de06cfc058617c3\": rpc error: code = NotFound desc = could not find container \"8767e6d745ebe43375c7c0701f055091a79f7e289b8be4fd0de06cfc058617c3\": container with ID starting with 8767e6d745ebe43375c7c0701f055091a79f7e289b8be4fd0de06cfc058617c3 not found: ID does not exist" Jul 16 13:18:38.123578 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:38.123504 2582 scope.go:117] "RemoveContainer" containerID="64ade314bd6166dac88752a773b3db80ddea819384f90de67922e43baa292df4" Jul 16 13:18:38.123763 ip-10-0-140-130 kubenswrapper[2582]: E0716 13:18:38.123744 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64ade314bd6166dac88752a773b3db80ddea819384f90de67922e43baa292df4\": container with ID starting with 64ade314bd6166dac88752a773b3db80ddea819384f90de67922e43baa292df4 not found: ID does not exist" containerID="64ade314bd6166dac88752a773b3db80ddea819384f90de67922e43baa292df4" Jul 16 13:18:38.123826 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:38.123770 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ade314bd6166dac88752a773b3db80ddea819384f90de67922e43baa292df4"} err="failed to get container status \"64ade314bd6166dac88752a773b3db80ddea819384f90de67922e43baa292df4\": rpc error: code = NotFound desc = could not find container \"64ade314bd6166dac88752a773b3db80ddea819384f90de67922e43baa292df4\": container with ID starting with 64ade314bd6166dac88752a773b3db80ddea819384f90de67922e43baa292df4 not found: ID does not exist" Jul 16 13:18:38.124076 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:38.124057 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf"] Jul 16 13:18:38.129786 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:38.129766 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-8d69f8-openshift-default-6b57b755-p4hvf"] Jul 16 13:18:39.770184 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:39.770149 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-f6mzx_1432236c-bfc1-4616-b394-08d76a14e655/manager/0.log" Jul 16 13:18:39.977330 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:39.977297 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="245797fa-8338-4d44-90fd-363f6b375cf7" path="/var/lib/kubelet/pods/245797fa-8338-4d44-90fd-363f6b375cf7/volumes" Jul 16 13:18:40.174581 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:40.174556 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65f8bdf75c-nsqgx_bacb49f8-640b-4c93-9958-a8a0eb492f11/copy-manifests/0.log" Jul 16 13:18:40.181944 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:40.181910 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65f8bdf75c-nsqgx_bacb49f8-640b-4c93-9958-a8a0eb492f11/manager/0.log" Jul 16 13:18:43.313965 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:43.313936 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-shared-b-8d69f8-29736795-6s4ll_c3d3cdf4-c99f-4d12-a2a9-ffc8a9b1e418/cleanup/0.log" Jul 16 13:18:44.293766 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:44.293734 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp_38c90c2c-d74a-4c6e-b6a7-f5d82958cc49/pull/0.log" Jul 16 13:18:44.300340 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:44.300320 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp_38c90c2c-d74a-4c6e-b6a7-f5d82958cc49/extract/0.log" Jul 16 13:18:44.306761 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:44.306744 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp_38c90c2c-d74a-4c6e-b6a7-f5d82958cc49/util/0.log" Jul 16 13:18:44.420325 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:44.420293 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz_aa451150-aecc-4259-baa2-b590d4bceed0/util/0.log" Jul 16 13:18:44.427287 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:44.427265 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz_aa451150-aecc-4259-baa2-b590d4bceed0/pull/0.log" Jul 16 13:18:44.433945 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:44.433925 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz_aa451150-aecc-4259-baa2-b590d4bceed0/extract/0.log" Jul 16 13:18:44.545544 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:44.545465 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss_f992ddba-708b-4d18-9d23-b92f474c885a/util/0.log" Jul 16 13:18:44.553234 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:44.553211 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss_f992ddba-708b-4d18-9d23-b92f474c885a/pull/0.log" Jul 16 13:18:44.559491 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:44.559474 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss_f992ddba-708b-4d18-9d23-b92f474c885a/extract/0.log" Jul 16 13:18:44.698122 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:44.698079 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6_e98fbe22-4c45-412d-b87a-010ea40c07c3/util/0.log" Jul 16 13:18:44.704764 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:44.704714 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6_e98fbe22-4c45-412d-b87a-010ea40c07c3/pull/0.log" Jul 16 13:18:44.712901 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:44.712850 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6_e98fbe22-4c45-412d-b87a-010ea40c07c3/extract/0.log" Jul 16 13:18:44.854823 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:44.854742 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-74fd59b54b-6cnkq_d7b42a40-f9e4-41a0-920f-5fdba34878a8/authorino/0.log" Jul 16 13:18:46.089167 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:46.089134 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4nglsn_1dca87b5-5935-411a-bd0c-ac7335a1f9e8/istio-proxy/0.log" Jul 16 13:18:46.463060 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:46.463028 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6678559b48-mz44r_e6b692ad-70c0-42bd-ae15-4caddafc8a79/kube-auth-proxy/0.log" Jul 16 13:18:47.486313 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:47.486279 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-bcx57_971c7ce3-5be4-4a9e-920c-e1620ffbb196/main/0.log" Jul 16 13:18:54.352623 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:54.352587 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-48rx4/must-gather-5wc9r"] Jul 16 13:18:54.353090 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:54.352999 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80415598-8795-489d-b5b1-a19908646f84" containerName="istio-proxy" Jul 16 13:18:54.353090 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:54.353016 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="80415598-8795-489d-b5b1-a19908646f84" containerName="istio-proxy" Jul 16 13:18:54.353090 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:54.353027 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dcc2f41-b8f1-48fa-8b3e-be7672201817" containerName="test-curl-7563-139693653627504" Jul 16 13:18:54.353090 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:54.353036 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dcc2f41-b8f1-48fa-8b3e-be7672201817" containerName="test-curl-7563-139693653627504" Jul 16 13:18:54.353090 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:54.353048 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="245797fa-8338-4d44-90fd-363f6b375cf7" containerName="istio-proxy" Jul 16 13:18:54.353090 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:54.353056 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="245797fa-8338-4d44-90fd-363f6b375cf7" containerName="istio-proxy" Jul 16 13:18:54.353090 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:54.353066 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="245797fa-8338-4d44-90fd-363f6b375cf7" containerName="istio-proxy" Jul 16 13:18:54.353090 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:54.353073 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="245797fa-8338-4d44-90fd-363f6b375cf7" containerName="istio-proxy" Jul 16 13:18:54.353542 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:54.353105 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80415598-8795-489d-b5b1-a19908646f84" containerName="istio-proxy" Jul 16 13:18:54.353542 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:54.353114 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="80415598-8795-489d-b5b1-a19908646f84" containerName="istio-proxy" Jul 16 13:18:54.353542 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:54.353195 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="80415598-8795-489d-b5b1-a19908646f84" containerName="istio-proxy" Jul 16 13:18:54.353542 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:54.353206 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="245797fa-8338-4d44-90fd-363f6b375cf7" containerName="istio-proxy" Jul 16 13:18:54.353542 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:54.353217 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dcc2f41-b8f1-48fa-8b3e-be7672201817" containerName="test-curl-7563-139693653627504" Jul 16 13:18:54.353542 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:54.353230 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="245797fa-8338-4d44-90fd-363f6b375cf7" containerName="istio-proxy" Jul 16 13:18:54.353542 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:54.353422 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="80415598-8795-489d-b5b1-a19908646f84" containerName="istio-proxy" Jul 16 13:18:54.356837 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:54.356814 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-48rx4/must-gather-5wc9r" Jul 16 13:18:54.360062 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:54.360034 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-48rx4\"/\"default-dockercfg-rvmwq\"" Jul 16 13:18:54.360167 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:54.360112 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-48rx4\"/\"openshift-service-ca.crt\"" Jul 16 13:18:54.361017 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:54.361003 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-48rx4\"/\"kube-root-ca.crt\"" Jul 16 13:18:54.386331 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:54.386305 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-48rx4/must-gather-5wc9r"] Jul 16 13:18:54.404188 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:54.404158 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:18:54.469165 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:54.469133 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1f2428aa-da83-4342-b00b-e00522b73fdb-must-gather-output\") pod \"must-gather-5wc9r\" (UID: \"1f2428aa-da83-4342-b00b-e00522b73fdb\") " pod="openshift-must-gather-48rx4/must-gather-5wc9r" Jul 16 13:18:54.469299 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:54.469184 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml8sc\" (UniqueName: \"kubernetes.io/projected/1f2428aa-da83-4342-b00b-e00522b73fdb-kube-api-access-ml8sc\") pod \"must-gather-5wc9r\" (UID: \"1f2428aa-da83-4342-b00b-e00522b73fdb\") " pod="openshift-must-gather-48rx4/must-gather-5wc9r" Jul 16 13:18:54.502669 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:54.502638 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" limit=1024 Jul 16 13:18:54.569922 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:54.569900 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1f2428aa-da83-4342-b00b-e00522b73fdb-must-gather-output\") pod \"must-gather-5wc9r\" (UID: \"1f2428aa-da83-4342-b00b-e00522b73fdb\") " pod="openshift-must-gather-48rx4/must-gather-5wc9r" Jul 16 13:18:54.570021 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:54.569943 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml8sc\" (UniqueName: \"kubernetes.io/projected/1f2428aa-da83-4342-b00b-e00522b73fdb-kube-api-access-ml8sc\") pod \"must-gather-5wc9r\" (UID: \"1f2428aa-da83-4342-b00b-e00522b73fdb\") " pod="openshift-must-gather-48rx4/must-gather-5wc9r" Jul 16 13:18:54.570231 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:54.570214 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1f2428aa-da83-4342-b00b-e00522b73fdb-must-gather-output\") pod \"must-gather-5wc9r\" (UID: \"1f2428aa-da83-4342-b00b-e00522b73fdb\") " pod="openshift-must-gather-48rx4/must-gather-5wc9r" Jul 16 13:18:54.578803 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:54.578780 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml8sc\" (UniqueName: \"kubernetes.io/projected/1f2428aa-da83-4342-b00b-e00522b73fdb-kube-api-access-ml8sc\") pod \"must-gather-5wc9r\" (UID: \"1f2428aa-da83-4342-b00b-e00522b73fdb\") " pod="openshift-must-gather-48rx4/must-gather-5wc9r" Jul 16 13:18:54.665856 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:54.665832 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-48rx4/must-gather-5wc9r" Jul 16 13:18:54.796792 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:54.796766 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-48rx4/must-gather-5wc9r"] Jul 16 13:18:54.798898 ip-10-0-140-130 kubenswrapper[2582]: W0716 13:18:54.798871 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f2428aa_da83_4342_b00b_e00522b73fdb.slice/crio-71a5c9c736c38fe350a3653923798744842659273becfcd35fdefbc9af678777 WatchSource:0}: Error finding container 71a5c9c736c38fe350a3653923798744842659273becfcd35fdefbc9af678777: Status 404 returned error can't find the container with id 71a5c9c736c38fe350a3653923798744842659273becfcd35fdefbc9af678777 Jul 16 13:18:55.193898 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:55.193866 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-48rx4/must-gather-5wc9r" event={"ID":"1f2428aa-da83-4342-b00b-e00522b73fdb","Type":"ContainerStarted","Data":"71a5c9c736c38fe350a3653923798744842659273becfcd35fdefbc9af678777"} Jul 16 13:18:56.202814 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:56.202775 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-48rx4/must-gather-5wc9r" event={"ID":"1f2428aa-da83-4342-b00b-e00522b73fdb","Type":"ContainerStarted","Data":"296f40bd866aef5b00b1778dc6684c38aff124432dc35faf21bb86655f4a9627"} Jul 16 13:18:56.203300 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:56.202830 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-48rx4/must-gather-5wc9r" event={"ID":"1f2428aa-da83-4342-b00b-e00522b73fdb","Type":"ContainerStarted","Data":"7f73126e6912f94d1605774f5e93dc14793010e29401b37c6af461253f29bbc4"} Jul 16 13:18:57.314433 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:57.314400 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zd992_b9a976d1-7bb2-47d6-b285-788a6ff40cef/global-pull-secret-syncer/0.log" Jul 16 13:18:57.365758 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:57.365729 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4ns96_a80d1676-7c47-4427-992c-234fbc36cf7b/konnectivity-agent/0.log" Jul 16 13:18:57.463912 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:18:57.463871 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-130.ec2.internal_42e2219b6620be69c895eb04c7d0d0b4/haproxy/0.log" Jul 16 13:19:01.305672 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:01.305628 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp_38c90c2c-d74a-4c6e-b6a7-f5d82958cc49/extract/0.log" Jul 16 13:19:01.329824 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:01.329793 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp_38c90c2c-d74a-4c6e-b6a7-f5d82958cc49/util/0.log" Jul 16 13:19:01.356309 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:01.356270 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bwdxp_38c90c2c-d74a-4c6e-b6a7-f5d82958cc49/pull/0.log" Jul 16 13:19:01.384736 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:01.384706 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz_aa451150-aecc-4259-baa2-b590d4bceed0/extract/0.log" Jul 16 13:19:01.408973 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:01.408949 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz_aa451150-aecc-4259-baa2-b590d4bceed0/util/0.log" Jul 16 13:19:01.432351 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:01.432319 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05fksz_aa451150-aecc-4259-baa2-b590d4bceed0/pull/0.log" Jul 16 13:19:01.465465 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:01.465352 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss_f992ddba-708b-4d18-9d23-b92f474c885a/extract/0.log" Jul 16 13:19:01.488591 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:01.488563 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss_f992ddba-708b-4d18-9d23-b92f474c885a/util/0.log" Jul 16 13:19:01.524770 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:01.524709 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73w4rss_f992ddba-708b-4d18-9d23-b92f474c885a/pull/0.log" Jul 16 13:19:01.561102 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:01.561017 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6_e98fbe22-4c45-412d-b87a-010ea40c07c3/extract/0.log" Jul 16 13:19:01.582616 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:01.582585 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6_e98fbe22-4c45-412d-b87a-010ea40c07c3/util/0.log" Jul 16 13:19:01.613720 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:01.613690 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xp2c6_e98fbe22-4c45-412d-b87a-010ea40c07c3/pull/0.log" Jul 16 13:19:02.170036 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:02.169947 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-74fd59b54b-6cnkq_d7b42a40-f9e4-41a0-920f-5fdba34878a8/authorino/0.log" Jul 16 13:19:02.191245 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:02.189103 2582 scope.go:117] "RemoveContainer" containerID="49c505394b5fab0c7a3ed81a365d935c8ee0c0de832481221ee18bb44a2183b4" Jul 16 13:19:04.716865 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:04.716827 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jwm54_0ab69574-a013-488c-aec8-71640bb7eddf/node-exporter/0.log" Jul 16 13:19:04.740420 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:04.740367 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jwm54_0ab69574-a013-488c-aec8-71640bb7eddf/kube-rbac-proxy/0.log" Jul 16 13:19:04.763036 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:04.763010 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jwm54_0ab69574-a013-488c-aec8-71640bb7eddf/init-textfile/0.log" Jul 16 13:19:07.979688 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:07.979637 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-48rx4/must-gather-5wc9r" podStartSLOduration=13.03715676 podStartE2EDuration="13.979621578s" totalImagesPullingTime="942.464818ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:18:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:18:56.220662272 +0000 UTC m=+2394.698407889" watchObservedRunningTime="2026-07-16 13:19:07.979621578 +0000 UTC m=+2406.457367197" Jul 16 13:19:07.980319 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:07.979855 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-48rx4/perf-node-gather-daemonset-dwfbf"] Jul 16 13:19:07.987114 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:07.987082 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-48rx4/perf-node-gather-daemonset-dwfbf" Jul 16 13:19:07.992539 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:07.992516 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-48rx4/perf-node-gather-daemonset-dwfbf"] Jul 16 13:19:08.118650 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:08.118604 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2078a0a-1604-49da-940d-a0377e7ab611-lib-modules\") pod \"perf-node-gather-daemonset-dwfbf\" (UID: \"f2078a0a-1604-49da-940d-a0377e7ab611\") " pod="openshift-must-gather-48rx4/perf-node-gather-daemonset-dwfbf" Jul 16 13:19:08.118650 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:08.118643 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9tvx\" (UniqueName: \"kubernetes.io/projected/f2078a0a-1604-49da-940d-a0377e7ab611-kube-api-access-x9tvx\") pod \"perf-node-gather-daemonset-dwfbf\" (UID: \"f2078a0a-1604-49da-940d-a0377e7ab611\") " pod="openshift-must-gather-48rx4/perf-node-gather-daemonset-dwfbf" Jul 16 13:19:08.118890 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:08.118691 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2078a0a-1604-49da-940d-a0377e7ab611-sys\") pod \"perf-node-gather-daemonset-dwfbf\" (UID: \"f2078a0a-1604-49da-940d-a0377e7ab611\") " pod="openshift-must-gather-48rx4/perf-node-gather-daemonset-dwfbf" Jul 16 13:19:08.118890 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:08.118727 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f2078a0a-1604-49da-940d-a0377e7ab611-proc\") pod \"perf-node-gather-daemonset-dwfbf\" (UID: \"f2078a0a-1604-49da-940d-a0377e7ab611\") " pod="openshift-must-gather-48rx4/perf-node-gather-daemonset-dwfbf" Jul 16 13:19:08.118890 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:08.118823 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f2078a0a-1604-49da-940d-a0377e7ab611-podres\") pod \"perf-node-gather-daemonset-dwfbf\" (UID: \"f2078a0a-1604-49da-940d-a0377e7ab611\") " pod="openshift-must-gather-48rx4/perf-node-gather-daemonset-dwfbf" Jul 16 13:19:08.219829 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:08.219792 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2078a0a-1604-49da-940d-a0377e7ab611-sys\") pod \"perf-node-gather-daemonset-dwfbf\" (UID: \"f2078a0a-1604-49da-940d-a0377e7ab611\") " pod="openshift-must-gather-48rx4/perf-node-gather-daemonset-dwfbf" Jul 16 13:19:08.220245 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:08.220229 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f2078a0a-1604-49da-940d-a0377e7ab611-proc\") pod \"perf-node-gather-daemonset-dwfbf\" (UID: \"f2078a0a-1604-49da-940d-a0377e7ab611\") " pod="openshift-must-gather-48rx4/perf-node-gather-daemonset-dwfbf" Jul 16 13:19:08.220368 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:08.220090 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2078a0a-1604-49da-940d-a0377e7ab611-sys\") pod \"perf-node-gather-daemonset-dwfbf\" (UID: \"f2078a0a-1604-49da-940d-a0377e7ab611\") " pod="openshift-must-gather-48rx4/perf-node-gather-daemonset-dwfbf" Jul 16 13:19:08.220493 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:08.220296 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f2078a0a-1604-49da-940d-a0377e7ab611-proc\") pod \"perf-node-gather-daemonset-dwfbf\" (UID: \"f2078a0a-1604-49da-940d-a0377e7ab611\") " pod="openshift-must-gather-48rx4/perf-node-gather-daemonset-dwfbf" Jul 16 13:19:08.220628 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:08.220614 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f2078a0a-1604-49da-940d-a0377e7ab611-podres\") pod \"perf-node-gather-daemonset-dwfbf\" (UID: \"f2078a0a-1604-49da-940d-a0377e7ab611\") " pod="openshift-must-gather-48rx4/perf-node-gather-daemonset-dwfbf" Jul 16 13:19:08.220902 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:08.220887 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2078a0a-1604-49da-940d-a0377e7ab611-lib-modules\") pod \"perf-node-gather-daemonset-dwfbf\" (UID: \"f2078a0a-1604-49da-940d-a0377e7ab611\") " pod="openshift-must-gather-48rx4/perf-node-gather-daemonset-dwfbf" Jul 16 13:19:08.221126 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:08.221112 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9tvx\" (UniqueName: \"kubernetes.io/projected/f2078a0a-1604-49da-940d-a0377e7ab611-kube-api-access-x9tvx\") pod \"perf-node-gather-daemonset-dwfbf\" (UID: \"f2078a0a-1604-49da-940d-a0377e7ab611\") " pod="openshift-must-gather-48rx4/perf-node-gather-daemonset-dwfbf" Jul 16 13:19:08.221655 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:08.221072 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2078a0a-1604-49da-940d-a0377e7ab611-lib-modules\") pod \"perf-node-gather-daemonset-dwfbf\" (UID: \"f2078a0a-1604-49da-940d-a0377e7ab611\") " pod="openshift-must-gather-48rx4/perf-node-gather-daemonset-dwfbf" Jul 16 13:19:08.221806 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:08.220797 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f2078a0a-1604-49da-940d-a0377e7ab611-podres\") pod \"perf-node-gather-daemonset-dwfbf\" (UID: \"f2078a0a-1604-49da-940d-a0377e7ab611\") " pod="openshift-must-gather-48rx4/perf-node-gather-daemonset-dwfbf" Jul 16 13:19:08.232110 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:08.232051 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9tvx\" (UniqueName: \"kubernetes.io/projected/f2078a0a-1604-49da-940d-a0377e7ab611-kube-api-access-x9tvx\") pod \"perf-node-gather-daemonset-dwfbf\" (UID: \"f2078a0a-1604-49da-940d-a0377e7ab611\") " pod="openshift-must-gather-48rx4/perf-node-gather-daemonset-dwfbf" Jul 16 13:19:08.298863 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:08.298825 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-48rx4/perf-node-gather-daemonset-dwfbf" Jul 16 13:19:08.477333 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:08.477306 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-48rx4/perf-node-gather-daemonset-dwfbf"] Jul 16 13:19:08.479450 ip-10-0-140-130 kubenswrapper[2582]: W0716 13:19:08.479413 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf2078a0a_1604_49da_940d_a0377e7ab611.slice/crio-caf78f3b1741aa2438178ad314ccb2cffedfb5e26adff85290f75e81638fcbee WatchSource:0}: Error finding container caf78f3b1741aa2438178ad314ccb2cffedfb5e26adff85290f75e81638fcbee: Status 404 returned error can't find the container with id caf78f3b1741aa2438178ad314ccb2cffedfb5e26adff85290f75e81638fcbee Jul 16 13:19:08.923214 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:08.923169 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-44js4_f2085320-522b-4e21-ba87-b92e42ed1f78/volume-data-source-validator/0.log" Jul 16 13:19:09.330533 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:09.330449 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-48rx4/perf-node-gather-daemonset-dwfbf" event={"ID":"f2078a0a-1604-49da-940d-a0377e7ab611","Type":"ContainerStarted","Data":"3c3448b4feced2d10934f13a2723af9570aa44d82bb08ddac3efae1eeed7bc35"} Jul 16 13:19:09.331320 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:09.331295 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-48rx4/perf-node-gather-daemonset-dwfbf" event={"ID":"f2078a0a-1604-49da-940d-a0377e7ab611","Type":"ContainerStarted","Data":"caf78f3b1741aa2438178ad314ccb2cffedfb5e26adff85290f75e81638fcbee"} Jul 16 13:19:09.331527 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:09.331512 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-48rx4/perf-node-gather-daemonset-dwfbf" Jul 16 13:19:09.350522 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:09.350477 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-48rx4/perf-node-gather-daemonset-dwfbf" podStartSLOduration=2.350456744 podStartE2EDuration="2.350456744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 13:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:19:09.35045908 +0000 UTC m=+2407.828204720" watchObservedRunningTime="2026-07-16 13:19:09.350456744 +0000 UTC m=+2407.828202367" Jul 16 13:19:10.012659 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:10.012622 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qsszd_8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8/dns/0.log" Jul 16 13:19:10.033106 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:10.033076 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qsszd_8aa4a2f1-20ee-4202-b1f6-452b7a6a78a8/kube-rbac-proxy/0.log" Jul 16 13:19:10.102932 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:10.102898 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lv8x6_7e332a21-d3fd-468d-a704-8a98f55f1596/dns-node-resolver/0.log" Jul 16 13:19:10.672491 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:10.672441 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-76lb5_b95cb0aa-837a-41a4-913c-7e3b15f04fdf/node-ca/0.log" Jul 16 13:19:11.663061 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:11.663027 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4nglsn_1dca87b5-5935-411a-bd0c-ac7335a1f9e8/istio-proxy/0.log" Jul 16 13:19:11.882800 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:11.882769 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6678559b48-mz44r_e6b692ad-70c0-42bd-ae15-4caddafc8a79/kube-auth-proxy/0.log" Jul 16 13:19:12.736098 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:12.736066 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r5ftr_7769852e-c7d8-4f0e-b59e-8b5dd8035f4d/serve-healthcheck-canary/0.log" Jul 16 13:19:13.194246 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:13.194213 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-2fzwz_5dd68711-4227-4f4c-b6b9-2b4eb11b7478/insights-operator/0.log" Jul 16 13:19:13.195842 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:13.195814 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-2fzwz_5dd68711-4227-4f4c-b6b9-2b4eb11b7478/insights-operator/1.log" Jul 16 13:19:13.348545 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:13.348507 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fdthr_bda475ac-da25-423d-a7ca-ebc9c29b1720/kube-rbac-proxy/0.log" Jul 16 13:19:13.369104 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:13.369079 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fdthr_bda475ac-da25-423d-a7ca-ebc9c29b1720/exporter/0.log" Jul 16 13:19:13.391894 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:13.391867 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fdthr_bda475ac-da25-423d-a7ca-ebc9c29b1720/extractor/0.log" Jul 16 13:19:15.965405 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:15.965348 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-f6mzx_1432236c-bfc1-4616-b394-08d76a14e655/manager/0.log" Jul 16 13:19:16.176701 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:16.176668 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65f8bdf75c-nsqgx_bacb49f8-640b-4c93-9958-a8a0eb492f11/manager/0.log" Jul 16 13:19:16.196205 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:16.196168 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65f8bdf75c-nsqgx_bacb49f8-640b-4c93-9958-a8a0eb492f11/copy-manifests/0.log" Jul 16 13:19:16.354630 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:16.354560 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-48rx4/perf-node-gather-daemonset-dwfbf" Jul 16 13:19:17.315913 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:17.315889 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-596f9f8f6b-4kxc8_b76ee530-c81b-43cb-a571-b9a8dcaf1923/manager/0.log" Jul 16 13:19:17.361337 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:17.361308 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4nglsn_1dca87b5-5935-411a-bd0c-ac7335a1f9e8/istio-proxy/0.log" Jul 16 13:19:17.370091 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:17.370068 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-w75pc_733e00d9-ce5a-4a74-8990-746cd68081f5/openshift-lws-operator/0.log" Jul 16 13:19:17.566278 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:17.566199 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6678559b48-mz44r_e6b692ad-70c0-42bd-ae15-4caddafc8a79/kube-auth-proxy/0.log" Jul 16 13:19:19.065131 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:19.065103 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4nglsn_1dca87b5-5935-411a-bd0c-ac7335a1f9e8/istio-proxy/0.log" Jul 16 13:19:19.273001 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:19.272927 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6678559b48-mz44r_e6b692ad-70c0-42bd-ae15-4caddafc8a79/kube-auth-proxy/0.log" Jul 16 13:19:22.949958 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:22.949855 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-h7g92_8fd6e2f8-4515-47e3-94e9-32bc0f1a9e75/kube-storage-version-migrator-operator/1.log" Jul 16 13:19:22.951492 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:22.951468 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-h7g92_8fd6e2f8-4515-47e3-94e9-32bc0f1a9e75/kube-storage-version-migrator-operator/0.log" Jul 16 13:19:23.969495 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:23.969466 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-5s6sz_431ba1c9-15f3-4a03-ab1d-8a2b983a0704/kube-multus/0.log" Jul 16 13:19:23.991930 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:23.991906 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m8pdn_04c215b4-7f84-4763-9ad8-3db823a6c2c6/kube-multus-additional-cni-plugins/0.log" Jul 16 13:19:24.011991 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:24.011967 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m8pdn_04c215b4-7f84-4763-9ad8-3db823a6c2c6/egress-router-binary-copy/0.log" Jul 16 13:19:24.031206 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:24.031186 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m8pdn_04c215b4-7f84-4763-9ad8-3db823a6c2c6/cni-plugins/0.log" Jul 16 13:19:24.048472 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:24.048451 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m8pdn_04c215b4-7f84-4763-9ad8-3db823a6c2c6/bond-cni-plugin/0.log" Jul 16 13:19:24.079835 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:24.079809 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m8pdn_04c215b4-7f84-4763-9ad8-3db823a6c2c6/routeoverride-cni/0.log" Jul 16 13:19:24.100656 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:24.100574 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m8pdn_04c215b4-7f84-4763-9ad8-3db823a6c2c6/whereabouts-cni-bincopy/0.log" Jul 16 13:19:24.120958 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:24.120936 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m8pdn_04c215b4-7f84-4763-9ad8-3db823a6c2c6/whereabouts-cni/0.log" Jul 16 13:19:24.564431 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:24.564383 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-q5cp8_fdd6ebe0-1750-4a12-a4c7-023cfadcbabc/network-metrics-daemon/0.log" Jul 16 13:19:24.584646 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:24.584625 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-q5cp8_fdd6ebe0-1750-4a12-a4c7-023cfadcbabc/kube-rbac-proxy/0.log" Jul 16 13:19:25.444748 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:25.444702 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-986g4_ffe9c110-1bbe-42a5-bd97-b2f3c70e680d/ovn-controller/0.log" Jul 16 13:19:25.460569 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:25.460543 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-986g4_ffe9c110-1bbe-42a5-bd97-b2f3c70e680d/ovn-acl-logging/0.log" Jul 16 13:19:25.482601 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:25.482581 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-986g4_ffe9c110-1bbe-42a5-bd97-b2f3c70e680d/kube-rbac-proxy-node/0.log" Jul 16 13:19:25.503438 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:25.503412 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-986g4_ffe9c110-1bbe-42a5-bd97-b2f3c70e680d/kube-rbac-proxy-ovn-metrics/0.log" Jul 16 13:19:25.519769 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:25.519747 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-986g4_ffe9c110-1bbe-42a5-bd97-b2f3c70e680d/northd/0.log" Jul 16 13:19:25.538445 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:25.538423 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-986g4_ffe9c110-1bbe-42a5-bd97-b2f3c70e680d/nbdb/0.log" Jul 16 13:19:25.559818 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:25.559798 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-986g4_ffe9c110-1bbe-42a5-bd97-b2f3c70e680d/sbdb/0.log" Jul 16 13:19:25.642076 ip-10-0-140-130 kubenswrapper[2582]: I0716 13:19:25.642044 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-986g4_ffe9c110-1bbe-42a5-bd97-b2f3c70e680d/ovnkube-controller/0.log"