Jul 06 15:15:20.176404 ip-10-0-130-108 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 06 15:15:20.176416 ip-10-0-130-108 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 06 15:15:20.176423 ip-10-0-130-108 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 06 15:15:20.176630 ip-10-0-130-108 systemd[1]: Failed to start Kubernetes Kubelet. Jul 06 15:15:30.332281 ip-10-0-130-108 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 06 15:15:30.332300 ip-10-0-130-108 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot bcdbb2b20b444cb79cbadad1dacc848b -- Jul 06 15:17:33.269844 ip-10-0-130-108 systemd[1]: Starting Kubernetes Kubelet... Jul 06 15:17:34.502557 ip-10-0-130-108 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 15:17:34.502557 ip-10-0-130-108 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 15:17:34.502557 ip-10-0-130-108 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 15:17:34.502557 ip-10-0-130-108 kubenswrapper[2570]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 15:17:34.502557 ip-10-0-130-108 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 15:17:34.505176 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.505040 2570 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 15:17:34.511967 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.511942 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 15:17:34.511967 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.511962 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 15:17:34.511967 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.511965 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 15:17:34.511967 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.511969 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 15:17:34.511967 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.511973 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 15:17:34.511967 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.511977 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 15:17:34.512199 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.511980 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 15:17:34.512199 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.511984 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 15:17:34.512199 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.511987 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 15:17:34.512199 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.511990 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 15:17:34.512199 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.511993 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 15:17:34.512199 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.511996 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 15:17:34.512199 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.511999 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 15:17:34.512199 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512004 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 15:17:34.512199 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512008 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 15:17:34.512199 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512012 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 15:17:34.512199 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512014 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 15:17:34.512199 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512017 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 15:17:34.512199 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512020 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 15:17:34.512199 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512023 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 15:17:34.512199 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512026 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 06 15:17:34.512199 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512028 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 15:17:34.512199 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512031 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 15:17:34.512199 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512034 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 15:17:34.512199 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512037 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 15:17:34.512680 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512040 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 15:17:34.512680 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512044 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 15:17:34.512680 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512047 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 15:17:34.512680 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512049 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 15:17:34.512680 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512053 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 15:17:34.512680 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512055 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 15:17:34.512680 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512058 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 15:17:34.512680 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512061 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 15:17:34.512680 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512064 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 15:17:34.512680 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512067 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 15:17:34.512680 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512070 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 15:17:34.512680 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512072 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 15:17:34.512680 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512075 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 15:17:34.512680 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512078 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 15:17:34.512680 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512081 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 15:17:34.512680 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512084 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 15:17:34.512680 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512087 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 15:17:34.512680 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512090 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 15:17:34.512680 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512093 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 15:17:34.513160 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512096 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 15:17:34.513160 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512099 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 15:17:34.513160 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512102 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 15:17:34.513160 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512105 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 15:17:34.513160 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512108 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 15:17:34.513160 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512111 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 15:17:34.513160 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512113 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 15:17:34.513160 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512116 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 15:17:34.513160 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512119 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 15:17:34.513160 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512122 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 15:17:34.513160 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512125 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 15:17:34.513160 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512127 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 15:17:34.513160 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512130 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 15:17:34.513160 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512133 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 15:17:34.513160 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512136 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 15:17:34.513160 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512138 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 15:17:34.513160 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512141 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 15:17:34.513160 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512144 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 15:17:34.513160 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512147 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 15:17:34.513160 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512150 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 15:17:34.513732 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512152 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 15:17:34.513732 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512155 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 15:17:34.513732 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512157 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 15:17:34.513732 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512160 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 15:17:34.513732 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512163 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 15:17:34.513732 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512165 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 15:17:34.513732 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512168 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 15:17:34.513732 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512171 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 15:17:34.513732 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512174 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 15:17:34.513732 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512177 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 15:17:34.513732 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512179 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 15:17:34.513732 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512182 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 15:17:34.513732 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512185 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 15:17:34.513732 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512187 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 15:17:34.513732 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512190 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 15:17:34.513732 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512192 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 15:17:34.513732 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512195 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 15:17:34.513732 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512198 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 15:17:34.513732 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512200 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 15:17:34.513732 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512203 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 15:17:34.514229 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512209 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 15:17:34.514229 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512212 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 15:17:34.514229 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512215 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 15:17:34.514229 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512218 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 15:17:34.514229 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512220 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 15:17:34.514229 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512223 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 15:17:34.514229 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512226 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 15:17:34.514229 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512229 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 15:17:34.514229 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512232 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 15:17:34.514229 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512237 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 15:17:34.514229 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512241 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 15:17:34.514229 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512244 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 15:17:34.514229 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512247 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 15:17:34.514229 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512250 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 15:17:34.514229 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.512253 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 15:17:34.514229 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513016 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 15:17:34.514229 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513023 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 15:17:34.514229 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513027 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 15:17:34.514229 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513030 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 15:17:34.514229 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513033 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 15:17:34.514754 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513036 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 15:17:34.514754 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513038 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 15:17:34.514754 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513041 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 15:17:34.514754 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513044 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 15:17:34.514754 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513046 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 15:17:34.514754 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513049 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 15:17:34.514754 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513052 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 15:17:34.514754 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513055 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 15:17:34.514754 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513057 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 15:17:34.514754 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513060 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 15:17:34.514754 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513064 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 15:17:34.514754 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513066 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 15:17:34.514754 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513070 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 15:17:34.514754 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513072 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 15:17:34.514754 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513075 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 15:17:34.514754 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513078 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 15:17:34.514754 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513081 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 15:17:34.514754 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513085 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 15:17:34.514754 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513088 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 15:17:34.515247 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513091 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 15:17:34.515247 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513093 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 15:17:34.515247 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513098 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 15:17:34.515247 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513103 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 15:17:34.515247 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513106 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 15:17:34.515247 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513109 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 15:17:34.515247 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513112 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 06 15:17:34.515247 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513115 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 15:17:34.515247 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513119 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 15:17:34.515247 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513122 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 15:17:34.515247 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513125 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 15:17:34.515247 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513128 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 15:17:34.515247 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513131 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 15:17:34.515247 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513134 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 15:17:34.515247 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513137 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 15:17:34.515247 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513140 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 15:17:34.515247 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513143 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 15:17:34.515247 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513147 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 15:17:34.515247 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513150 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 15:17:34.515247 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513152 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 15:17:34.515803 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513155 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 15:17:34.515803 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513158 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 15:17:34.515803 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513161 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 15:17:34.515803 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513164 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 15:17:34.515803 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513167 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 15:17:34.515803 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513170 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 15:17:34.515803 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513173 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 15:17:34.515803 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513176 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 15:17:34.515803 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513179 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 15:17:34.515803 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513183 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 15:17:34.515803 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513185 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 15:17:34.515803 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513188 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 15:17:34.515803 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513203 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 15:17:34.515803 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513206 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 15:17:34.515803 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513209 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 15:17:34.515803 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513212 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 15:17:34.515803 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513216 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 15:17:34.515803 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513219 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 15:17:34.515803 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513222 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 15:17:34.515803 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513224 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 15:17:34.516311 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513227 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 15:17:34.516311 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513230 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 15:17:34.516311 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513233 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 15:17:34.516311 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513236 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 15:17:34.516311 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513239 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 15:17:34.516311 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513242 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 15:17:34.516311 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513245 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 15:17:34.516311 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513248 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 15:17:34.516311 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513251 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 15:17:34.516311 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513253 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 15:17:34.516311 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513256 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 15:17:34.516311 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513259 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 15:17:34.516311 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513262 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 15:17:34.516311 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513264 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 15:17:34.516311 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513270 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 15:17:34.516311 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513274 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 15:17:34.516311 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513276 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 15:17:34.516311 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513279 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 15:17:34.516311 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513282 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 15:17:34.516311 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513286 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 15:17:34.516855 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513289 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 15:17:34.516855 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513292 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 15:17:34.516855 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513295 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 15:17:34.516855 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513298 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 15:17:34.516855 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513302 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 15:17:34.516855 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513306 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 15:17:34.516855 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513309 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 15:17:34.516855 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513311 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 15:17:34.516855 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513314 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 15:17:34.516855 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513317 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 15:17:34.516855 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513320 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 15:17:34.516855 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513323 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 15:17:34.516855 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513325 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 15:17:34.516855 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513328 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 15:17:34.516855 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513330 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 15:17:34.516855 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513741 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 15:17:34.516855 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513747 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 15:17:34.516855 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513750 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 15:17:34.516855 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513753 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 15:17:34.517329 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513756 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 15:17:34.517329 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513759 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 15:17:34.517329 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513762 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 15:17:34.517329 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513765 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 15:17:34.517329 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513768 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 15:17:34.517329 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513770 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 15:17:34.517329 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513773 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 15:17:34.517329 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513776 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 15:17:34.517329 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513779 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 15:17:34.517329 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513782 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 15:17:34.517329 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513785 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 15:17:34.517329 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513788 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 15:17:34.517329 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513792 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 15:17:34.517329 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513796 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 15:17:34.517329 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513798 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 15:17:34.517329 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513801 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 15:17:34.517329 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513804 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 15:17:34.517329 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513807 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 15:17:34.517329 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513810 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 15:17:34.517849 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513813 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 15:17:34.517849 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513816 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 15:17:34.517849 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513820 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 15:17:34.517849 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513823 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 15:17:34.517849 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513827 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 15:17:34.517849 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513829 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 15:17:34.517849 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513832 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 15:17:34.517849 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513835 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 15:17:34.517849 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513837 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 15:17:34.517849 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513840 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 15:17:34.517849 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513843 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 15:17:34.517849 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513845 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 15:17:34.517849 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513848 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 15:17:34.517849 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513850 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 15:17:34.517849 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513853 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 15:17:34.517849 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513855 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 15:17:34.517849 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513858 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 15:17:34.517849 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513860 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 15:17:34.517849 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513863 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 15:17:34.517849 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513866 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 15:17:34.518379 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513869 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 15:17:34.518379 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513872 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 15:17:34.518379 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513874 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 15:17:34.518379 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513877 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 15:17:34.518379 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513880 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 15:17:34.518379 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513883 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 15:17:34.518379 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513886 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 15:17:34.518379 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513889 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 15:17:34.518379 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513891 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 15:17:34.518379 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513894 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 15:17:34.518379 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513897 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 15:17:34.518379 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513899 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 15:17:34.518379 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513903 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 15:17:34.518379 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513905 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 15:17:34.518379 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513908 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 15:17:34.518379 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513911 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 15:17:34.518379 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513913 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 15:17:34.518379 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513916 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 15:17:34.518379 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513918 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 15:17:34.518379 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513921 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 15:17:34.518898 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513924 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 15:17:34.518898 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513926 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 15:17:34.518898 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513929 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 15:17:34.518898 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513931 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 15:17:34.518898 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513934 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 15:17:34.518898 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513936 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 15:17:34.518898 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513939 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 15:17:34.518898 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513942 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 15:17:34.518898 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513944 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 15:17:34.518898 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513950 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 06 15:17:34.518898 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513952 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 15:17:34.518898 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513955 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 15:17:34.518898 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513958 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 15:17:34.518898 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513963 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 15:17:34.518898 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513967 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 15:17:34.518898 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513970 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 15:17:34.518898 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513974 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 15:17:34.518898 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513977 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 15:17:34.518898 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513980 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 15:17:34.519371 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513984 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 15:17:34.519371 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513987 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 15:17:34.519371 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513989 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 15:17:34.519371 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513992 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 15:17:34.519371 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513995 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 15:17:34.519371 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.513997 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 15:17:34.519371 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514000 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 15:17:34.519371 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514002 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 15:17:34.519371 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514005 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 15:17:34.519371 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514008 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 15:17:34.519371 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514010 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 15:17:34.519371 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514013 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 15:17:34.519371 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514015 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 15:17:34.519371 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514018 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 15:17:34.519371 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514022 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 15:17:34.519371 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514025 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 15:17:34.519371 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514029 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 15:17:34.519371 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514103 2570 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 15:17:34.519371 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514110 2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 15:17:34.519371 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514117 2570 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 15:17:34.519371 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514122 2570 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 15:17:34.519926 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514126 2570 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 15:17:34.519926 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514131 2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 15:17:34.519926 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514135 2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 15:17:34.519926 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514140 2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 15:17:34.519926 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514144 2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 15:17:34.519926 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514147 2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 15:17:34.519926 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514150 2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 15:17:34.519926 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514154 2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 15:17:34.519926 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514158 2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 15:17:34.519926 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514161 2570 flags.go:64] FLAG: --cgroup-root="" Jul 06 15:17:34.519926 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514164 2570 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 15:17:34.519926 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514167 2570 flags.go:64] FLAG: --client-ca-file="" Jul 06 15:17:34.519926 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514171 2570 flags.go:64] FLAG: --cloud-provider="external" Jul 06 15:17:34.519926 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514174 2570 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 15:17:34.519926 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514179 2570 flags.go:64] FLAG: --cluster-domain="" Jul 06 15:17:34.519926 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514182 2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 15:17:34.519926 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514185 2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 15:17:34.519926 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514189 2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 15:17:34.519926 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514192 2570 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 15:17:34.519926 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514196 2570 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 15:17:34.519926 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514199 2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 15:17:34.519926 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514202 2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 15:17:34.519926 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514206 2570 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 15:17:34.520512 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514209 2570 flags.go:64] FLAG: --contention-profiling="false" Jul 06 15:17:34.520512 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514212 2570 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 15:17:34.520512 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514215 2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 15:17:34.520512 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514218 2570 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 15:17:34.520512 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514221 2570 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 15:17:34.520512 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514225 2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 15:17:34.520512 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514229 2570 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 15:17:34.520512 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514232 2570 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 15:17:34.520512 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514236 2570 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 15:17:34.520512 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514239 2570 flags.go:64] FLAG: --enable-server="true" Jul 06 15:17:34.520512 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514243 2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 15:17:34.520512 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514248 2570 flags.go:64] FLAG: --event-burst="100" Jul 06 15:17:34.520512 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514252 2570 flags.go:64] FLAG: --event-qps="50" Jul 06 15:17:34.520512 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514255 2570 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 15:17:34.520512 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514258 2570 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 15:17:34.520512 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514261 2570 flags.go:64] FLAG: --eviction-hard="" Jul 06 15:17:34.520512 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514264 2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 15:17:34.520512 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514268 2570 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 15:17:34.520512 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514271 2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 15:17:34.520512 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514274 2570 flags.go:64] FLAG: --eviction-soft="" Jul 06 15:17:34.520512 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514278 2570 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 15:17:34.520512 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514280 2570 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 15:17:34.520512 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514285 2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 15:17:34.520512 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514288 2570 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 15:17:34.520512 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514291 2570 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 15:17:34.521188 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514294 2570 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 15:17:34.521188 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514297 2570 flags.go:64] FLAG: --feature-gates="" Jul 06 15:17:34.521188 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514301 2570 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 15:17:34.521188 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514304 2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 15:17:34.521188 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514308 2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 15:17:34.521188 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514326 2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 15:17:34.521188 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514330 2570 flags.go:64] FLAG: --healthz-port="10248" Jul 06 15:17:34.521188 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514334 2570 flags.go:64] FLAG: --help="false" Jul 06 15:17:34.521188 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514337 2570 flags.go:64] FLAG: --hostname-override="ip-10-0-130-108.ec2.internal" Jul 06 15:17:34.521188 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514340 2570 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 15:17:34.521188 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514344 2570 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 15:17:34.521188 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514347 2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 15:17:34.521188 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514351 2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 15:17:34.521188 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514354 2570 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 15:17:34.521188 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514357 2570 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 15:17:34.521188 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514361 2570 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 15:17:34.521188 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514364 2570 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 15:17:34.521188 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514368 2570 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 15:17:34.521188 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514371 2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 15:17:34.521188 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514374 2570 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 15:17:34.521188 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514377 2570 flags.go:64] FLAG: --kube-reserved="" Jul 06 15:17:34.521188 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514380 2570 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 15:17:34.521188 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514383 2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 15:17:34.521188 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514386 2570 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 15:17:34.521911 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514389 2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 15:17:34.521911 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514392 2570 flags.go:64] FLAG: --lock-file="" Jul 06 15:17:34.521911 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514395 2570 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 15:17:34.521911 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514398 2570 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 15:17:34.521911 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514401 2570 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 15:17:34.521911 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514406 2570 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 15:17:34.521911 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514410 2570 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 15:17:34.521911 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514413 2570 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 15:17:34.521911 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514416 2570 flags.go:64] FLAG: --logging-format="text" Jul 06 15:17:34.521911 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514419 2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 15:17:34.521911 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514423 2570 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 15:17:34.521911 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514425 2570 flags.go:64] FLAG: --manifest-url="" Jul 06 15:17:34.521911 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514428 2570 flags.go:64] FLAG: --manifest-url-header="" Jul 06 15:17:34.521911 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514433 2570 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 15:17:34.521911 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514436 2570 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 15:17:34.521911 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514440 2570 flags.go:64] FLAG: --max-pods="110" Jul 06 15:17:34.521911 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514443 2570 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 15:17:34.521911 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514446 2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 15:17:34.521911 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514449 2570 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 15:17:34.521911 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514452 2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 15:17:34.521911 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514455 2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 15:17:34.521911 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514458 2570 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 15:17:34.521911 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514461 2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 15:17:34.521911 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514468 2570 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 15:17:34.521911 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514471 2570 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 15:17:34.522595 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514475 2570 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 15:17:34.522595 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514478 2570 flags.go:64] FLAG: --pod-cidr="" Jul 06 15:17:34.522595 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514481 2570 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 15:17:34.522595 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514487 2570 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 15:17:34.522595 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514490 2570 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 15:17:34.522595 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514494 2570 flags.go:64] FLAG: --pods-per-core="0" Jul 06 15:17:34.522595 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514497 2570 flags.go:64] FLAG: --port="10250" Jul 06 15:17:34.522595 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514500 2570 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 15:17:34.522595 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514505 2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0eaafd0b64004e789" Jul 06 15:17:34.522595 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514508 2570 flags.go:64] FLAG: --qos-reserved="" Jul 06 15:17:34.522595 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514511 2570 flags.go:64] FLAG: --read-only-port="10255" Jul 06 15:17:34.522595 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514514 2570 flags.go:64] FLAG: --register-node="true" Jul 06 15:17:34.522595 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514518 2570 flags.go:64] FLAG: --register-with-taints="" Jul 06 15:17:34.522595 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514521 2570 flags.go:64] FLAG: --registry-burst="10" Jul 06 15:17:34.522595 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514525 2570 flags.go:64] FLAG: --registry-qps="5" Jul 06 15:17:34.522595 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514529 2570 flags.go:64] FLAG: --reserved-cpus="" Jul 06 15:17:34.522595 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514531 2570 flags.go:64] FLAG: --reserved-memory="" Jul 06 15:17:34.522595 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514535 2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 15:17:34.522595 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514539 2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 15:17:34.522595 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514542 2570 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 15:17:34.522595 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514545 2570 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 15:17:34.522595 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514548 2570 flags.go:64] FLAG: --runonce="false" Jul 06 15:17:34.522595 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514551 2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 15:17:34.522595 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514554 2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 15:17:34.522595 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514558 2570 flags.go:64] FLAG: --seccomp-default="false" Jul 06 15:17:34.523243 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514561 2570 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 15:17:34.523243 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514564 2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 15:17:34.523243 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514567 2570 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 15:17:34.523243 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514571 2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 15:17:34.523243 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514587 2570 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 15:17:34.523243 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514591 2570 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 15:17:34.523243 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514594 2570 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 15:17:34.523243 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514597 2570 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 15:17:34.523243 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514600 2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 15:17:34.523243 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514604 2570 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 15:17:34.523243 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514607 2570 flags.go:64] FLAG: --system-cgroups="" Jul 06 15:17:34.523243 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514610 2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 15:17:34.523243 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514616 2570 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 15:17:34.523243 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514619 2570 flags.go:64] FLAG: --tls-cert-file="" Jul 06 15:17:34.523243 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514623 2570 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 15:17:34.523243 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514630 2570 flags.go:64] FLAG: --tls-min-version="" Jul 06 15:17:34.523243 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514633 2570 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 15:17:34.523243 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514636 2570 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 15:17:34.523243 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514639 2570 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 15:17:34.523243 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514642 2570 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 15:17:34.523243 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514645 2570 flags.go:64] FLAG: --v="2" Jul 06 15:17:34.523243 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514650 2570 flags.go:64] FLAG: --version="false" Jul 06 15:17:34.523243 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514655 2570 flags.go:64] FLAG: --vmodule="" Jul 06 15:17:34.523243 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514659 2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 15:17:34.523243 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.514662 2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 15:17:34.523897 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514788 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 15:17:34.523897 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514792 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 15:17:34.523897 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514796 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 15:17:34.523897 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514799 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 15:17:34.523897 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514803 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 15:17:34.523897 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514806 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 15:17:34.523897 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514809 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 15:17:34.523897 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514812 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 15:17:34.523897 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514814 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 15:17:34.523897 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514817 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 15:17:34.523897 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514820 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 15:17:34.523897 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514823 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 15:17:34.523897 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514825 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 15:17:34.523897 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514828 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 15:17:34.523897 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514831 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 15:17:34.523897 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514834 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 06 15:17:34.523897 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514837 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 15:17:34.523897 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514840 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 15:17:34.523897 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514844 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 15:17:34.523897 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514848 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 15:17:34.524485 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514851 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 15:17:34.524485 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514854 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 15:17:34.524485 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514858 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 15:17:34.524485 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514861 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 15:17:34.524485 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514864 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 15:17:34.524485 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514867 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 15:17:34.524485 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514870 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 15:17:34.524485 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514873 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 15:17:34.524485 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514876 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 15:17:34.524485 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514879 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 15:17:34.524485 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514882 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 15:17:34.524485 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514885 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 15:17:34.524485 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514889 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 15:17:34.524485 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514892 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 15:17:34.524485 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514895 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 15:17:34.524485 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514898 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 15:17:34.524485 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514901 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 15:17:34.524485 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514903 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 15:17:34.524485 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514906 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 15:17:34.524485 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514909 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 15:17:34.525046 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514911 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 15:17:34.525046 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514914 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 15:17:34.525046 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514916 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 15:17:34.525046 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514919 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 15:17:34.525046 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514922 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 15:17:34.525046 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514924 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 15:17:34.525046 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514927 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 15:17:34.525046 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514930 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 15:17:34.525046 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514932 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 15:17:34.525046 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514935 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 15:17:34.525046 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514937 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 15:17:34.525046 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514940 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 15:17:34.525046 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514943 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 15:17:34.525046 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514945 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 15:17:34.525046 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514950 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 15:17:34.525046 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514952 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 15:17:34.525046 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514955 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 15:17:34.525046 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514958 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 15:17:34.525046 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514961 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 15:17:34.525046 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514963 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 15:17:34.525565 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514966 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 15:17:34.525565 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514969 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 15:17:34.525565 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514972 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 15:17:34.525565 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514975 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 15:17:34.525565 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514979 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 15:17:34.525565 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514982 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 15:17:34.525565 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514984 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 15:17:34.525565 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514987 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 15:17:34.525565 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514990 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 15:17:34.525565 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514993 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 15:17:34.525565 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514995 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 15:17:34.525565 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.514998 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 15:17:34.525565 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.515001 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 15:17:34.525565 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.515003 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 15:17:34.525565 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.515006 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 15:17:34.525565 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.515009 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 15:17:34.525565 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.515011 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 15:17:34.525565 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.515014 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 15:17:34.525565 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.515017 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 15:17:34.525565 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.515020 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 15:17:34.526106 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.515022 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 15:17:34.526106 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.515025 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 15:17:34.526106 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.515028 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 15:17:34.526106 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.515030 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 15:17:34.526106 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.515034 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 15:17:34.526106 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.515038 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 15:17:34.526106 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.515042 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 15:17:34.526106 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.515045 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 15:17:34.526106 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.515048 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 15:17:34.526106 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.515053 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 15:17:34.526106 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.515057 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 15:17:34.526106 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.515061 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 15:17:34.526106 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.515064 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 15:17:34.526106 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.515067 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 15:17:34.526106 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.515070 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 15:17:34.526106 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.515073 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 15:17:34.526106 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.515078 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 15:17:34.526106 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.515080 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 15:17:34.526106 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.515083 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 15:17:34.526624 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.516008 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 15:17:34.526624 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.523302 2570 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 15:17:34.526624 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.523319 2570 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 15:17:34.526624 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.524188 2570 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 15:17:34.526624 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.524200 2570 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 15:17:34.526624 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524262 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 15:17:34.526624 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524268 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 15:17:34.526624 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524272 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 15:17:34.526624 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524275 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 15:17:34.526624 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524278 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 15:17:34.526624 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524282 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 15:17:34.526624 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524285 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 15:17:34.526624 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524289 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 15:17:34.526624 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524292 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 15:17:34.527005 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524295 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 15:17:34.527005 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524297 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 15:17:34.527005 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524300 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 15:17:34.527005 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524303 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 15:17:34.527005 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524306 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 15:17:34.527005 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524308 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 15:17:34.527005 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524312 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 15:17:34.527005 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524315 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 15:17:34.527005 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524317 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 15:17:34.527005 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524320 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 15:17:34.527005 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524323 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 15:17:34.527005 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524326 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 15:17:34.527005 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524329 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 15:17:34.527005 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524332 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 15:17:34.527005 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524334 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 15:17:34.527005 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524337 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 15:17:34.527005 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524340 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 15:17:34.527005 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524358 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 15:17:34.527005 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524361 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 15:17:34.527005 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524365 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 15:17:34.527533 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524371 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 15:17:34.527533 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524374 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 15:17:34.527533 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524377 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 15:17:34.527533 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524380 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 15:17:34.527533 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524383 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 15:17:34.527533 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524386 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 15:17:34.527533 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524390 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 15:17:34.527533 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524394 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 15:17:34.527533 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524398 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 06 15:17:34.527533 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524401 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 15:17:34.527533 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524404 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 15:17:34.527533 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524407 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 15:17:34.527533 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524409 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 15:17:34.527533 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524412 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 15:17:34.527533 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524415 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 15:17:34.527533 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524418 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 15:17:34.527533 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524422 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 15:17:34.527533 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524426 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 15:17:34.527533 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524429 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 15:17:34.528073 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524432 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 15:17:34.528073 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524434 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 15:17:34.528073 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524437 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 15:17:34.528073 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524440 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 15:17:34.528073 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524444 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 15:17:34.528073 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524448 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 15:17:34.528073 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524452 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 15:17:34.528073 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524455 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 15:17:34.528073 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524458 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 15:17:34.528073 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524460 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 15:17:34.528073 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524463 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 15:17:34.528073 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524466 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 15:17:34.528073 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524469 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 15:17:34.528073 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524473 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 15:17:34.528073 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524476 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 15:17:34.528073 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524479 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 15:17:34.528073 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524481 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 15:17:34.528073 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524484 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 15:17:34.528073 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524487 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 15:17:34.528560 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524490 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 15:17:34.528560 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524493 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 15:17:34.528560 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524495 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 15:17:34.528560 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524499 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 15:17:34.528560 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524502 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 15:17:34.528560 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524505 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 15:17:34.528560 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524507 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 15:17:34.528560 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524510 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 15:17:34.528560 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524513 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 15:17:34.528560 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524515 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 15:17:34.528560 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524518 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 15:17:34.528560 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524521 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 15:17:34.528560 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524524 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 15:17:34.528560 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524526 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 15:17:34.528560 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524529 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 15:17:34.528560 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524532 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 15:17:34.528560 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524535 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 15:17:34.528560 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524537 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 15:17:34.528560 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524540 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 15:17:34.528560 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524543 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 15:17:34.529083 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524545 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 15:17:34.529083 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524548 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 15:17:34.529083 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524551 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 15:17:34.529083 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524554 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 15:17:34.529083 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524556 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 15:17:34.529083 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524559 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 15:17:34.529083 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524562 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 15:17:34.529083 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524565 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 15:17:34.529083 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524568 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 15:17:34.529083 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524571 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 15:17:34.529083 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524585 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 15:17:34.529083 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524588 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 15:17:34.529083 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.524594 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 15:17:34.529083 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524720 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 15:17:34.529083 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524727 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 15:17:34.529476 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524730 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 15:17:34.529476 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524733 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 15:17:34.529476 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524736 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 15:17:34.529476 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524739 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 15:17:34.529476 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524742 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 15:17:34.529476 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524745 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 15:17:34.529476 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524747 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 15:17:34.529476 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524751 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 15:17:34.529476 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524754 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 15:17:34.529476 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524756 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 15:17:34.529476 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524759 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 15:17:34.529476 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524762 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 15:17:34.529476 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524764 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 15:17:34.529476 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524767 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 15:17:34.529476 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524770 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 15:17:34.529476 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524772 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 15:17:34.529476 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524775 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 15:17:34.529476 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524778 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 15:17:34.529476 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524781 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 15:17:34.529476 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524784 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 15:17:34.530052 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524786 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 15:17:34.530052 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524789 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 15:17:34.530052 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524792 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 15:17:34.530052 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524795 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 15:17:34.530052 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524797 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 15:17:34.530052 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524800 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 15:17:34.530052 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524803 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 15:17:34.530052 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524806 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 15:17:34.530052 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524809 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 15:17:34.530052 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524811 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 15:17:34.530052 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524814 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 15:17:34.530052 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524818 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 15:17:34.530052 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524820 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 15:17:34.530052 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524823 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 15:17:34.530052 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524825 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 15:17:34.530052 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524828 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 15:17:34.530052 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524831 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 15:17:34.530052 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524844 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 15:17:34.530052 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524847 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 15:17:34.530052 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524850 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 15:17:34.530598 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524853 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 15:17:34.530598 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524856 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 15:17:34.530598 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524859 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 15:17:34.530598 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524862 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 15:17:34.530598 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524865 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 15:17:34.530598 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524868 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 15:17:34.530598 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524870 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 15:17:34.530598 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524873 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 15:17:34.530598 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524876 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 15:17:34.530598 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524878 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 15:17:34.530598 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524881 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 15:17:34.530598 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524884 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 15:17:34.530598 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524887 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 15:17:34.530598 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524889 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 15:17:34.530598 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524892 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 15:17:34.530598 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524895 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 15:17:34.530598 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524898 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 15:17:34.530598 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524900 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 15:17:34.530598 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524903 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 15:17:34.530598 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524906 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 15:17:34.531119 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524911 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 15:17:34.531119 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524914 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 15:17:34.531119 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524918 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 15:17:34.531119 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524920 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 15:17:34.531119 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524923 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 15:17:34.531119 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524926 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 15:17:34.531119 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524930 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 15:17:34.531119 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524934 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 15:17:34.531119 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524937 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 15:17:34.531119 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524940 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 15:17:34.531119 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524943 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 15:17:34.531119 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524946 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 15:17:34.531119 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524949 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 06 15:17:34.531119 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524952 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 15:17:34.531119 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524955 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 15:17:34.531119 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524958 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 15:17:34.531119 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524961 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 15:17:34.531119 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524963 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 15:17:34.531119 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524966 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 15:17:34.531613 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524969 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 15:17:34.531613 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524972 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 15:17:34.531613 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524975 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 15:17:34.531613 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524977 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 15:17:34.531613 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524981 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 15:17:34.531613 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524985 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 15:17:34.531613 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524987 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 15:17:34.531613 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524990 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 15:17:34.531613 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524993 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 15:17:34.531613 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524996 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 15:17:34.531613 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.524998 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 15:17:34.531613 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.525001 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 15:17:34.531613 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.525004 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 15:17:34.531613 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.525007 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 15:17:34.531613 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.525010 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 15:17:34.531613 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.525013 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 15:17:34.531613 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.525015 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 15:17:34.531613 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:34.525018 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 15:17:34.532114 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.525024 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 15:17:34.532114 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.525137 2570 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 15:17:34.532114 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.527502 2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 15:17:34.532114 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.529148 2570 server.go:1019] "Starting client certificate rotation" Jul 06 15:17:34.532114 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.529248 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 15:17:34.532114 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.529290 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 15:17:34.532114 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.531814 2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 15:17:34.537169 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.537142 2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 15:17:34.546460 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.546438 2570 log.go:25] "Validated CRI v1 runtime API" Jul 06 15:17:34.549187 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.549170 2570 log.go:25] "Validated CRI v1 image API" Jul 06 15:17:34.550104 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.550087 2570 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 15:17:34.555083 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.555061 2570 fs.go:135] Filesystem UUIDs: map[397a8c8d-383c-4abe-b3cb-d9b70ce33a70:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 82f56b9d-9278-4a0e-8b5a-3ac9889d9975:/dev/nvme0n1p3] Jul 06 15:17:34.555151 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.555085 2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 15:17:34.559883 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.559765 2570 manager.go:216] Machine: {Timestamp:2026-07-06 15:17:34.559204372 +0000 UTC m=+1.172952902 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106118 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bcfc7835c8bcf9f8c3185bfe622f6 SystemUUID:ec2bcfc7-835c-8bcf-9f8c-3185bfe622f6 BootID:bcdbb2b2-0b44-4cb7-9cba-dad1dacc848b Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a7:93:7e:7e:c9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a7:93:7e:7e:c9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:24:21:44:c7:77 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 15:17:34.559883 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.559876 2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 15:17:34.560000 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.559958 2570 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 15:17:34.561467 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.561451 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 15:17:34.561943 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.561912 2570 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 15:17:34.562092 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.561946 2570 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-108.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 15:17:34.562705 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.562695 2570 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 15:17:34.562744 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.562708 2570 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 15:17:34.562744 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.562717 2570 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 15:17:34.562744 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.562729 2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 15:17:34.565285 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.565273 2570 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 15:17:34.568436 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.568424 2570 state_mem.go:36] "Initialized new in-memory state store" Jul 06 15:17:34.568559 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.568549 2570 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 15:17:34.568616 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.568593 2570 kubelet.go:486] "Attempting to sync node with API server" Jul 06 15:17:34.568616 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.568608 2570 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 15:17:34.568686 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.568621 2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 15:17:34.568686 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.568632 2570 kubelet.go:398] "Adding apiserver pod source" Jul 06 15:17:34.568686 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.568642 2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 15:17:34.569069 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.569057 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 15:17:34.569107 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.569077 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 15:17:34.571402 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.571385 2570 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 15:17:34.576053 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.576032 2570 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 15:17:34.576145 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.576068 2570 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 15:17:34.577270 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.577257 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 15:17:34.577308 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.577275 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 15:17:34.577308 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.577282 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 15:17:34.577308 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.577289 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 15:17:34.577308 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.577298 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 15:17:34.577308 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.577304 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 15:17:34.577449 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.577313 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 15:17:34.577449 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.577320 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 15:17:34.577449 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.577327 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 15:17:34.577449 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.577333 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 15:17:34.577449 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.577342 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 15:17:34.577449 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.577347 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 15:17:34.578150 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.578138 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 15:17:34.578180 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.578151 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 15:17:34.578883 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:34.578858 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-108.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 15:17:34.578955 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:34.578861 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 15:17:34.580745 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.580729 2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-108.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 15:17:34.581710 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.581697 2570 server.go:1268] "Started kubelet" Jul 06 15:17:34.581853 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.581781 2570 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 15:17:34.581853 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.581807 2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 15:17:34.581999 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.581884 2570 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 15:17:34.582683 ip-10-0-130-108 systemd[1]: Started Kubernetes Kubelet. Jul 06 15:17:34.584100 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.584085 2570 server.go:310] "Adding debug handlers to kubelet server" Jul 06 15:17:34.588003 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.587984 2570 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 15:17:34.588218 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:34.588200 2570 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 15:17:34.593450 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:34.592538 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-108.ec2.internal.18bfbc55e6289ef8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-108.ec2.internal,UID:ip-10-0-130-108.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-108.ec2.internal,},FirstTimestamp:2026-07-06 15:17:34.581669624 +0000 UTC m=+1.195418154,LastTimestamp:2026-07-06 15:17:34.581669624 +0000 UTC m=+1.195418154,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-108.ec2.internal,}" Jul 06 15:17:34.594301 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.594280 2570 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 15:17:34.594301 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.594296 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 15:17:34.594469 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:34.594418 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-108.ec2.internal\" not found" Jul 06 15:17:34.595254 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.595230 2570 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 15:17:34.595254 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.595256 2570 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 15:17:34.595411 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.595365 2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 15:17:34.595411 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.595393 2570 factory.go:55] Registering systemd factory Jul 06 15:17:34.595411 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.595397 2570 reconstruct.go:97] "Volume reconstruction finished" Jul 06 15:17:34.595411 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.595403 2570 factory.go:223] Registration of the systemd container factory successfully Jul 06 15:17:34.595411 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.595405 2570 reconciler.go:29] "Reconciler: start to sync state" Jul 06 15:17:34.595740 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.595720 2570 factory.go:153] Registering CRI-O factory Jul 06 15:17:34.595852 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.595840 2570 factory.go:223] Registration of the crio container factory successfully Jul 06 15:17:34.595953 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.595944 2570 factory.go:103] Registering Raw factory Jul 06 15:17:34.596054 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.596038 2570 manager.go:1196] Started watching for new ooms in manager Jul 06 15:17:34.596144 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.595904 2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 15:17:34.596873 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.596851 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c5gqw" Jul 06 15:17:34.597317 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.597300 2570 manager.go:349] Starting recovery of all containers Jul 06 15:17:34.598954 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:34.598930 2570 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-108.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 15:17:34.599051 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:34.598976 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 15:17:34.606299 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.604597 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c5gqw" Jul 06 15:17:34.609722 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.609704 2570 manager.go:354] Recovery completed Jul 06 15:17:34.614148 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.614028 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 15:17:34.616454 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.616435 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-108.ec2.internal" event="NodeHasSufficientMemory" Jul 06 15:17:34.616534 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.616465 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-108.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 15:17:34.616534 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.616477 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-108.ec2.internal" event="NodeHasSufficientPID" Jul 06 15:17:34.617166 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.617149 2570 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 15:17:34.617166 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.617161 2570 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 15:17:34.617298 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.617179 2570 state_mem.go:36] "Initialized new in-memory state store" Jul 06 15:17:34.617298 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.617200 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 15:17:34.618062 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.618041 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-108.ec2.internal" event="NodeHasSufficientMemory" Jul 06 15:17:34.618159 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.618073 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-108.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 15:17:34.618159 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.618085 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-108.ec2.internal" event="NodeHasSufficientPID" Jul 06 15:17:34.618537 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.618522 2570 policy_none.go:49] "None policy: Start" Jul 06 15:17:34.618537 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.618539 2570 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 15:17:34.618677 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.618551 2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 15:17:34.619082 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:34.619013 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-108.ec2.internal.18bfbc55e83b5e0d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-108.ec2.internal,UID:ip-10-0-130-108.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-108.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-108.ec2.internal,},FirstTimestamp:2026-07-06 15:17:34.616452621 +0000 UTC m=+1.230201154,LastTimestamp:2026-07-06 15:17:34.616452621 +0000 UTC m=+1.230201154,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-108.ec2.internal,}" Jul 06 15:17:34.619644 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.619631 2570 policy_none.go:47] "Start" Jul 06 15:17:34.654207 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.654190 2570 manager.go:337] "Starting Device Plugin manager" Jul 06 15:17:34.654304 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:34.654225 2570 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 15:17:34.654304 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.654235 2570 server.go:87] "Starting device plugin registration server" Jul 06 15:17:34.654526 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.654506 2570 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 15:17:34.654626 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.654526 2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 15:17:34.654743 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.654724 2570 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 15:17:34.654861 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.654812 2570 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 15:17:34.654861 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.654823 2570 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 15:17:34.655330 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:34.655308 2570 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 15:17:34.655391 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:34.655368 2570 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-108.ec2.internal\" not found" Jul 06 15:17:34.759083 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.759002 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 15:17:34.760410 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.760144 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-108.ec2.internal" event="NodeHasSufficientMemory" Jul 06 15:17:34.760410 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.760174 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-108.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 15:17:34.760410 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.760184 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-108.ec2.internal" event="NodeHasSufficientPID" Jul 06 15:17:34.760410 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.760220 2570 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-108.ec2.internal" Jul 06 15:17:34.771996 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.771972 2570 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-108.ec2.internal" Jul 06 15:17:34.772075 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:34.772002 2570 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-108.ec2.internal\": node \"ip-10-0-130-108.ec2.internal\" not found" Jul 06 15:17:34.791646 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.791609 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 15:17:34.792941 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.792925 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 15:17:34.793007 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.792965 2570 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 15:17:34.793007 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.792987 2570 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 15:17:34.793099 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:34.793026 2570 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 15:17:34.799220 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.799198 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 15:17:34.809924 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:34.809907 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-108.ec2.internal\" not found" Jul 06 15:17:34.897662 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.897601 2570 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-108.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-108.ec2.internal"] Jul 06 15:17:34.897842 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.897753 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 15:17:34.898789 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.898773 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-108.ec2.internal" event="NodeHasSufficientMemory" Jul 06 15:17:34.898904 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.898804 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-108.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 15:17:34.898904 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.898816 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-108.ec2.internal" event="NodeHasSufficientPID" Jul 06 15:17:34.900357 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.900336 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 15:17:34.900450 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.900434 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-108.ec2.internal" Jul 06 15:17:34.900488 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.900468 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 15:17:34.901048 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.901034 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-108.ec2.internal" event="NodeHasSufficientMemory" Jul 06 15:17:34.901128 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.901056 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-108.ec2.internal" event="NodeHasSufficientMemory" Jul 06 15:17:34.901128 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.901080 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-108.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 15:17:34.901128 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.901092 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-108.ec2.internal" event="NodeHasSufficientPID" Jul 06 15:17:34.901128 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.901063 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-108.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 15:17:34.901262 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.901143 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-108.ec2.internal" event="NodeHasSufficientPID" Jul 06 15:17:34.902387 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.902365 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-108.ec2.internal" Jul 06 15:17:34.902387 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.902390 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 15:17:34.903106 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.903091 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-108.ec2.internal" event="NodeHasSufficientMemory" Jul 06 15:17:34.903185 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.903111 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-108.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 15:17:34.903185 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.903124 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-108.ec2.internal" event="NodeHasSufficientPID" Jul 06 15:17:34.910983 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:34.910964 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-108.ec2.internal\" not found" Jul 06 15:17:34.926410 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:34.926383 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-108.ec2.internal\" not found" node="ip-10-0-130-108.ec2.internal" Jul 06 15:17:34.931038 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:34.931020 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-108.ec2.internal\" not found" node="ip-10-0-130-108.ec2.internal" Jul 06 15:17:34.998475 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.998449 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84b26db8de42be2719159be061d921ee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-108.ec2.internal\" (UID: \"84b26db8de42be2719159be061d921ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-108.ec2.internal" Jul 06 15:17:34.998475 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.998477 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6f497d407b6748f0adb639ce1a9ea113-config\") pod \"kube-apiserver-proxy-ip-10-0-130-108.ec2.internal\" (UID: \"6f497d407b6748f0adb639ce1a9ea113\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-108.ec2.internal" Jul 06 15:17:34.998636 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:34.998493 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/84b26db8de42be2719159be061d921ee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-108.ec2.internal\" (UID: \"84b26db8de42be2719159be061d921ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-108.ec2.internal" Jul 06 15:17:35.012122 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:35.012067 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-108.ec2.internal\" not found" Jul 06 15:17:35.099272 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.099245 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/84b26db8de42be2719159be061d921ee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-108.ec2.internal\" (UID: \"84b26db8de42be2719159be061d921ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-108.ec2.internal" Jul 06 15:17:35.099336 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.099278 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84b26db8de42be2719159be061d921ee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-108.ec2.internal\" (UID: \"84b26db8de42be2719159be061d921ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-108.ec2.internal" Jul 06 15:17:35.099336 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.099295 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6f497d407b6748f0adb639ce1a9ea113-config\") pod \"kube-apiserver-proxy-ip-10-0-130-108.ec2.internal\" (UID: \"6f497d407b6748f0adb639ce1a9ea113\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-108.ec2.internal" Jul 06 15:17:35.099410 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.099325 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6f497d407b6748f0adb639ce1a9ea113-config\") pod \"kube-apiserver-proxy-ip-10-0-130-108.ec2.internal\" (UID: \"6f497d407b6748f0adb639ce1a9ea113\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-108.ec2.internal" Jul 06 15:17:35.099410 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.099350 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/84b26db8de42be2719159be061d921ee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-108.ec2.internal\" (UID: \"84b26db8de42be2719159be061d921ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-108.ec2.internal" Jul 06 15:17:35.099410 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.099351 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84b26db8de42be2719159be061d921ee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-108.ec2.internal\" (UID: \"84b26db8de42be2719159be061d921ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-108.ec2.internal" Jul 06 15:17:35.112309 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:35.112287 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-108.ec2.internal\" not found" Jul 06 15:17:35.212797 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:35.212751 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-108.ec2.internal\" not found" Jul 06 15:17:35.230223 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.230196 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-108.ec2.internal" Jul 06 15:17:35.234667 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.234441 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-108.ec2.internal" Jul 06 15:17:35.314025 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:35.313953 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-108.ec2.internal\" not found" Jul 06 15:17:35.415039 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:35.414999 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-108.ec2.internal\" not found" Jul 06 15:17:35.421255 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.421225 2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 15:17:35.498925 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.498878 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-108.ec2.internal" Jul 06 15:17:35.514467 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.514437 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 15:17:35.516419 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.516397 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-108.ec2.internal" Jul 06 15:17:35.528745 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.528721 2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 15:17:35.528964 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.528934 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 15:17:35.529010 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:35.528959 2570 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://a1726f74684e14eb892bb0f401116777-7ae0b4b8ed5baa14.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-machine-config-operator/pods\": read tcp 10.0.130.108:32856->44.213.253.56:6443: use of closed network connection" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-108.ec2.internal" Jul 06 15:17:35.570260 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.570157 2570 apiserver.go:52] "Watching apiserver" Jul 06 15:17:35.580638 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.580610 2570 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 15:17:35.581072 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.581042 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-66jdx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skmhv","openshift-dns/node-resolver-mgd8j","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-108.ec2.internal","openshift-multus/network-metrics-daemon-6x68d","kube-system/kube-apiserver-proxy-ip-10-0-130-108.ec2.internal","openshift-cluster-node-tuning-operator/tuned-prvr8","openshift-image-registry/node-ca-fbvqs","openshift-multus/multus-94md2","openshift-multus/multus-additional-cni-plugins-9rvk6","openshift-network-diagnostics/network-check-target-v9nsl","openshift-network-operator/iptables-alerter-jlwgn","openshift-ovn-kubernetes/ovnkube-node-9hxpw","kube-system/global-pull-secret-syncer-l77mc"] Jul 06 15:17:35.582626 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.582610 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-66jdx" Jul 06 15:17:35.583924 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.583896 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skmhv" Jul 06 15:17:35.585501 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.585443 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mgd8j" Jul 06 15:17:35.586772 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.586739 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 15:17:35.586888 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.586747 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 15:17:35.586945 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.586896 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rd9dd\"" Jul 06 15:17:35.587076 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.587055 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 15:17:35.587129 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.587103 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 15:17:35.587804 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.587446 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:17:35.587804 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:35.587578 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x68d" podUID="5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b" Jul 06 15:17:35.587804 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.587682 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 15:17:35.588239 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.588219 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 15:17:35.588320 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.588245 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9vhkj\"" Jul 06 15:17:35.588914 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.588897 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-x62fk\"" Jul 06 15:17:35.588977 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.588902 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 15:17:35.589157 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.589137 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.590531 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.590494 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fbvqs" Jul 06 15:17:35.591618 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.591603 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-94md2" Jul 06 15:17:35.591703 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.591694 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 15:17:35.592236 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.592221 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 15:17:35.592690 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.592672 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kc4hf\"" Jul 06 15:17:35.592870 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.592857 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 15:17:35.593091 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.593076 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9rvk6" Jul 06 15:17:35.593484 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.593465 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 15:17:35.593484 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.593481 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hswq6\"" Jul 06 15:17:35.593660 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.593471 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 15:17:35.594302 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.594285 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:17:35.594389 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:35.594343 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9nsl" podUID="e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac" Jul 06 15:17:35.594389 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.594365 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 15:17:35.594948 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.594606 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 15:17:35.594948 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.594606 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 15:17:35.594948 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.594692 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 15:17:35.594948 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.594835 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 15:17:35.595195 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.594999 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-x6ccw\"" Jul 06 15:17:35.595749 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.595732 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jlwgn" Jul 06 15:17:35.596210 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.596191 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ts4hg\"" Jul 06 15:17:35.596297 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.596238 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 15:17:35.596509 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.596493 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 15:17:35.597225 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.597210 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.598442 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.598423 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 15:17:35.598557 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.598465 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:17:35.598557 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.598423 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 15:17:35.598557 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:35.598525 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l77mc" podUID="9aaa45f1-1a99-4e94-96f3-a12ddbe62e83" Jul 06 15:17:35.598707 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.598562 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4jl77\"" Jul 06 15:17:35.598707 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.598653 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 15:17:35.601287 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.601272 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 15:17:35.601405 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.601384 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 15:17:35.602395 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.602378 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e8405516-7f84-421a-a858-e1d77bd5ab96-etc-selinux\") pod \"aws-ebs-csi-driver-node-skmhv\" (UID: \"e8405516-7f84-421a-a858-e1d77bd5ab96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skmhv" Jul 06 15:17:35.602484 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.602401 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-system-cni-dir\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.602484 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.602418 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-node-log\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.602484 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.602435 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ab57a97c-0e04-4482-9444-836a8b6a2f37-agent-certs\") pod \"konnectivity-agent-66jdx\" (UID: \"ab57a97c-0e04-4482-9444-836a8b6a2f37\") " pod="kube-system/konnectivity-agent-66jdx" Jul 06 15:17:35.602484 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.602451 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/65d43ca0-dc12-4e4f-8a7d-a42c0f10a651-hosts-file\") pod \"node-resolver-mgd8j\" (UID: \"65d43ca0-dc12-4e4f-8a7d-a42c0f10a651\") " pod="openshift-dns/node-resolver-mgd8j" Jul 06 15:17:35.602681 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.602487 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-etc-sysconfig\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.602681 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.602524 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-etc-sysctl-d\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.602681 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.602553 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-etc-sysctl-conf\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.602681 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.602590 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 15:17:35.602681 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.602596 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-systemd-units\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.602681 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.602655 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 15:17:35.602681 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.602657 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 15:17:35.603004 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.602653 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ab57a97c-0e04-4482-9444-836a8b6a2f37-konnectivity-ca\") pod \"konnectivity-agent-66jdx\" (UID: \"ab57a97c-0e04-4482-9444-836a8b6a2f37\") " pod="kube-system/konnectivity-agent-66jdx" Jul 06 15:17:35.603004 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.602736 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-etc-modprobe-d\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.603004 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.602761 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-etc-kubernetes\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.603004 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.602785 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-etc-systemd\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.603004 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.602707 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fs6r9\"" Jul 06 15:17:35.603004 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.602818 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-host-run-netns\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.603004 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.602842 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7ebda45d-abe9-429e-a983-2907701d2e61-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9rvk6\" (UID: \"7ebda45d-abe9-429e-a983-2907701d2e61\") " pod="openshift-multus/multus-additional-cni-plugins-9rvk6" Jul 06 15:17:35.603004 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.602865 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpnqm\" (UniqueName: \"kubernetes.io/projected/70fa8e04-5b44-4fbc-b2fc-502965ab8355-kube-api-access-fpnqm\") pod \"iptables-alerter-jlwgn\" (UID: \"70fa8e04-5b44-4fbc-b2fc-502965ab8355\") " pod="openshift-network-operator/iptables-alerter-jlwgn" Jul 06 15:17:35.603004 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.602884 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-etc-openvswitch\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.603004 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.602902 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e8405516-7f84-421a-a858-e1d77bd5ab96-registration-dir\") pod \"aws-ebs-csi-driver-node-skmhv\" (UID: \"e8405516-7f84-421a-a858-e1d77bd5ab96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skmhv" Jul 06 15:17:35.603004 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.602934 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-hostroot\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.603004 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.602962 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ef54bfc9-8403-41f5-9199-8f5f71639838-multus-daemon-config\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.603004 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.602989 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-host-run-ovn-kubernetes\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.603472 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603015 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-etc-tuned\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.603472 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603037 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-tmp\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.603472 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603064 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-host-run-k8s-cni-cncf-io\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.603472 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603089 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-var-lib-openvswitch\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.603472 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603116 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.603472 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603159 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3eca6180-d678-4729-ad6f-979a025af841-ovn-node-metrics-cert\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.603472 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603183 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-host\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.603472 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603200 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btx5n\" (UniqueName: \"kubernetes.io/projected/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-kube-api-access-btx5n\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.603472 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603227 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e8405516-7f84-421a-a858-e1d77bd5ab96-device-dir\") pod \"aws-ebs-csi-driver-node-skmhv\" (UID: \"e8405516-7f84-421a-a858-e1d77bd5ab96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skmhv" Jul 06 15:17:35.603472 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603244 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-etc-kubernetes\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.603472 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603259 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-run-systemd\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.603472 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603276 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-run-ovn\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.603472 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603320 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-multus-cni-dir\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.603472 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603338 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65d43ca0-dc12-4e4f-8a7d-a42c0f10a651-tmp-dir\") pod \"node-resolver-mgd8j\" (UID: \"65d43ca0-dc12-4e4f-8a7d-a42c0f10a651\") " pod="openshift-dns/node-resolver-mgd8j" Jul 06 15:17:35.603472 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603353 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4k7v\" (UniqueName: \"kubernetes.io/projected/65d43ca0-dc12-4e4f-8a7d-a42c0f10a651-kube-api-access-r4k7v\") pod \"node-resolver-mgd8j\" (UID: \"65d43ca0-dc12-4e4f-8a7d-a42c0f10a651\") " pod="openshift-dns/node-resolver-mgd8j" Jul 06 15:17:35.603472 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603371 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-run\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.603472 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603416 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-var-lib-kubelet\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.604154 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603442 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e8405516-7f84-421a-a858-e1d77bd5ab96-kubelet-dir\") pod \"aws-ebs-csi-driver-node-skmhv\" (UID: \"e8405516-7f84-421a-a858-e1d77bd5ab96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skmhv" Jul 06 15:17:35.604154 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603460 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqplx\" (UniqueName: \"kubernetes.io/projected/e8405516-7f84-421a-a858-e1d77bd5ab96-kube-api-access-sqplx\") pod \"aws-ebs-csi-driver-node-skmhv\" (UID: \"e8405516-7f84-421a-a858-e1d77bd5ab96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skmhv" Jul 06 15:17:35.604154 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603477 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70fa8e04-5b44-4fbc-b2fc-502965ab8355-host-slash\") pod \"iptables-alerter-jlwgn\" (UID: \"70fa8e04-5b44-4fbc-b2fc-502965ab8355\") " pod="openshift-network-operator/iptables-alerter-jlwgn" Jul 06 15:17:35.604154 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603502 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-run-openvswitch\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.604154 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603532 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-log-socket\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.604154 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603571 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pf4h\" (UniqueName: \"kubernetes.io/projected/5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b-kube-api-access-5pf4h\") pod \"network-metrics-daemon-6x68d\" (UID: \"5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b\") " pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:17:35.604154 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603588 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-multus-socket-dir-parent\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.604154 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603610 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ebda45d-abe9-429e-a983-2907701d2e61-system-cni-dir\") pod \"multus-additional-cni-plugins-9rvk6\" (UID: \"7ebda45d-abe9-429e-a983-2907701d2e61\") " pod="openshift-multus/multus-additional-cni-plugins-9rvk6" Jul 06 15:17:35.604154 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603641 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7ebda45d-abe9-429e-a983-2907701d2e61-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9rvk6\" (UID: \"7ebda45d-abe9-429e-a983-2907701d2e61\") " pod="openshift-multus/multus-additional-cni-plugins-9rvk6" Jul 06 15:17:35.604154 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603669 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ef54bfc9-8403-41f5-9199-8f5f71639838-cni-binary-copy\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.604154 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603692 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-multus-conf-dir\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.604154 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603715 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7ebda45d-abe9-429e-a983-2907701d2e61-cni-binary-copy\") pod \"multus-additional-cni-plugins-9rvk6\" (UID: \"7ebda45d-abe9-429e-a983-2907701d2e61\") " pod="openshift-multus/multus-additional-cni-plugins-9rvk6" Jul 06 15:17:35.604154 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603767 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3eca6180-d678-4729-ad6f-979a025af841-env-overrides\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.604154 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603784 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3eca6180-d678-4729-ad6f-979a025af841-ovnkube-script-lib\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.604154 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603800 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b-metrics-certs\") pod \"network-metrics-daemon-6x68d\" (UID: \"5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b\") " pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:17:35.604154 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603817 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-lib-modules\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.604889 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603855 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-cnibin\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.604889 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603870 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7ebda45d-abe9-429e-a983-2907701d2e61-os-release\") pod \"multus-additional-cni-plugins-9rvk6\" (UID: \"7ebda45d-abe9-429e-a983-2907701d2e61\") " pod="openshift-multus/multus-additional-cni-plugins-9rvk6" Jul 06 15:17:35.604889 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603900 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7ebda45d-abe9-429e-a983-2907701d2e61-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9rvk6\" (UID: \"7ebda45d-abe9-429e-a983-2907701d2e61\") " pod="openshift-multus/multus-additional-cni-plugins-9rvk6" Jul 06 15:17:35.604889 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603926 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqbdn\" (UniqueName: \"kubernetes.io/projected/3eca6180-d678-4729-ad6f-979a025af841-kube-api-access-xqbdn\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.604889 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603941 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-sys\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.604889 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603958 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbgxx\" (UniqueName: \"kubernetes.io/projected/9a7b5582-0829-4212-8ff0-6379e5eb5306-kube-api-access-gbgxx\") pod \"node-ca-fbvqs\" (UID: \"9a7b5582-0829-4212-8ff0-6379e5eb5306\") " pod="openshift-image-registry/node-ca-fbvqs" Jul 06 15:17:35.604889 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.603979 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-host-var-lib-cni-bin\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.604889 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.604017 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-host-kubelet\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.604889 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.604046 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrc5d\" (UniqueName: \"kubernetes.io/projected/7ebda45d-abe9-429e-a983-2907701d2e61-kube-api-access-mrc5d\") pod \"multus-additional-cni-plugins-9rvk6\" (UID: \"7ebda45d-abe9-429e-a983-2907701d2e61\") " pod="openshift-multus/multus-additional-cni-plugins-9rvk6" Jul 06 15:17:35.604889 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.604074 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-host-slash\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.604889 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.604099 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-host-cni-bin\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.604889 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.604121 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-os-release\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.604889 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.604146 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/70fa8e04-5b44-4fbc-b2fc-502965ab8355-iptables-alerter-script\") pod \"iptables-alerter-jlwgn\" (UID: \"70fa8e04-5b44-4fbc-b2fc-502965ab8355\") " pod="openshift-network-operator/iptables-alerter-jlwgn" Jul 06 15:17:35.604889 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.604157 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 15:17:35.604889 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.604173 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-host-run-netns\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.604889 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.604223 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3eca6180-d678-4729-ad6f-979a025af841-ovnkube-config\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.604889 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.604260 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e8405516-7f84-421a-a858-e1d77bd5ab96-sys-fs\") pod \"aws-ebs-csi-driver-node-skmhv\" (UID: \"e8405516-7f84-421a-a858-e1d77bd5ab96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skmhv" Jul 06 15:17:35.605402 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.604287 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a7b5582-0829-4212-8ff0-6379e5eb5306-host\") pod \"node-ca-fbvqs\" (UID: \"9a7b5582-0829-4212-8ff0-6379e5eb5306\") " pod="openshift-image-registry/node-ca-fbvqs" Jul 06 15:17:35.605402 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.604313 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a7b5582-0829-4212-8ff0-6379e5eb5306-serviceca\") pod \"node-ca-fbvqs\" (UID: \"9a7b5582-0829-4212-8ff0-6379e5eb5306\") " pod="openshift-image-registry/node-ca-fbvqs" Jul 06 15:17:35.605402 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.604338 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-host-var-lib-cni-multus\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.605402 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.604365 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-host-run-multus-certs\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.605402 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.604392 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzwcw\" (UniqueName: \"kubernetes.io/projected/ef54bfc9-8403-41f5-9199-8f5f71639838-kube-api-access-zzwcw\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.605402 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.604422 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkznd\" (UniqueName: \"kubernetes.io/projected/e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac-kube-api-access-wkznd\") pod \"network-check-target-v9nsl\" (UID: \"e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac\") " pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:17:35.605402 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.604447 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e8405516-7f84-421a-a858-e1d77bd5ab96-socket-dir\") pod \"aws-ebs-csi-driver-node-skmhv\" (UID: \"e8405516-7f84-421a-a858-e1d77bd5ab96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skmhv" Jul 06 15:17:35.605402 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.604470 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-host-var-lib-kubelet\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.605402 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.604496 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7ebda45d-abe9-429e-a983-2907701d2e61-cnibin\") pod \"multus-additional-cni-plugins-9rvk6\" (UID: \"7ebda45d-abe9-429e-a983-2907701d2e61\") " pod="openshift-multus/multus-additional-cni-plugins-9rvk6" Jul 06 15:17:35.605402 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.604520 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-host-cni-netd\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.606858 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.606812 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 15:12:34 +0000 UTC" deadline="2028-02-27 22:12:27.151481879 +0000 UTC" Jul 06 15:17:35.606923 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.606858 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14430h54m51.544627353s" Jul 06 15:17:35.611265 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.611249 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 15:17:35.634951 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.634925 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vctnv" Jul 06 15:17:35.645688 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.645665 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vctnv" Jul 06 15:17:35.697016 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.696994 2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 15:17:35.705086 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705058 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e8405516-7f84-421a-a858-e1d77bd5ab96-device-dir\") pod \"aws-ebs-csi-driver-node-skmhv\" (UID: \"e8405516-7f84-421a-a858-e1d77bd5ab96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skmhv" Jul 06 15:17:35.705231 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705103 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-etc-kubernetes\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.705231 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705133 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-run-systemd\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.705231 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705159 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-run-ovn\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.705231 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705186 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-multus-cni-dir\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.705231 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705193 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e8405516-7f84-421a-a858-e1d77bd5ab96-device-dir\") pod \"aws-ebs-csi-driver-node-skmhv\" (UID: \"e8405516-7f84-421a-a858-e1d77bd5ab96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skmhv" Jul 06 15:17:35.705231 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705214 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65d43ca0-dc12-4e4f-8a7d-a42c0f10a651-tmp-dir\") pod \"node-resolver-mgd8j\" (UID: \"65d43ca0-dc12-4e4f-8a7d-a42c0f10a651\") " pod="openshift-dns/node-resolver-mgd8j" Jul 06 15:17:35.705580 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705255 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-run-systemd\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.705580 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705263 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-run-ovn\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.705580 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705291 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4k7v\" (UniqueName: \"kubernetes.io/projected/65d43ca0-dc12-4e4f-8a7d-a42c0f10a651-kube-api-access-r4k7v\") pod \"node-resolver-mgd8j\" (UID: \"65d43ca0-dc12-4e4f-8a7d-a42c0f10a651\") " pod="openshift-dns/node-resolver-mgd8j" Jul 06 15:17:35.705580 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705313 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-etc-kubernetes\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.705580 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705321 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-run\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.705580 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705349 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-var-lib-kubelet\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.705580 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705373 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e8405516-7f84-421a-a858-e1d77bd5ab96-kubelet-dir\") pod \"aws-ebs-csi-driver-node-skmhv\" (UID: \"e8405516-7f84-421a-a858-e1d77bd5ab96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skmhv" Jul 06 15:17:35.705580 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705382 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-multus-cni-dir\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.705580 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705400 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqplx\" (UniqueName: \"kubernetes.io/projected/e8405516-7f84-421a-a858-e1d77bd5ab96-kube-api-access-sqplx\") pod \"aws-ebs-csi-driver-node-skmhv\" (UID: \"e8405516-7f84-421a-a858-e1d77bd5ab96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skmhv" Jul 06 15:17:35.705580 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705440 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-var-lib-kubelet\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.705580 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705445 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70fa8e04-5b44-4fbc-b2fc-502965ab8355-host-slash\") pod \"iptables-alerter-jlwgn\" (UID: \"70fa8e04-5b44-4fbc-b2fc-502965ab8355\") " pod="openshift-network-operator/iptables-alerter-jlwgn" Jul 06 15:17:35.705580 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705481 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-run-openvswitch\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.705580 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705491 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70fa8e04-5b44-4fbc-b2fc-502965ab8355-host-slash\") pod \"iptables-alerter-jlwgn\" (UID: \"70fa8e04-5b44-4fbc-b2fc-502965ab8355\") " pod="openshift-network-operator/iptables-alerter-jlwgn" Jul 06 15:17:35.705580 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705507 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-log-socket\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.705580 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705530 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65d43ca0-dc12-4e4f-8a7d-a42c0f10a651-tmp-dir\") pod \"node-resolver-mgd8j\" (UID: \"65d43ca0-dc12-4e4f-8a7d-a42c0f10a651\") " pod="openshift-dns/node-resolver-mgd8j" Jul 06 15:17:35.705580 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705555 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pf4h\" (UniqueName: \"kubernetes.io/projected/5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b-kube-api-access-5pf4h\") pod \"network-metrics-daemon-6x68d\" (UID: \"5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b\") " pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:17:35.705580 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705562 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e8405516-7f84-421a-a858-e1d77bd5ab96-kubelet-dir\") pod \"aws-ebs-csi-driver-node-skmhv\" (UID: \"e8405516-7f84-421a-a858-e1d77bd5ab96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skmhv" Jul 06 15:17:35.706473 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705582 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-multus-socket-dir-parent\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.706473 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705640 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-log-socket\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.706473 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705710 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-run-openvswitch\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.706473 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705741 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ebda45d-abe9-429e-a983-2907701d2e61-system-cni-dir\") pod \"multus-additional-cni-plugins-9rvk6\" (UID: \"7ebda45d-abe9-429e-a983-2907701d2e61\") " pod="openshift-multus/multus-additional-cni-plugins-9rvk6" Jul 06 15:17:35.706473 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705763 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7ebda45d-abe9-429e-a983-2907701d2e61-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9rvk6\" (UID: \"7ebda45d-abe9-429e-a983-2907701d2e61\") " pod="openshift-multus/multus-additional-cni-plugins-9rvk6" Jul 06 15:17:35.706473 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705781 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ef54bfc9-8403-41f5-9199-8f5f71639838-cni-binary-copy\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.706473 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705805 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-multus-conf-dir\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.706473 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705816 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ebda45d-abe9-429e-a983-2907701d2e61-system-cni-dir\") pod \"multus-additional-cni-plugins-9rvk6\" (UID: \"7ebda45d-abe9-429e-a983-2907701d2e61\") " pod="openshift-multus/multus-additional-cni-plugins-9rvk6" Jul 06 15:17:35.706473 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705895 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-run\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.706473 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705920 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7ebda45d-abe9-429e-a983-2907701d2e61-cni-binary-copy\") pod \"multus-additional-cni-plugins-9rvk6\" (UID: \"7ebda45d-abe9-429e-a983-2907701d2e61\") " pod="openshift-multus/multus-additional-cni-plugins-9rvk6" Jul 06 15:17:35.706473 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705899 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-multus-socket-dir-parent\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.706473 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705949 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3eca6180-d678-4729-ad6f-979a025af841-env-overrides\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.706473 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.705978 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3eca6180-d678-4729-ad6f-979a025af841-ovnkube-script-lib\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.706473 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.706004 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b-metrics-certs\") pod \"network-metrics-daemon-6x68d\" (UID: \"5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b\") " pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:17:35.706473 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.706030 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-lib-modules\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.706473 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.706046 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-multus-conf-dir\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.706473 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.706050 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-cnibin\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.707196 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.706084 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7ebda45d-abe9-429e-a983-2907701d2e61-os-release\") pod \"multus-additional-cni-plugins-9rvk6\" (UID: \"7ebda45d-abe9-429e-a983-2907701d2e61\") " pod="openshift-multus/multus-additional-cni-plugins-9rvk6" Jul 06 15:17:35.707196 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.706111 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7ebda45d-abe9-429e-a983-2907701d2e61-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9rvk6\" (UID: \"7ebda45d-abe9-429e-a983-2907701d2e61\") " pod="openshift-multus/multus-additional-cni-plugins-9rvk6" Jul 06 15:17:35.707196 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:35.706130 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:17:35.707196 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.706142 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqbdn\" (UniqueName: \"kubernetes.io/projected/3eca6180-d678-4729-ad6f-979a025af841-kube-api-access-xqbdn\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.707196 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.706168 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-sys\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.707196 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:35.706192 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b-metrics-certs podName:5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b nodeName:}" failed. No retries permitted until 2026-07-06 15:17:36.206170346 +0000 UTC m=+2.819918884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b-metrics-certs") pod "network-metrics-daemon-6x68d" (UID: "5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:17:35.707196 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.706220 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-sys\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.707196 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.706225 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-lib-modules\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.707196 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.706222 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9aaa45f1-1a99-4e94-96f3-a12ddbe62e83-kubelet-config\") pod \"global-pull-secret-syncer-l77mc\" (UID: \"9aaa45f1-1a99-4e94-96f3-a12ddbe62e83\") " pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:17:35.707196 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.706324 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-cnibin\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.707196 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.706526 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7ebda45d-abe9-429e-a983-2907701d2e61-cni-binary-copy\") pod \"multus-additional-cni-plugins-9rvk6\" (UID: \"7ebda45d-abe9-429e-a983-2907701d2e61\") " pod="openshift-multus/multus-additional-cni-plugins-9rvk6" Jul 06 15:17:35.707196 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.706539 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9aaa45f1-1a99-4e94-96f3-a12ddbe62e83-dbus\") pod \"global-pull-secret-syncer-l77mc\" (UID: \"9aaa45f1-1a99-4e94-96f3-a12ddbe62e83\") " pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:17:35.707196 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.706677 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3eca6180-d678-4729-ad6f-979a025af841-env-overrides\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.707196 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.706718 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7ebda45d-abe9-429e-a983-2907701d2e61-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9rvk6\" (UID: \"7ebda45d-abe9-429e-a983-2907701d2e61\") " pod="openshift-multus/multus-additional-cni-plugins-9rvk6" Jul 06 15:17:35.707196 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.706782 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbgxx\" (UniqueName: \"kubernetes.io/projected/9a7b5582-0829-4212-8ff0-6379e5eb5306-kube-api-access-gbgxx\") pod \"node-ca-fbvqs\" (UID: \"9a7b5582-0829-4212-8ff0-6379e5eb5306\") " pod="openshift-image-registry/node-ca-fbvqs" Jul 06 15:17:35.707196 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.706793 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7ebda45d-abe9-429e-a983-2907701d2e61-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9rvk6\" (UID: \"7ebda45d-abe9-429e-a983-2907701d2e61\") " pod="openshift-multus/multus-additional-cni-plugins-9rvk6" Jul 06 15:17:35.707196 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.706817 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-host-var-lib-cni-bin\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.708026 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.706846 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-host-var-lib-cni-bin\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.708026 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.706884 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-host-kubelet\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.708026 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.706938 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrc5d\" (UniqueName: \"kubernetes.io/projected/7ebda45d-abe9-429e-a983-2907701d2e61-kube-api-access-mrc5d\") pod \"multus-additional-cni-plugins-9rvk6\" (UID: \"7ebda45d-abe9-429e-a983-2907701d2e61\") " pod="openshift-multus/multus-additional-cni-plugins-9rvk6" Jul 06 15:17:35.708026 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.706971 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-host-kubelet\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.708026 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.706973 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3eca6180-d678-4729-ad6f-979a025af841-ovnkube-script-lib\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.708026 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707036 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-host-slash\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.708026 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707067 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-host-cni-bin\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.708026 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.706691 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7ebda45d-abe9-429e-a983-2907701d2e61-os-release\") pod \"multus-additional-cni-plugins-9rvk6\" (UID: \"7ebda45d-abe9-429e-a983-2907701d2e61\") " pod="openshift-multus/multus-additional-cni-plugins-9rvk6" Jul 06 15:17:35.708026 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707102 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-os-release\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.708026 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707127 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-host-slash\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.708026 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707128 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/70fa8e04-5b44-4fbc-b2fc-502965ab8355-iptables-alerter-script\") pod \"iptables-alerter-jlwgn\" (UID: \"70fa8e04-5b44-4fbc-b2fc-502965ab8355\") " pod="openshift-network-operator/iptables-alerter-jlwgn" Jul 06 15:17:35.708026 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707144 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-host-cni-bin\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.708026 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707178 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-host-run-netns\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.708026 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707186 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-os-release\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.708026 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707203 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3eca6180-d678-4729-ad6f-979a025af841-ovnkube-config\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.708026 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707236 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e8405516-7f84-421a-a858-e1d77bd5ab96-sys-fs\") pod \"aws-ebs-csi-driver-node-skmhv\" (UID: \"e8405516-7f84-421a-a858-e1d77bd5ab96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skmhv" Jul 06 15:17:35.708026 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707192 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ef54bfc9-8403-41f5-9199-8f5f71639838-cni-binary-copy\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.708026 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707260 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a7b5582-0829-4212-8ff0-6379e5eb5306-host\") pod \"node-ca-fbvqs\" (UID: \"9a7b5582-0829-4212-8ff0-6379e5eb5306\") " pod="openshift-image-registry/node-ca-fbvqs" Jul 06 15:17:35.708886 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707284 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a7b5582-0829-4212-8ff0-6379e5eb5306-serviceca\") pod \"node-ca-fbvqs\" (UID: \"9a7b5582-0829-4212-8ff0-6379e5eb5306\") " pod="openshift-image-registry/node-ca-fbvqs" Jul 06 15:17:35.708886 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707309 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-host-var-lib-cni-multus\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.708886 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707337 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a7b5582-0829-4212-8ff0-6379e5eb5306-host\") pod \"node-ca-fbvqs\" (UID: \"9a7b5582-0829-4212-8ff0-6379e5eb5306\") " pod="openshift-image-registry/node-ca-fbvqs" Jul 06 15:17:35.708886 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707336 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-host-run-multus-certs\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.708886 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707376 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-host-run-multus-certs\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.708886 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707387 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzwcw\" (UniqueName: \"kubernetes.io/projected/ef54bfc9-8403-41f5-9199-8f5f71639838-kube-api-access-zzwcw\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.708886 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707406 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkznd\" (UniqueName: \"kubernetes.io/projected/e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac-kube-api-access-wkznd\") pod \"network-check-target-v9nsl\" (UID: \"e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac\") " pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:17:35.708886 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707236 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-host-run-netns\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.708886 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707723 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/70fa8e04-5b44-4fbc-b2fc-502965ab8355-iptables-alerter-script\") pod \"iptables-alerter-jlwgn\" (UID: \"70fa8e04-5b44-4fbc-b2fc-502965ab8355\") " pod="openshift-network-operator/iptables-alerter-jlwgn" Jul 06 15:17:35.708886 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707742 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3eca6180-d678-4729-ad6f-979a025af841-ovnkube-config\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.708886 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707313 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e8405516-7f84-421a-a858-e1d77bd5ab96-sys-fs\") pod \"aws-ebs-csi-driver-node-skmhv\" (UID: \"e8405516-7f84-421a-a858-e1d77bd5ab96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skmhv" Jul 06 15:17:35.708886 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707781 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-host-var-lib-cni-multus\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.708886 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707785 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e8405516-7f84-421a-a858-e1d77bd5ab96-socket-dir\") pod \"aws-ebs-csi-driver-node-skmhv\" (UID: \"e8405516-7f84-421a-a858-e1d77bd5ab96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skmhv" Jul 06 15:17:35.708886 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707827 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-host-var-lib-kubelet\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.708886 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707854 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7ebda45d-abe9-429e-a983-2907701d2e61-cnibin\") pod \"multus-additional-cni-plugins-9rvk6\" (UID: \"7ebda45d-abe9-429e-a983-2907701d2e61\") " pod="openshift-multus/multus-additional-cni-plugins-9rvk6" Jul 06 15:17:35.708886 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707869 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e8405516-7f84-421a-a858-e1d77bd5ab96-socket-dir\") pod \"aws-ebs-csi-driver-node-skmhv\" (UID: \"e8405516-7f84-421a-a858-e1d77bd5ab96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skmhv" Jul 06 15:17:35.708886 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707881 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-host-cni-netd\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.709602 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707918 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-host-var-lib-kubelet\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.709602 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707923 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7ebda45d-abe9-429e-a983-2907701d2e61-cnibin\") pod \"multus-additional-cni-plugins-9rvk6\" (UID: \"7ebda45d-abe9-429e-a983-2907701d2e61\") " pod="openshift-multus/multus-additional-cni-plugins-9rvk6" Jul 06 15:17:35.709602 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707931 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-host-cni-netd\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.709602 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707942 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9aaa45f1-1a99-4e94-96f3-a12ddbe62e83-original-pull-secret\") pod \"global-pull-secret-syncer-l77mc\" (UID: \"9aaa45f1-1a99-4e94-96f3-a12ddbe62e83\") " pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:17:35.709602 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707966 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e8405516-7f84-421a-a858-e1d77bd5ab96-etc-selinux\") pod \"aws-ebs-csi-driver-node-skmhv\" (UID: \"e8405516-7f84-421a-a858-e1d77bd5ab96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skmhv" Jul 06 15:17:35.709602 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.707992 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-system-cni-dir\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.709602 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708031 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-system-cni-dir\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.709602 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708056 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-node-log\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.709602 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708085 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ab57a97c-0e04-4482-9444-836a8b6a2f37-agent-certs\") pod \"konnectivity-agent-66jdx\" (UID: \"ab57a97c-0e04-4482-9444-836a8b6a2f37\") " pod="kube-system/konnectivity-agent-66jdx" Jul 06 15:17:35.709602 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708109 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/65d43ca0-dc12-4e4f-8a7d-a42c0f10a651-hosts-file\") pod \"node-resolver-mgd8j\" (UID: \"65d43ca0-dc12-4e4f-8a7d-a42c0f10a651\") " pod="openshift-dns/node-resolver-mgd8j" Jul 06 15:17:35.709602 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708112 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e8405516-7f84-421a-a858-e1d77bd5ab96-etc-selinux\") pod \"aws-ebs-csi-driver-node-skmhv\" (UID: \"e8405516-7f84-421a-a858-e1d77bd5ab96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skmhv" Jul 06 15:17:35.709602 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708120 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-node-log\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.709602 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708165 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/65d43ca0-dc12-4e4f-8a7d-a42c0f10a651-hosts-file\") pod \"node-resolver-mgd8j\" (UID: \"65d43ca0-dc12-4e4f-8a7d-a42c0f10a651\") " pod="openshift-dns/node-resolver-mgd8j" Jul 06 15:17:35.709602 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708218 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-etc-sysconfig\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.709602 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708247 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-etc-sysctl-d\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.709602 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708274 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-etc-sysctl-conf\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.709602 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708301 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-systemd-units\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.709602 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708274 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-etc-sysconfig\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.710197 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708326 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ab57a97c-0e04-4482-9444-836a8b6a2f37-konnectivity-ca\") pod \"konnectivity-agent-66jdx\" (UID: \"ab57a97c-0e04-4482-9444-836a8b6a2f37\") " pod="kube-system/konnectivity-agent-66jdx" Jul 06 15:17:35.710197 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708353 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-etc-modprobe-d\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.710197 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708374 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-etc-sysctl-d\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.710197 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708381 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-systemd-units\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.710197 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708377 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-etc-kubernetes\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.710197 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708420 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-etc-systemd\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.710197 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708427 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-etc-kubernetes\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.710197 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708448 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-host-run-netns\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.710197 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708458 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-etc-sysctl-conf\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.710197 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708477 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7ebda45d-abe9-429e-a983-2907701d2e61-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9rvk6\" (UID: \"7ebda45d-abe9-429e-a983-2907701d2e61\") " pod="openshift-multus/multus-additional-cni-plugins-9rvk6" Jul 06 15:17:35.710197 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708489 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-etc-modprobe-d\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.710197 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708477 2570 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 15:17:35.710197 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708506 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpnqm\" (UniqueName: \"kubernetes.io/projected/70fa8e04-5b44-4fbc-b2fc-502965ab8355-kube-api-access-fpnqm\") pod \"iptables-alerter-jlwgn\" (UID: \"70fa8e04-5b44-4fbc-b2fc-502965ab8355\") " pod="openshift-network-operator/iptables-alerter-jlwgn" Jul 06 15:17:35.710197 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708540 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-host-run-netns\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.710197 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708548 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-etc-openvswitch\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.710197 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708540 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-etc-systemd\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.710197 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708575 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e8405516-7f84-421a-a858-e1d77bd5ab96-registration-dir\") pod \"aws-ebs-csi-driver-node-skmhv\" (UID: \"e8405516-7f84-421a-a858-e1d77bd5ab96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skmhv" Jul 06 15:17:35.710197 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708600 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-hostroot\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.710787 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708629 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ef54bfc9-8403-41f5-9199-8f5f71639838-multus-daemon-config\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.710787 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708639 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a7b5582-0829-4212-8ff0-6379e5eb5306-serviceca\") pod \"node-ca-fbvqs\" (UID: \"9a7b5582-0829-4212-8ff0-6379e5eb5306\") " pod="openshift-image-registry/node-ca-fbvqs" Jul 06 15:17:35.710787 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708655 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-host-run-ovn-kubernetes\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.710787 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708674 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e8405516-7f84-421a-a858-e1d77bd5ab96-registration-dir\") pod \"aws-ebs-csi-driver-node-skmhv\" (UID: \"e8405516-7f84-421a-a858-e1d77bd5ab96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skmhv" Jul 06 15:17:35.710787 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708682 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-etc-tuned\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.710787 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708693 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-hostroot\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.710787 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708699 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7ebda45d-abe9-429e-a983-2907701d2e61-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9rvk6\" (UID: \"7ebda45d-abe9-429e-a983-2907701d2e61\") " pod="openshift-multus/multus-additional-cni-plugins-9rvk6" Jul 06 15:17:35.710787 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708708 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-tmp\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.710787 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708738 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-host-run-k8s-cni-cncf-io\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.710787 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708749 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-host-run-ovn-kubernetes\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.710787 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708717 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-etc-openvswitch\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.710787 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708772 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-var-lib-openvswitch\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.710787 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708788 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ef54bfc9-8403-41f5-9199-8f5f71639838-host-run-k8s-cni-cncf-io\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.710787 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708801 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.710787 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708831 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3eca6180-d678-4729-ad6f-979a025af841-ovn-node-metrics-cert\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.710787 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708855 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-host\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.710787 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708839 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-var-lib-openvswitch\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.711271 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708897 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3eca6180-d678-4729-ad6f-979a025af841-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.711271 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.708901 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btx5n\" (UniqueName: \"kubernetes.io/projected/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-kube-api-access-btx5n\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.711271 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.709009 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-host\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.711271 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.709094 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ab57a97c-0e04-4482-9444-836a8b6a2f37-konnectivity-ca\") pod \"konnectivity-agent-66jdx\" (UID: \"ab57a97c-0e04-4482-9444-836a8b6a2f37\") " pod="kube-system/konnectivity-agent-66jdx" Jul 06 15:17:35.711271 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.709239 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ef54bfc9-8403-41f5-9199-8f5f71639838-multus-daemon-config\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.711936 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.711906 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-tmp\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.711936 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.711928 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-etc-tuned\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.712060 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.711933 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3eca6180-d678-4729-ad6f-979a025af841-ovn-node-metrics-cert\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.712126 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.712111 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ab57a97c-0e04-4482-9444-836a8b6a2f37-agent-certs\") pod \"konnectivity-agent-66jdx\" (UID: \"ab57a97c-0e04-4482-9444-836a8b6a2f37\") " pod="kube-system/konnectivity-agent-66jdx" Jul 06 15:17:35.716975 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.716952 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4k7v\" (UniqueName: \"kubernetes.io/projected/65d43ca0-dc12-4e4f-8a7d-a42c0f10a651-kube-api-access-r4k7v\") pod \"node-resolver-mgd8j\" (UID: \"65d43ca0-dc12-4e4f-8a7d-a42c0f10a651\") " pod="openshift-dns/node-resolver-mgd8j" Jul 06 15:17:35.717809 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.717785 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrc5d\" (UniqueName: \"kubernetes.io/projected/7ebda45d-abe9-429e-a983-2907701d2e61-kube-api-access-mrc5d\") pod \"multus-additional-cni-plugins-9rvk6\" (UID: \"7ebda45d-abe9-429e-a983-2907701d2e61\") " pod="openshift-multus/multus-additional-cni-plugins-9rvk6" Jul 06 15:17:35.721696 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:35.721678 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 15:17:35.721775 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:35.721699 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 15:17:35.721775 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:35.721709 2570 projected.go:196] Error preparing data for projected volume kube-api-access-wkznd for pod openshift-network-diagnostics/network-check-target-v9nsl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:17:35.721775 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:35.721762 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac-kube-api-access-wkznd podName:e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac nodeName:}" failed. No retries permitted until 2026-07-06 15:17:36.2217478 +0000 UTC m=+2.835496335 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wkznd" (UniqueName: "kubernetes.io/projected/e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac-kube-api-access-wkznd") pod "network-check-target-v9nsl" (UID: "e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:17:35.724174 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.724150 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqbdn\" (UniqueName: \"kubernetes.io/projected/3eca6180-d678-4729-ad6f-979a025af841-kube-api-access-xqbdn\") pod \"ovnkube-node-9hxpw\" (UID: \"3eca6180-d678-4729-ad6f-979a025af841\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:35.724885 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.724857 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpnqm\" (UniqueName: \"kubernetes.io/projected/70fa8e04-5b44-4fbc-b2fc-502965ab8355-kube-api-access-fpnqm\") pod \"iptables-alerter-jlwgn\" (UID: \"70fa8e04-5b44-4fbc-b2fc-502965ab8355\") " pod="openshift-network-operator/iptables-alerter-jlwgn" Jul 06 15:17:35.725026 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.725008 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pf4h\" (UniqueName: \"kubernetes.io/projected/5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b-kube-api-access-5pf4h\") pod \"network-metrics-daemon-6x68d\" (UID: \"5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b\") " pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:17:35.725026 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.725017 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbgxx\" (UniqueName: \"kubernetes.io/projected/9a7b5582-0829-4212-8ff0-6379e5eb5306-kube-api-access-gbgxx\") pod \"node-ca-fbvqs\" (UID: \"9a7b5582-0829-4212-8ff0-6379e5eb5306\") " pod="openshift-image-registry/node-ca-fbvqs" Jul 06 15:17:35.725282 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.725262 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btx5n\" (UniqueName: \"kubernetes.io/projected/17759dc6-09ce-4dd1-b5e9-4076390ec9e1-kube-api-access-btx5n\") pod \"tuned-prvr8\" (UID: \"17759dc6-09ce-4dd1-b5e9-4076390ec9e1\") " pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.725565 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.725545 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqplx\" (UniqueName: \"kubernetes.io/projected/e8405516-7f84-421a-a858-e1d77bd5ab96-kube-api-access-sqplx\") pod \"aws-ebs-csi-driver-node-skmhv\" (UID: \"e8405516-7f84-421a-a858-e1d77bd5ab96\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skmhv" Jul 06 15:17:35.725729 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.725714 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzwcw\" (UniqueName: \"kubernetes.io/projected/ef54bfc9-8403-41f5-9199-8f5f71639838-kube-api-access-zzwcw\") pod \"multus-94md2\" (UID: \"ef54bfc9-8403-41f5-9199-8f5f71639838\") " pod="openshift-multus/multus-94md2" Jul 06 15:17:35.809690 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.809657 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9aaa45f1-1a99-4e94-96f3-a12ddbe62e83-kubelet-config\") pod \"global-pull-secret-syncer-l77mc\" (UID: \"9aaa45f1-1a99-4e94-96f3-a12ddbe62e83\") " pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:17:35.809887 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.809698 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9aaa45f1-1a99-4e94-96f3-a12ddbe62e83-dbus\") pod \"global-pull-secret-syncer-l77mc\" (UID: \"9aaa45f1-1a99-4e94-96f3-a12ddbe62e83\") " pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:17:35.809887 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.809784 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9aaa45f1-1a99-4e94-96f3-a12ddbe62e83-original-pull-secret\") pod \"global-pull-secret-syncer-l77mc\" (UID: \"9aaa45f1-1a99-4e94-96f3-a12ddbe62e83\") " pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:17:35.809887 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.809790 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9aaa45f1-1a99-4e94-96f3-a12ddbe62e83-kubelet-config\") pod \"global-pull-secret-syncer-l77mc\" (UID: \"9aaa45f1-1a99-4e94-96f3-a12ddbe62e83\") " pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:17:35.809887 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.809875 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9aaa45f1-1a99-4e94-96f3-a12ddbe62e83-dbus\") pod \"global-pull-secret-syncer-l77mc\" (UID: \"9aaa45f1-1a99-4e94-96f3-a12ddbe62e83\") " pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:17:35.810054 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:35.809885 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 15:17:35.812699 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:35.810515 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aaa45f1-1a99-4e94-96f3-a12ddbe62e83-original-pull-secret podName:9aaa45f1-1a99-4e94-96f3-a12ddbe62e83 nodeName:}" failed. No retries permitted until 2026-07-06 15:17:36.310488192 +0000 UTC m=+2.924236723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9aaa45f1-1a99-4e94-96f3-a12ddbe62e83-original-pull-secret") pod "global-pull-secret-syncer-l77mc" (UID: "9aaa45f1-1a99-4e94-96f3-a12ddbe62e83") : object "kube-system"/"original-pull-secret" not registered Jul 06 15:17:35.905061 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.905002 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-66jdx" Jul 06 15:17:35.913072 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.913053 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skmhv" Jul 06 15:17:35.939700 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.939679 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mgd8j" Jul 06 15:17:35.947391 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:35.947357 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8405516_7f84_421a_a858_e1d77bd5ab96.slice/crio-94d6ed938f664d67b801a41e32a336d7b10aa6d70644a7109b8e49645852b6b7 WatchSource:0}: Error finding container 94d6ed938f664d67b801a41e32a336d7b10aa6d70644a7109b8e49645852b6b7: Status 404 returned error can't find the container with id 94d6ed938f664d67b801a41e32a336d7b10aa6d70644a7109b8e49645852b6b7 Jul 06 15:17:35.947872 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:35.947850 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f497d407b6748f0adb639ce1a9ea113.slice/crio-01e0a7659bdb795df5dbf77cf15eedcf1749aa15231c37442de15224fc684176 WatchSource:0}: Error finding container 01e0a7659bdb795df5dbf77cf15eedcf1749aa15231c37442de15224fc684176: Status 404 returned error can't find the container with id 01e0a7659bdb795df5dbf77cf15eedcf1749aa15231c37442de15224fc684176 Jul 06 15:17:35.948366 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.948345 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-prvr8" Jul 06 15:17:35.948446 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:35.948363 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84b26db8de42be2719159be061d921ee.slice/crio-02b53052008b3b2af10b25c9ec07647815299ec3dfb106211f63802ce32b1e3e WatchSource:0}: Error finding container 02b53052008b3b2af10b25c9ec07647815299ec3dfb106211f63802ce32b1e3e: Status 404 returned error can't find the container with id 02b53052008b3b2af10b25c9ec07647815299ec3dfb106211f63802ce32b1e3e Jul 06 15:17:35.949795 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:35.949767 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab57a97c_0e04_4482_9444_836a8b6a2f37.slice/crio-908d607f90e1b92ffbb658d612d7d52eb7581a78146fa350d1e71068b4b71238 WatchSource:0}: Error finding container 908d607f90e1b92ffbb658d612d7d52eb7581a78146fa350d1e71068b4b71238: Status 404 returned error can't find the container with id 908d607f90e1b92ffbb658d612d7d52eb7581a78146fa350d1e71068b4b71238 Jul 06 15:17:35.952089 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:35.952066 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d43ca0_dc12_4e4f_8a7d_a42c0f10a651.slice/crio-f4e41c1969daa3f705a8e814cbabaf2ab18d84a03b15c23e43faf3f1340d0ba2 WatchSource:0}: Error finding container f4e41c1969daa3f705a8e814cbabaf2ab18d84a03b15c23e43faf3f1340d0ba2: Status 404 returned error can't find the container with id f4e41c1969daa3f705a8e814cbabaf2ab18d84a03b15c23e43faf3f1340d0ba2 Jul 06 15:17:35.954539 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.954520 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 15:17:35.957757 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:35.957666 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17759dc6_09ce_4dd1_b5e9_4076390ec9e1.slice/crio-343cc6b9ffac85be3a3658dfd0fa600deb44ed55ab29ba54204648d95026dce0 WatchSource:0}: Error finding container 343cc6b9ffac85be3a3658dfd0fa600deb44ed55ab29ba54204648d95026dce0: Status 404 returned error can't find the container with id 343cc6b9ffac85be3a3658dfd0fa600deb44ed55ab29ba54204648d95026dce0 Jul 06 15:17:35.968545 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.968523 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fbvqs" Jul 06 15:17:35.976727 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:35.976699 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a7b5582_0829_4212_8ff0_6379e5eb5306.slice/crio-f23b75c39e25d8a6a1e1bf888a3a43c2f77de892d60aeeed7c0dce5542b54c1d WatchSource:0}: Error finding container f23b75c39e25d8a6a1e1bf888a3a43c2f77de892d60aeeed7c0dce5542b54c1d: Status 404 returned error can't find the container with id f23b75c39e25d8a6a1e1bf888a3a43c2f77de892d60aeeed7c0dce5542b54c1d Jul 06 15:17:35.980147 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.980100 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-94md2" Jul 06 15:17:35.985062 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.985039 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9rvk6" Jul 06 15:17:35.987115 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:35.987095 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef54bfc9_8403_41f5_9199_8f5f71639838.slice/crio-fe07e2ce63e1ec93e5d9a950cf5e9809cfc2831d82f544084c0f8e9e7947cea4 WatchSource:0}: Error finding container fe07e2ce63e1ec93e5d9a950cf5e9809cfc2831d82f544084c0f8e9e7947cea4: Status 404 returned error can't find the container with id fe07e2ce63e1ec93e5d9a950cf5e9809cfc2831d82f544084c0f8e9e7947cea4 Jul 06 15:17:35.990446 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:35.990347 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jlwgn" Jul 06 15:17:35.994787 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:35.994767 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ebda45d_abe9_429e_a983_2907701d2e61.slice/crio-c5f168c47827d4095d02c71b8760f48d1a4b2f4a213b63d0e10d7ae56f55c480 WatchSource:0}: Error finding container c5f168c47827d4095d02c71b8760f48d1a4b2f4a213b63d0e10d7ae56f55c480: Status 404 returned error can't find the container with id c5f168c47827d4095d02c71b8760f48d1a4b2f4a213b63d0e10d7ae56f55c480 Jul 06 15:17:35.998780 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:35.998760 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70fa8e04_5b44_4fbc_b2fc_502965ab8355.slice/crio-e17abbf645100675b7e8a298f691732183624118089f2050306dd4f68b68226a WatchSource:0}: Error finding container e17abbf645100675b7e8a298f691732183624118089f2050306dd4f68b68226a: Status 404 returned error can't find the container with id e17abbf645100675b7e8a298f691732183624118089f2050306dd4f68b68226a Jul 06 15:17:36.009326 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:36.009305 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:17:36.017488 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:17:36.017461 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eca6180_d678_4729_ad6f_979a025af841.slice/crio-b1d74e3472eebd56220a8e9f884c4f76f5695089442d3bf72733b693d6bd0969 WatchSource:0}: Error finding container b1d74e3472eebd56220a8e9f884c4f76f5695089442d3bf72733b693d6bd0969: Status 404 returned error can't find the container with id b1d74e3472eebd56220a8e9f884c4f76f5695089442d3bf72733b693d6bd0969 Jul 06 15:17:36.019226 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:36.019208 2570 kuberuntime_manager.go:1459] "Unhandled Error" err="container ovn-controller start failed in pod ovnkube-node-9hxpw_openshift-ovn-kubernetes(3eca6180-d678-4729-ad6f-979a025af841): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Jul 06 15:17:36.020558 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:36.020541 2570 kuberuntime_manager.go:1459] "Unhandled Error" err="container kube-rbac-proxy-node start failed in pod ovnkube-node-9hxpw_openshift-ovn-kubernetes(3eca6180-d678-4729-ad6f-979a025af841): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Jul 06 15:17:36.023595 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:36.023542 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72c58cd87dea47bcff20bc07f2a9a5def3716e1a5b3d3e46ddcca8b49e9abb94\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-node\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-ovn-metrics\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:52d26498fbce9f8ab56799e6e6f1920bc238d2d31f38945eab91ebc5f68b43dc\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"northd\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72c58cd87dea47bcff20bc07f2a9a5def3716e1a5b3d3e46ddcca8b49e9abb94\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"nbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72c58cd87dea47bcff20bc07f2a9a5def3716e1a5b3d3e46ddcca8b49e9abb94\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"sbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72c58cd87dea47bcff20bc07f2a9a5def3716e1a5b3d3e46ddcca8b49e9abb94\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovnkube-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72c58cd87dea47bcff20bc07f2a9a5def3716e1a5b3d3e46ddcca8b49e9abb94\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" podUID="3eca6180-d678-4729-ad6f-979a025af841" Jul 06 15:17:36.160402 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:36.160318 2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 15:17:36.182681 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:36.182654 2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 15:17:36.212519 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:36.212473 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b-metrics-certs\") pod \"network-metrics-daemon-6x68d\" (UID: \"5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b\") " pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:17:36.212674 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:36.212653 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:17:36.212758 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:36.212731 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b-metrics-certs podName:5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b nodeName:}" failed. No retries permitted until 2026-07-06 15:17:37.212709268 +0000 UTC m=+3.826457784 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b-metrics-certs") pod "network-metrics-daemon-6x68d" (UID: "5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:17:36.312920 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:36.312877 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkznd\" (UniqueName: \"kubernetes.io/projected/e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac-kube-api-access-wkznd\") pod \"network-check-target-v9nsl\" (UID: \"e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac\") " pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:17:36.312920 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:36.312924 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9aaa45f1-1a99-4e94-96f3-a12ddbe62e83-original-pull-secret\") pod \"global-pull-secret-syncer-l77mc\" (UID: \"9aaa45f1-1a99-4e94-96f3-a12ddbe62e83\") " pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:17:36.313129 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:36.313044 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 15:17:36.313129 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:36.313065 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 15:17:36.313129 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:36.313082 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 15:17:36.313129 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:36.313096 2570 projected.go:196] Error preparing data for projected volume kube-api-access-wkznd for pod openshift-network-diagnostics/network-check-target-v9nsl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:17:36.313129 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:36.313113 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aaa45f1-1a99-4e94-96f3-a12ddbe62e83-original-pull-secret podName:9aaa45f1-1a99-4e94-96f3-a12ddbe62e83 nodeName:}" failed. No retries permitted until 2026-07-06 15:17:37.313095172 +0000 UTC m=+3.926843689 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9aaa45f1-1a99-4e94-96f3-a12ddbe62e83-original-pull-secret") pod "global-pull-secret-syncer-l77mc" (UID: "9aaa45f1-1a99-4e94-96f3-a12ddbe62e83") : object "kube-system"/"original-pull-secret" not registered Jul 06 15:17:36.313298 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:36.313134 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac-kube-api-access-wkznd podName:e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac nodeName:}" failed. No retries permitted until 2026-07-06 15:17:37.31312377 +0000 UTC m=+3.926872296 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wkznd" (UniqueName: "kubernetes.io/projected/e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac-kube-api-access-wkznd") pod "network-check-target-v9nsl" (UID: "e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:17:36.438966 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:36.438875 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 15:17:36.647721 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:36.647623 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 15:12:35 +0000 UTC" deadline="2028-01-17 03:44:19.818902544 +0000 UTC" Jul 06 15:17:36.647721 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:36.647665 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13428h26m43.171242476s" Jul 06 15:17:36.797559 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:36.796824 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:17:36.797559 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:36.796957 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9nsl" podUID="e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac" Jul 06 15:17:36.808328 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:36.806252 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" event={"ID":"3eca6180-d678-4729-ad6f-979a025af841","Type":"ContainerStarted","Data":"b1d74e3472eebd56220a8e9f884c4f76f5695089442d3bf72733b693d6bd0969"} Jul 06 15:17:36.816432 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:36.816156 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jlwgn" event={"ID":"70fa8e04-5b44-4fbc-b2fc-502965ab8355","Type":"ContainerStarted","Data":"e17abbf645100675b7e8a298f691732183624118089f2050306dd4f68b68226a"} Jul 06 15:17:36.820402 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:36.820313 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-94md2" event={"ID":"ef54bfc9-8403-41f5-9199-8f5f71639838","Type":"ContainerStarted","Data":"fe07e2ce63e1ec93e5d9a950cf5e9809cfc2831d82f544084c0f8e9e7947cea4"} Jul 06 15:17:36.822621 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:36.822456 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72c58cd87dea47bcff20bc07f2a9a5def3716e1a5b3d3e46ddcca8b49e9abb94\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72c58cd87dea47bcff20bc07f2a9a5def3716e1a5b3d3e46ddcca8b49e9abb94\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-node\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:52d26498fbce9f8ab56799e6e6f1920bc238d2d31f38945eab91ebc5f68b43dc\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-ovn-metrics\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:52d26498fbce9f8ab56799e6e6f1920bc238d2d31f38945eab91ebc5f68b43dc\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"northd\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72c58cd87dea47bcff20bc07f2a9a5def3716e1a5b3d3e46ddcca8b49e9abb94\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"nbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72c58cd87dea47bcff20bc07f2a9a5def3716e1a5b3d3e46ddcca8b49e9abb94\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"sbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72c58cd87dea47bcff20bc07f2a9a5def3716e1a5b3d3e46ddcca8b49e9abb94\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovnkube-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72c58cd87dea47bcff20bc07f2a9a5def3716e1a5b3d3e46ddcca8b49e9abb94\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" podUID="3eca6180-d678-4729-ad6f-979a025af841" Jul 06 15:17:36.826540 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:36.826510 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fbvqs" event={"ID":"9a7b5582-0829-4212-8ff0-6379e5eb5306","Type":"ContainerStarted","Data":"f23b75c39e25d8a6a1e1bf888a3a43c2f77de892d60aeeed7c0dce5542b54c1d"} Jul 06 15:17:36.830151 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:36.830123 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-prvr8" event={"ID":"17759dc6-09ce-4dd1-b5e9-4076390ec9e1","Type":"ContainerStarted","Data":"343cc6b9ffac85be3a3658dfd0fa600deb44ed55ab29ba54204648d95026dce0"} Jul 06 15:17:36.835220 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:36.835111 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mgd8j" event={"ID":"65d43ca0-dc12-4e4f-8a7d-a42c0f10a651","Type":"ContainerStarted","Data":"f4e41c1969daa3f705a8e814cbabaf2ab18d84a03b15c23e43faf3f1340d0ba2"} Jul 06 15:17:36.838862 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:36.838801 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-108.ec2.internal" event={"ID":"6f497d407b6748f0adb639ce1a9ea113","Type":"ContainerStarted","Data":"01e0a7659bdb795df5dbf77cf15eedcf1749aa15231c37442de15224fc684176"} Jul 06 15:17:36.844785 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:36.844759 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-66jdx" event={"ID":"ab57a97c-0e04-4482-9444-836a8b6a2f37","Type":"ContainerStarted","Data":"908d607f90e1b92ffbb658d612d7d52eb7581a78146fa350d1e71068b4b71238"} Jul 06 15:17:36.848314 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:36.848041 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9rvk6" event={"ID":"7ebda45d-abe9-429e-a983-2907701d2e61","Type":"ContainerStarted","Data":"c5f168c47827d4095d02c71b8760f48d1a4b2f4a213b63d0e10d7ae56f55c480"} Jul 06 15:17:36.852654 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:36.852571 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-108.ec2.internal" event={"ID":"84b26db8de42be2719159be061d921ee","Type":"ContainerStarted","Data":"02b53052008b3b2af10b25c9ec07647815299ec3dfb106211f63802ce32b1e3e"} Jul 06 15:17:36.858137 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:36.858109 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skmhv" event={"ID":"e8405516-7f84-421a-a858-e1d77bd5ab96","Type":"ContainerStarted","Data":"94d6ed938f664d67b801a41e32a336d7b10aa6d70644a7109b8e49645852b6b7"} Jul 06 15:17:37.221903 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:37.221761 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b-metrics-certs\") pod \"network-metrics-daemon-6x68d\" (UID: \"5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b\") " pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:17:37.222109 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:37.221939 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:17:37.222109 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:37.222015 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b-metrics-certs podName:5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b nodeName:}" failed. No retries permitted until 2026-07-06 15:17:39.221993542 +0000 UTC m=+5.835742064 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b-metrics-certs") pod "network-metrics-daemon-6x68d" (UID: "5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:17:37.322790 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:37.322746 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkznd\" (UniqueName: \"kubernetes.io/projected/e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac-kube-api-access-wkznd\") pod \"network-check-target-v9nsl\" (UID: \"e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac\") " pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:17:37.322992 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:37.322808 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9aaa45f1-1a99-4e94-96f3-a12ddbe62e83-original-pull-secret\") pod \"global-pull-secret-syncer-l77mc\" (UID: \"9aaa45f1-1a99-4e94-96f3-a12ddbe62e83\") " pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:17:37.322992 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:37.322981 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 15:17:37.323112 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:37.323052 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aaa45f1-1a99-4e94-96f3-a12ddbe62e83-original-pull-secret podName:9aaa45f1-1a99-4e94-96f3-a12ddbe62e83 nodeName:}" failed. No retries permitted until 2026-07-06 15:17:39.32303081 +0000 UTC m=+5.936779328 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9aaa45f1-1a99-4e94-96f3-a12ddbe62e83-original-pull-secret") pod "global-pull-secret-syncer-l77mc" (UID: "9aaa45f1-1a99-4e94-96f3-a12ddbe62e83") : object "kube-system"/"original-pull-secret" not registered Jul 06 15:17:37.323505 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:37.323484 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 15:17:37.323631 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:37.323512 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 15:17:37.323631 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:37.323527 2570 projected.go:196] Error preparing data for projected volume kube-api-access-wkznd for pod openshift-network-diagnostics/network-check-target-v9nsl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:17:37.323631 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:37.323576 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac-kube-api-access-wkznd podName:e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac nodeName:}" failed. No retries permitted until 2026-07-06 15:17:39.32356005 +0000 UTC m=+5.937308571 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wkznd" (UniqueName: "kubernetes.io/projected/e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac-kube-api-access-wkznd") pod "network-check-target-v9nsl" (UID: "e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:17:37.648878 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:37.648760 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 15:12:35 +0000 UTC" deadline="2028-01-06 11:10:12.683150982 +0000 UTC" Jul 06 15:17:37.648878 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:37.648826 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13171h52m35.034328743s" Jul 06 15:17:37.793329 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:37.793292 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:17:37.793530 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:37.793457 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l77mc" podUID="9aaa45f1-1a99-4e94-96f3-a12ddbe62e83" Jul 06 15:17:37.793964 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:37.793943 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:17:37.794097 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:37.794073 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x68d" podUID="5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b" Jul 06 15:17:38.795675 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:38.795637 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:17:38.796286 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:38.795809 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9nsl" podUID="e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac" Jul 06 15:17:39.240788 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:39.240639 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b-metrics-certs\") pod \"network-metrics-daemon-6x68d\" (UID: \"5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b\") " pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:17:39.240984 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:39.240809 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:17:39.240984 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:39.240885 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b-metrics-certs podName:5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b nodeName:}" failed. No retries permitted until 2026-07-06 15:17:43.240864808 +0000 UTC m=+9.854613324 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b-metrics-certs") pod "network-metrics-daemon-6x68d" (UID: "5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:17:39.342868 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:39.342021 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkznd\" (UniqueName: \"kubernetes.io/projected/e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac-kube-api-access-wkznd\") pod \"network-check-target-v9nsl\" (UID: \"e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac\") " pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:17:39.342868 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:39.342078 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9aaa45f1-1a99-4e94-96f3-a12ddbe62e83-original-pull-secret\") pod \"global-pull-secret-syncer-l77mc\" (UID: \"9aaa45f1-1a99-4e94-96f3-a12ddbe62e83\") " pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:17:39.342868 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:39.342279 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 15:17:39.342868 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:39.342351 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aaa45f1-1a99-4e94-96f3-a12ddbe62e83-original-pull-secret podName:9aaa45f1-1a99-4e94-96f3-a12ddbe62e83 nodeName:}" failed. No retries permitted until 2026-07-06 15:17:43.342321896 +0000 UTC m=+9.956070413 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9aaa45f1-1a99-4e94-96f3-a12ddbe62e83-original-pull-secret") pod "global-pull-secret-syncer-l77mc" (UID: "9aaa45f1-1a99-4e94-96f3-a12ddbe62e83") : object "kube-system"/"original-pull-secret" not registered Jul 06 15:17:39.342868 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:39.342833 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 15:17:39.342868 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:39.342856 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 15:17:39.342868 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:39.342869 2570 projected.go:196] Error preparing data for projected volume kube-api-access-wkznd for pod openshift-network-diagnostics/network-check-target-v9nsl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:17:39.343322 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:39.342918 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac-kube-api-access-wkznd podName:e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac nodeName:}" failed. No retries permitted until 2026-07-06 15:17:43.342902099 +0000 UTC m=+9.956650631 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wkznd" (UniqueName: "kubernetes.io/projected/e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac-kube-api-access-wkznd") pod "network-check-target-v9nsl" (UID: "e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:17:39.793847 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:39.793808 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:17:39.793847 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:39.793839 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:17:39.794094 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:39.793919 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l77mc" podUID="9aaa45f1-1a99-4e94-96f3-a12ddbe62e83" Jul 06 15:17:39.794094 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:39.794074 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x68d" podUID="5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b" Jul 06 15:17:40.794092 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:40.793601 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:17:40.794092 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:40.793727 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9nsl" podUID="e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac" Jul 06 15:17:41.794117 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:41.794026 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:17:41.794556 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:41.794026 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:17:41.794556 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:41.794128 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l77mc" podUID="9aaa45f1-1a99-4e94-96f3-a12ddbe62e83" Jul 06 15:17:41.794556 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:41.794210 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x68d" podUID="5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b" Jul 06 15:17:42.793829 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:42.793792 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:17:42.794018 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:42.793934 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9nsl" podUID="e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac" Jul 06 15:17:43.274127 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:43.274095 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b-metrics-certs\") pod \"network-metrics-daemon-6x68d\" (UID: \"5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b\") " pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:17:43.274562 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:43.274266 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:17:43.274562 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:43.274344 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b-metrics-certs podName:5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b nodeName:}" failed. No retries permitted until 2026-07-06 15:17:51.274322495 +0000 UTC m=+17.888071019 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b-metrics-certs") pod "network-metrics-daemon-6x68d" (UID: "5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:17:43.375276 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:43.375245 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkznd\" (UniqueName: \"kubernetes.io/projected/e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac-kube-api-access-wkznd\") pod \"network-check-target-v9nsl\" (UID: \"e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac\") " pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:17:43.375276 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:43.375286 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9aaa45f1-1a99-4e94-96f3-a12ddbe62e83-original-pull-secret\") pod \"global-pull-secret-syncer-l77mc\" (UID: \"9aaa45f1-1a99-4e94-96f3-a12ddbe62e83\") " pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:17:43.375554 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:43.375395 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 15:17:43.375554 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:43.375423 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 15:17:43.375554 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:43.375444 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 15:17:43.375554 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:43.375455 2570 projected.go:196] Error preparing data for projected volume kube-api-access-wkznd for pod openshift-network-diagnostics/network-check-target-v9nsl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:17:43.375554 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:43.375455 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aaa45f1-1a99-4e94-96f3-a12ddbe62e83-original-pull-secret podName:9aaa45f1-1a99-4e94-96f3-a12ddbe62e83 nodeName:}" failed. No retries permitted until 2026-07-06 15:17:51.375438693 +0000 UTC m=+17.989187207 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9aaa45f1-1a99-4e94-96f3-a12ddbe62e83-original-pull-secret") pod "global-pull-secret-syncer-l77mc" (UID: "9aaa45f1-1a99-4e94-96f3-a12ddbe62e83") : object "kube-system"/"original-pull-secret" not registered Jul 06 15:17:43.375554 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:43.375526 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac-kube-api-access-wkznd podName:e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac nodeName:}" failed. No retries permitted until 2026-07-06 15:17:51.375512464 +0000 UTC m=+17.989260983 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wkznd" (UniqueName: "kubernetes.io/projected/e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac-kube-api-access-wkznd") pod "network-check-target-v9nsl" (UID: "e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:17:43.793463 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:43.793425 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:17:43.793631 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:43.793561 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l77mc" podUID="9aaa45f1-1a99-4e94-96f3-a12ddbe62e83" Jul 06 15:17:43.793631 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:43.793620 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:17:43.793765 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:43.793747 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x68d" podUID="5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b" Jul 06 15:17:44.794317 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:44.794280 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:17:44.794742 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:44.794399 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9nsl" podUID="e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac" Jul 06 15:17:45.793967 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:45.793781 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:17:45.794065 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:45.793783 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:17:45.794065 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:45.794049 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l77mc" podUID="9aaa45f1-1a99-4e94-96f3-a12ddbe62e83" Jul 06 15:17:45.794244 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:45.794217 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x68d" podUID="5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b" Jul 06 15:17:45.884546 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:45.884451 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fbvqs" event={"ID":"9a7b5582-0829-4212-8ff0-6379e5eb5306","Type":"ContainerStarted","Data":"0588832ea0665da5a250247fe39827964ec6b8d76b207b1849c120e9c5b9c9a5"} Jul 06 15:17:45.885882 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:45.885855 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-prvr8" event={"ID":"17759dc6-09ce-4dd1-b5e9-4076390ec9e1","Type":"ContainerStarted","Data":"1d6ce351ed89f59c8bea4f61e2010c59aaa5a5a9cc6f8f2c5e16471c72f5f7c1"} Jul 06 15:17:45.887207 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:45.887187 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mgd8j" event={"ID":"65d43ca0-dc12-4e4f-8a7d-a42c0f10a651","Type":"ContainerStarted","Data":"bc04532ddab158423465a11ffd21f573b4e661c617e52fdccdbca464a3ddad8e"} Jul 06 15:17:45.888589 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:45.888568 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-108.ec2.internal" event={"ID":"6f497d407b6748f0adb639ce1a9ea113","Type":"ContainerStarted","Data":"b043810b2db5ee1c700b0beca5a4d8c6e2165196a67688970c5f142826d54958"} Jul 06 15:17:45.889808 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:45.889789 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-66jdx" event={"ID":"ab57a97c-0e04-4482-9444-836a8b6a2f37","Type":"ContainerStarted","Data":"7cc6d70466e967cd6dd4d13dfe10755af43fcb8c907b6ff1a0566fca57efabdc"} Jul 06 15:17:45.891415 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:45.891389 2570 generic.go:356] "Generic (PLEG): container finished" podID="7ebda45d-abe9-429e-a983-2907701d2e61" containerID="7f6bb1b60613aa79b95089c89d86da6e85c96138abb376ab34998431a9381bd0" exitCode=0 Jul 06 15:17:45.891525 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:45.891474 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9rvk6" event={"ID":"7ebda45d-abe9-429e-a983-2907701d2e61","Type":"ContainerDied","Data":"7f6bb1b60613aa79b95089c89d86da6e85c96138abb376ab34998431a9381bd0"} Jul 06 15:17:45.893035 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:45.893013 2570 generic.go:356] "Generic (PLEG): container finished" podID="84b26db8de42be2719159be061d921ee" containerID="6b226c7612587075523f15b446cceded41fa84f27bfa93ae4ad97525c1717c3f" exitCode=0 Jul 06 15:17:45.893134 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:45.893101 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-108.ec2.internal" event={"ID":"84b26db8de42be2719159be061d921ee","Type":"ContainerDied","Data":"6b226c7612587075523f15b446cceded41fa84f27bfa93ae4ad97525c1717c3f"} Jul 06 15:17:45.894846 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:45.894826 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skmhv" event={"ID":"e8405516-7f84-421a-a858-e1d77bd5ab96","Type":"ContainerStarted","Data":"2f0dd52351a72f94e667e64c72233864ef56b2b9ae342c9e4b246da21b0e98ef"} Jul 06 15:17:45.900678 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:45.900642 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fbvqs" podStartSLOduration=1.544244213 podStartE2EDuration="10.900630167s" totalImagesPullingTime="9.356385954s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:17:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:17:45.900041776 +0000 UTC m=+12.513790314" watchObservedRunningTime="2026-07-06 15:17:45.900630167 +0000 UTC m=+12.514378703" Jul 06 15:17:45.937073 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:45.937013 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-108.ec2.internal" podStartSLOduration=10.936994714 podStartE2EDuration="10.936994714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:17:45.936868327 +0000 UTC m=+12.550616864" watchObservedRunningTime="2026-07-06 15:17:45.936994714 +0000 UTC m=+12.550743251" Jul 06 15:17:45.969553 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:45.969487 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-66jdx" podStartSLOduration=1.58808726 podStartE2EDuration="10.969469512s" totalImagesPullingTime="9.381382252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:17:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:17:45.96902126 +0000 UTC m=+12.582769795" watchObservedRunningTime="2026-07-06 15:17:45.969469512 +0000 UTC m=+12.583218050" Jul 06 15:17:46.003454 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:46.003404 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mgd8j" podStartSLOduration=1.616705109 podStartE2EDuration="11.003388363s" totalImagesPullingTime="9.386683254s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:17:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:17:45.983066101 +0000 UTC m=+12.596814638" watchObservedRunningTime="2026-07-06 15:17:46.003388363 +0000 UTC m=+12.617136900" Jul 06 15:17:46.082797 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:46.082748 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:17:46.794199 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:46.794162 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:17:46.794371 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:46.794290 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9nsl" podUID="e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac" Jul 06 15:17:46.898119 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:46.898078 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jlwgn" event={"ID":"70fa8e04-5b44-4fbc-b2fc-502965ab8355","Type":"ContainerStarted","Data":"113f6c35ddd2ede41ebf8ee1d1a62c5d8e66a3d7c1124e2da982c81a41795382"} Jul 06 15:17:46.916758 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:46.916702 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-prvr8" podStartSLOduration=2.5134455449999997 podStartE2EDuration="11.916684774s" totalImagesPullingTime="9.403239229s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:17:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:17:46.00305098 +0000 UTC m=+12.616799515" watchObservedRunningTime="2026-07-06 15:17:46.916684774 +0000 UTC m=+13.530433310" Jul 06 15:17:46.917146 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:46.917107 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jlwgn" podStartSLOduration=2.581289098 podStartE2EDuration="11.917096356s" totalImagesPullingTime="9.335807258s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:17:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:17:46.916641154 +0000 UTC m=+13.530389690" watchObservedRunningTime="2026-07-06 15:17:46.917096356 +0000 UTC m=+13.530844919" Jul 06 15:17:47.042656 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:47.042524 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:17:47.145998 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:47.145952 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:17:47.226824 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:47.226782 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:17:47.794217 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:47.794182 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:17:47.794399 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:47.794188 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:17:47.794399 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:47.794289 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l77mc" podUID="9aaa45f1-1a99-4e94-96f3-a12ddbe62e83" Jul 06 15:17:47.794490 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:47.794404 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x68d" podUID="5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b" Jul 06 15:17:48.128041 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:48.128003 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:17:48.227133 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:48.227055 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:17:48.793633 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:48.793594 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:17:48.793818 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:48.793721 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9nsl" podUID="e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac" Jul 06 15:17:49.738327 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:49.738151 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:17:49.793208 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:49.793173 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:17:49.793362 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:49.793173 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:17:49.793362 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:49.793282 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l77mc" podUID="9aaa45f1-1a99-4e94-96f3-a12ddbe62e83" Jul 06 15:17:49.793434 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:49.793394 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x68d" podUID="5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b" Jul 06 15:17:49.802326 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:49.802297 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:17:49.905040 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:49.905007 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-108.ec2.internal" event={"ID":"84b26db8de42be2719159be061d921ee","Type":"ContainerStarted","Data":"c884e051f03392cbfeaaf35d2df632e675899edb01473443202750e242b188ab"} Jul 06 15:17:49.924599 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:49.924522 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-108.ec2.internal" podStartSLOduration=14.924488585 podStartE2EDuration="14.924488585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:17:49.924009398 +0000 UTC m=+16.537757935" watchObservedRunningTime="2026-07-06 15:17:49.924488585 +0000 UTC m=+16.538237123" Jul 06 15:17:50.173858 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:50.173569 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-66jdx" Jul 06 15:17:50.174970 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:50.174948 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-66jdx" Jul 06 15:17:50.463982 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:50.463941 2570 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 15:17:50.661610 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:50.661490 2570 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T15:17:50.463956012Z","UUID":"454cd320-3a9c-4960-ba65-f9458da17adc","Handler":null,"Name":"","Endpoint":""} Jul 06 15:17:50.663176 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:50.663152 2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 15:17:50.663310 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:50.663194 2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 15:17:50.793970 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:50.793893 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:17:50.794723 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:50.794018 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9nsl" podUID="e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac" Jul 06 15:17:50.909236 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:50.909195 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-94md2" event={"ID":"ef54bfc9-8403-41f5-9199-8f5f71639838","Type":"ContainerStarted","Data":"a57a2baaa7c7f004ae1c52f7932cad10ef64b9689fe131f0bb9e87461e9ce471"} Jul 06 15:17:50.911380 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:50.911340 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skmhv" event={"ID":"e8405516-7f84-421a-a858-e1d77bd5ab96","Type":"ContainerStarted","Data":"ab4a039c8790012ef31e7e15756b072ca267c563ef2d90ad23be976ef5524595"} Jul 06 15:17:50.911722 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:50.911660 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-66jdx" Jul 06 15:17:50.912638 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:50.912618 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-66jdx" Jul 06 15:17:50.930728 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:50.930598 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-94md2" podStartSLOduration=1.917118696 podStartE2EDuration="15.930579334s" totalImagesPullingTime="14.013460638s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:17:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:17:50.930001071 +0000 UTC m=+17.543749607" watchObservedRunningTime="2026-07-06 15:17:50.930579334 +0000 UTC m=+17.544327873" Jul 06 15:17:51.013129 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:51.013101 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:17:51.115664 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:51.115633 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:17:51.215370 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:51.215324 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:17:51.338284 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:51.338248 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b-metrics-certs\") pod \"network-metrics-daemon-6x68d\" (UID: \"5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b\") " pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:17:51.338595 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:51.338492 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:17:51.338695 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:51.338672 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b-metrics-certs podName:5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b nodeName:}" failed. No retries permitted until 2026-07-06 15:18:07.338651291 +0000 UTC m=+33.952399825 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b-metrics-certs") pod "network-metrics-daemon-6x68d" (UID: "5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:17:51.439606 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:51.439524 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkznd\" (UniqueName: \"kubernetes.io/projected/e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac-kube-api-access-wkznd\") pod \"network-check-target-v9nsl\" (UID: \"e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac\") " pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:17:51.439606 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:51.439578 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9aaa45f1-1a99-4e94-96f3-a12ddbe62e83-original-pull-secret\") pod \"global-pull-secret-syncer-l77mc\" (UID: \"9aaa45f1-1a99-4e94-96f3-a12ddbe62e83\") " pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:17:51.439805 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:51.439703 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 15:17:51.439805 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:51.439714 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 15:17:51.439805 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:51.439737 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 15:17:51.439805 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:51.439749 2570 projected.go:196] Error preparing data for projected volume kube-api-access-wkznd for pod openshift-network-diagnostics/network-check-target-v9nsl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:17:51.439805 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:51.439767 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aaa45f1-1a99-4e94-96f3-a12ddbe62e83-original-pull-secret podName:9aaa45f1-1a99-4e94-96f3-a12ddbe62e83 nodeName:}" failed. No retries permitted until 2026-07-06 15:18:07.439747789 +0000 UTC m=+34.053496309 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9aaa45f1-1a99-4e94-96f3-a12ddbe62e83-original-pull-secret") pod "global-pull-secret-syncer-l77mc" (UID: "9aaa45f1-1a99-4e94-96f3-a12ddbe62e83") : object "kube-system"/"original-pull-secret" not registered Jul 06 15:17:51.439805 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:51.439800 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac-kube-api-access-wkznd podName:e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac nodeName:}" failed. No retries permitted until 2026-07-06 15:18:07.439783091 +0000 UTC m=+34.053531611 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wkznd" (UniqueName: "kubernetes.io/projected/e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac-kube-api-access-wkznd") pod "network-check-target-v9nsl" (UID: "e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:17:51.794406 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:51.794063 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:17:51.794406 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:51.794130 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:17:51.794876 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:51.794423 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l77mc" podUID="9aaa45f1-1a99-4e94-96f3-a12ddbe62e83" Jul 06 15:17:51.794876 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:51.794491 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x68d" podUID="5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b" Jul 06 15:17:51.915449 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:51.915411 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skmhv" event={"ID":"e8405516-7f84-421a-a858-e1d77bd5ab96","Type":"ContainerStarted","Data":"4a39f1d04d28ebd1e51051f38194b75fa84a604369ba60b879279e7f303b12c3"} Jul 06 15:17:52.794156 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:52.794118 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:17:52.794366 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:52.794255 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9nsl" podUID="e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac" Jul 06 15:17:53.793677 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:53.793635 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:17:53.794167 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:53.793641 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:17:53.794167 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:53.793763 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l77mc" podUID="9aaa45f1-1a99-4e94-96f3-a12ddbe62e83" Jul 06 15:17:53.794167 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:53.793850 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x68d" podUID="5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b" Jul 06 15:17:54.794650 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:54.794616 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:17:54.795111 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:54.794729 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9nsl" podUID="e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac" Jul 06 15:17:55.625373 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:55.625334 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:17:55.706830 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:55.706795 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:17:55.794167 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:55.794090 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:17:55.794326 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:55.794221 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l77mc" podUID="9aaa45f1-1a99-4e94-96f3-a12ddbe62e83" Jul 06 15:17:55.794326 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:55.794288 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:17:55.794446 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:55.794396 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x68d" podUID="5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b" Jul 06 15:17:55.927640 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:55.927549 2570 generic.go:356] "Generic (PLEG): container finished" podID="7ebda45d-abe9-429e-a983-2907701d2e61" containerID="ade56c2c8336a77f024776dd165c2c33bc132d9d64b9e909fe2aea1aca72ad7b" exitCode=0 Jul 06 15:17:55.927640 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:55.927598 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9rvk6" event={"ID":"7ebda45d-abe9-429e-a983-2907701d2e61","Type":"ContainerDied","Data":"ade56c2c8336a77f024776dd165c2c33bc132d9d64b9e909fe2aea1aca72ad7b"} Jul 06 15:17:55.962449 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:55.960720 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skmhv" podStartSLOduration=5.841266568 podStartE2EDuration="20.960699308s" totalImagesPullingTime="15.11943274s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:17:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:17:51.941778422 +0000 UTC m=+18.555526984" watchObservedRunningTime="2026-07-06 15:17:55.960699308 +0000 UTC m=+22.574447844" Jul 06 15:17:56.793246 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:56.793223 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:17:56.793386 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:56.793322 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9nsl" podUID="e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac" Jul 06 15:17:56.931825 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:56.931790 2570 generic.go:356] "Generic (PLEG): container finished" podID="7ebda45d-abe9-429e-a983-2907701d2e61" containerID="cc40194c17901c191f93d74276e83b647289658e3f08b1b16a1fe10a73d9d8b5" exitCode=0 Jul 06 15:17:56.931825 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:56.931827 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9rvk6" event={"ID":"7ebda45d-abe9-429e-a983-2907701d2e61","Type":"ContainerDied","Data":"cc40194c17901c191f93d74276e83b647289658e3f08b1b16a1fe10a73d9d8b5"} Jul 06 15:17:57.794003 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:57.793713 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:17:57.794003 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:57.793864 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x68d" podUID="5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b" Jul 06 15:17:57.794003 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:57.793911 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:17:57.794264 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:57.794027 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l77mc" podUID="9aaa45f1-1a99-4e94-96f3-a12ddbe62e83" Jul 06 15:17:57.936247 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:57.936217 2570 generic.go:356] "Generic (PLEG): container finished" podID="7ebda45d-abe9-429e-a983-2907701d2e61" containerID="4bc493a84b5acd6d186e04118a8b8082a577f588c177af3f473ac4492c06a4b1" exitCode=0 Jul 06 15:17:57.936713 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:57.936269 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9rvk6" event={"ID":"7ebda45d-abe9-429e-a983-2907701d2e61","Type":"ContainerDied","Data":"4bc493a84b5acd6d186e04118a8b8082a577f588c177af3f473ac4492c06a4b1"} Jul 06 15:17:58.793366 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:58.793334 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:17:58.793571 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:58.793472 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9nsl" podUID="e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac" Jul 06 15:17:59.794160 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:59.794119 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:17:59.794160 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:17:59.794165 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:17:59.794729 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:59.794266 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x68d" podUID="5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b" Jul 06 15:17:59.794729 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:17:59.794376 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l77mc" podUID="9aaa45f1-1a99-4e94-96f3-a12ddbe62e83" Jul 06 15:18:00.156801 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:00.156767 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:18:00.231797 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:00.231763 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:18:00.793545 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:00.793496 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:18:00.793715 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:00.793645 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9nsl" podUID="e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac" Jul 06 15:18:01.706546 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:01.706413 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:18:01.794132 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:01.794099 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:18:01.794335 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:01.794097 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:18:01.794335 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:01.794255 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x68d" podUID="5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b" Jul 06 15:18:01.794335 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:01.794323 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l77mc" podUID="9aaa45f1-1a99-4e94-96f3-a12ddbe62e83" Jul 06 15:18:01.807622 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:01.807590 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:18:02.794254 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:02.794222 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:18:02.794762 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:02.794351 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9nsl" podUID="e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac" Jul 06 15:18:03.793249 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:03.793221 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:18:03.793450 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:03.793263 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:18:03.793450 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:03.793344 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l77mc" podUID="9aaa45f1-1a99-4e94-96f3-a12ddbe62e83" Jul 06 15:18:03.793589 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:03.793466 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x68d" podUID="5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b" Jul 06 15:18:04.794311 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:04.794272 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:18:04.794802 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:04.794386 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9nsl" podUID="e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac" Jul 06 15:18:05.794005 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:05.793969 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:18:05.794005 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:05.793991 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:18:05.794241 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:05.794088 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l77mc" podUID="9aaa45f1-1a99-4e94-96f3-a12ddbe62e83" Jul 06 15:18:05.794305 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:05.794248 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x68d" podUID="5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b" Jul 06 15:18:06.793881 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:06.793856 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:18:06.794208 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:06.793969 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9nsl" podUID="e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac" Jul 06 15:18:06.957487 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:06.957374 2570 generic.go:356] "Generic (PLEG): container finished" podID="7ebda45d-abe9-429e-a983-2907701d2e61" containerID="e7badbe579d66a46aaac90ca130b1abcaf5867a0b133c5e43667ee04b493c2c0" exitCode=0 Jul 06 15:18:06.957601 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:06.957455 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9rvk6" event={"ID":"7ebda45d-abe9-429e-a983-2907701d2e61","Type":"ContainerDied","Data":"e7badbe579d66a46aaac90ca130b1abcaf5867a0b133c5e43667ee04b493c2c0"} Jul 06 15:18:06.959677 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:06.959658 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hxpw_3eca6180-d678-4729-ad6f-979a025af841/ovn-acl-logging/0.log" Jul 06 15:18:06.960031 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:06.960009 2570 generic.go:356] "Generic (PLEG): container finished" podID="3eca6180-d678-4729-ad6f-979a025af841" containerID="7a9dd169e6e87c24ff2c765c474bee046ede572820cda7e52e0d384fd33ed8ec" exitCode=1 Jul 06 15:18:06.960121 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:06.960063 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" event={"ID":"3eca6180-d678-4729-ad6f-979a025af841","Type":"ContainerDied","Data":"7a9dd169e6e87c24ff2c765c474bee046ede572820cda7e52e0d384fd33ed8ec"} Jul 06 15:18:06.960179 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:06.960142 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" event={"ID":"3eca6180-d678-4729-ad6f-979a025af841","Type":"ContainerStarted","Data":"4d522db26fe5672c97a11e11ffb98a70ae5826df2b3778b37f52ad5703d45b84"} Jul 06 15:18:07.356204 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:07.356171 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b-metrics-certs\") pod \"network-metrics-daemon-6x68d\" (UID: \"5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b\") " pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:18:07.356358 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:07.356342 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:18:07.356432 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:07.356418 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b-metrics-certs podName:5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b nodeName:}" failed. No retries permitted until 2026-07-06 15:18:39.356396749 +0000 UTC m=+65.970145269 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b-metrics-certs") pod "network-metrics-daemon-6x68d" (UID: "5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:18:07.456731 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:07.456662 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkznd\" (UniqueName: \"kubernetes.io/projected/e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac-kube-api-access-wkznd\") pod \"network-check-target-v9nsl\" (UID: \"e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac\") " pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:18:07.456731 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:07.456695 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9aaa45f1-1a99-4e94-96f3-a12ddbe62e83-original-pull-secret\") pod \"global-pull-secret-syncer-l77mc\" (UID: \"9aaa45f1-1a99-4e94-96f3-a12ddbe62e83\") " pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:18:07.456905 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:07.456791 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 15:18:07.456905 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:07.456809 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 15:18:07.456905 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:07.456829 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 15:18:07.456905 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:07.456835 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aaa45f1-1a99-4e94-96f3-a12ddbe62e83-original-pull-secret podName:9aaa45f1-1a99-4e94-96f3-a12ddbe62e83 nodeName:}" failed. No retries permitted until 2026-07-06 15:18:39.456823252 +0000 UTC m=+66.070571767 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9aaa45f1-1a99-4e94-96f3-a12ddbe62e83-original-pull-secret") pod "global-pull-secret-syncer-l77mc" (UID: "9aaa45f1-1a99-4e94-96f3-a12ddbe62e83") : object "kube-system"/"original-pull-secret" not registered Jul 06 15:18:07.456905 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:07.456838 2570 projected.go:196] Error preparing data for projected volume kube-api-access-wkznd for pod openshift-network-diagnostics/network-check-target-v9nsl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:18:07.456905 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:07.456877 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac-kube-api-access-wkznd podName:e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac nodeName:}" failed. No retries permitted until 2026-07-06 15:18:39.45686516 +0000 UTC m=+66.070613677 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wkznd" (UniqueName: "kubernetes.io/projected/e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac-kube-api-access-wkznd") pod "network-check-target-v9nsl" (UID: "e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:18:07.793554 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:07.793454 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:18:07.793554 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:07.793462 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:18:07.793741 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:07.793574 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l77mc" podUID="9aaa45f1-1a99-4e94-96f3-a12ddbe62e83" Jul 06 15:18:07.793741 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:07.793630 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x68d" podUID="5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b" Jul 06 15:18:07.965003 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:07.964969 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hxpw_3eca6180-d678-4729-ad6f-979a025af841/ovn-acl-logging/0.log" Jul 06 15:18:07.965534 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:07.965440 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" event={"ID":"3eca6180-d678-4729-ad6f-979a025af841","Type":"ContainerStarted","Data":"5013239f06d20f8f2268ada3d72ff1cb0bb16324aa3ab979742cabbc374bc359"} Jul 06 15:18:07.965534 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:07.965519 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" event={"ID":"3eca6180-d678-4729-ad6f-979a025af841","Type":"ContainerStarted","Data":"fc96decc6415585df36bbcdf19deabc8d29659fc7c6ec73e07575332b44cbc7c"} Jul 06 15:18:07.965625 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:07.965568 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" event={"ID":"3eca6180-d678-4729-ad6f-979a025af841","Type":"ContainerStarted","Data":"734e97de0233cbc94eb31209d9dbbe622ca2be0e4c756d99babfea29a5137c47"} Jul 06 15:18:07.965625 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:07.965596 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" event={"ID":"3eca6180-d678-4729-ad6f-979a025af841","Type":"ContainerStarted","Data":"005f56285e86b236eff8fed6f92c492b441e96c2cf543b83179a4eaeeab7fd65"} Jul 06 15:18:07.968281 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:07.968256 2570 generic.go:356] "Generic (PLEG): container finished" podID="7ebda45d-abe9-429e-a983-2907701d2e61" containerID="9f6b8dec7b9251fd9dd5a523ba4602955ad5460fbf91e328da4523c658f0e576" exitCode=0 Jul 06 15:18:07.968368 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:07.968298 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9rvk6" event={"ID":"7ebda45d-abe9-429e-a983-2907701d2e61","Type":"ContainerDied","Data":"9f6b8dec7b9251fd9dd5a523ba4602955ad5460fbf91e328da4523c658f0e576"} Jul 06 15:18:08.794120 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:08.794088 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:18:08.794277 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:08.794193 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9nsl" podUID="e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac" Jul 06 15:18:08.973577 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:08.973535 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9rvk6" event={"ID":"7ebda45d-abe9-429e-a983-2907701d2e61","Type":"ContainerStarted","Data":"185576a79785c7f5b3cdefd0232b924662513a46c6cbc897c5a81b8bfb50a924"} Jul 06 15:18:09.001094 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:09.001041 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9rvk6" podStartSLOduration=5.02050586 podStartE2EDuration="34.00102807s" totalImagesPullingTime="28.98052221s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:17:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:18:09.000547354 +0000 UTC m=+35.614295890" watchObservedRunningTime="2026-07-06 15:18:09.00102807 +0000 UTC m=+35.614776606" Jul 06 15:18:09.128786 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:09.128759 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:18:09.227043 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:09.227008 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:18:09.793593 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:09.793493 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:18:09.793764 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:09.793512 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:18:09.793764 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:09.793637 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x68d" podUID="5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b" Jul 06 15:18:09.793764 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:09.793670 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l77mc" podUID="9aaa45f1-1a99-4e94-96f3-a12ddbe62e83" Jul 06 15:18:09.978461 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:09.978435 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hxpw_3eca6180-d678-4729-ad6f-979a025af841/ovn-acl-logging/0.log" Jul 06 15:18:09.978883 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:09.978844 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" event={"ID":"3eca6180-d678-4729-ad6f-979a025af841","Type":"ContainerStarted","Data":"40eef6fe9a9ad41f44207d934d7cf3fb6a0ef21758e76f22dc329418ba792eb4"} Jul 06 15:18:10.793884 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:10.793850 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:18:10.794073 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:10.793962 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9nsl" podUID="e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac" Jul 06 15:18:11.793629 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:11.793599 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:18:11.793629 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:11.793629 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:18:11.794214 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:11.793719 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l77mc" podUID="9aaa45f1-1a99-4e94-96f3-a12ddbe62e83" Jul 06 15:18:11.794214 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:11.793833 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x68d" podUID="5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b" Jul 06 15:18:11.986610 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:11.986405 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hxpw_3eca6180-d678-4729-ad6f-979a025af841/ovn-acl-logging/0.log" Jul 06 15:18:11.986960 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:11.986934 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" event={"ID":"3eca6180-d678-4729-ad6f-979a025af841","Type":"ContainerStarted","Data":"09fb15dc06af53a27e367bba2f3e77fe001f517cb4934f7ada9c9c9ddaacdbe9"} Jul 06 15:18:11.987248 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:11.987226 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:18:11.987360 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:11.987294 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:18:11.987440 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:11.987423 2570 scope.go:117] "RemoveContainer" containerID="7a9dd169e6e87c24ff2c765c474bee046ede572820cda7e52e0d384fd33ed8ec" Jul 06 15:18:12.006648 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:12.003718 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:18:12.793434 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:12.793396 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:18:12.793687 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:12.793532 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9nsl" podUID="e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac" Jul 06 15:18:12.992215 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:12.992189 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hxpw_3eca6180-d678-4729-ad6f-979a025af841/ovn-acl-logging/0.log" Jul 06 15:18:12.992535 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:12.992514 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" event={"ID":"3eca6180-d678-4729-ad6f-979a025af841","Type":"ContainerStarted","Data":"737aad9d7e5b9fc78daf1cc53a2e396d3e24b43e9c06418eb452658a3fa9aae7"} Jul 06 15:18:12.992711 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:12.992691 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:18:13.007518 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:13.007482 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:18:13.073697 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:13.073596 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" podStartSLOduration=7.504957546 podStartE2EDuration="38.073579938s" totalImagesPullingTime="30.568622392s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:17:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=2 observedRunningTime="2026-07-06 15:18:13.032590049 +0000 UTC m=+39.646338581" watchObservedRunningTime="2026-07-06 15:18:13.073579938 +0000 UTC m=+39.687328474" Jul 06 15:18:13.793844 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:13.793821 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:18:13.794166 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:13.793821 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:18:13.794166 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:13.793949 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x68d" podUID="5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b" Jul 06 15:18:13.794166 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:13.793989 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l77mc" podUID="9aaa45f1-1a99-4e94-96f3-a12ddbe62e83" Jul 06 15:18:13.989824 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:13.989793 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l77mc"] Jul 06 15:18:13.990478 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:13.990451 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6x68d"] Jul 06 15:18:13.990946 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:13.990925 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v9nsl"] Jul 06 15:18:13.991035 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:13.991023 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:18:13.991120 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:13.991101 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9nsl" podUID="e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac" Jul 06 15:18:13.995429 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:13.995409 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:18:13.995545 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:13.995513 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l77mc" podUID="9aaa45f1-1a99-4e94-96f3-a12ddbe62e83" Jul 06 15:18:13.995545 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:13.995409 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:18:13.995914 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:13.995885 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x68d" podUID="5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b" Jul 06 15:18:15.624551 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:15.624517 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:18:15.760146 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:15.760112 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:18:15.794066 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:15.794033 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:18:15.794223 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:15.794033 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:18:15.794268 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:15.794234 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x68d" podUID="5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b" Jul 06 15:18:15.794268 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:15.794136 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9nsl" podUID="e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac" Jul 06 15:18:15.794268 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:15.794034 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:18:15.794417 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:15.794318 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l77mc" podUID="9aaa45f1-1a99-4e94-96f3-a12ddbe62e83" Jul 06 15:18:17.005247 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:17.005213 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:18:17.088056 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:17.088021 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:18:17.794023 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:17.793995 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:18:17.794202 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:17.793998 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:18:17.794202 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:17.794103 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x68d" podUID="5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b" Jul 06 15:18:17.794202 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:17.794114 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:18:17.794202 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:17.794167 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9nsl" podUID="e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac" Jul 06 15:18:17.794342 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:17.794247 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l77mc" podUID="9aaa45f1-1a99-4e94-96f3-a12ddbe62e83" Jul 06 15:18:19.506010 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:19.505975 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:18:19.794045 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:19.793959 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:18:19.794203 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:19.793959 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:18:19.794203 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:19.794064 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l77mc" podUID="9aaa45f1-1a99-4e94-96f3-a12ddbe62e83" Jul 06 15:18:19.794203 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:19.794169 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6x68d" podUID="5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b" Jul 06 15:18:19.794203 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:19.793958 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:18:19.794426 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:19.794255 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9nsl" podUID="e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac" Jul 06 15:18:20.439187 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:20.439163 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-108.ec2.internal" event="NodeReady" Jul 06 15:18:20.439306 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:20.439300 2570 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 15:18:20.497533 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:20.497489 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8lh8v"] Jul 06 15:18:20.512163 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:20.512130 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-n5z5d"] Jul 06 15:18:20.512623 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:20.512276 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8lh8v" Jul 06 15:18:20.514972 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:20.514947 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 15:18:20.515127 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:20.515105 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 15:18:20.515233 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:20.515215 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8l7xg\"" Jul 06 15:18:20.522290 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:20.522265 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8lh8v"] Jul 06 15:18:20.522409 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:20.522309 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n5z5d"] Jul 06 15:18:20.522409 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:20.522381 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n5z5d" Jul 06 15:18:20.527368 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:20.527343 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 15:18:20.527466 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:20.527443 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 15:18:20.528164 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:20.528143 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-x9pws\"" Jul 06 15:18:20.528547 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:20.528527 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 15:18:20.652328 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:20.652233 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ce904e6-fad6-4078-ab8b-1a45fab198a9-tmp-dir\") pod \"dns-default-8lh8v\" (UID: \"1ce904e6-fad6-4078-ab8b-1a45fab198a9\") " pod="openshift-dns/dns-default-8lh8v" Jul 06 15:18:20.652328 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:20.652303 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1ce904e6-fad6-4078-ab8b-1a45fab198a9-config-volume\") pod \"dns-default-8lh8v\" (UID: \"1ce904e6-fad6-4078-ab8b-1a45fab198a9\") " pod="openshift-dns/dns-default-8lh8v" Jul 06 15:18:20.652328 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:20.652328 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr28r\" (UniqueName: \"kubernetes.io/projected/9da83319-4207-4990-ad20-a322c1f750ec-kube-api-access-lr28r\") pod \"ingress-canary-n5z5d\" (UID: \"9da83319-4207-4990-ad20-a322c1f750ec\") " pod="openshift-ingress-canary/ingress-canary-n5z5d" Jul 06 15:18:20.652583 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:20.652349 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shnlk\" (UniqueName: \"kubernetes.io/projected/1ce904e6-fad6-4078-ab8b-1a45fab198a9-kube-api-access-shnlk\") pod \"dns-default-8lh8v\" (UID: \"1ce904e6-fad6-4078-ab8b-1a45fab198a9\") " pod="openshift-dns/dns-default-8lh8v" Jul 06 15:18:20.652583 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:20.652378 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ce904e6-fad6-4078-ab8b-1a45fab198a9-metrics-tls\") pod \"dns-default-8lh8v\" (UID: \"1ce904e6-fad6-4078-ab8b-1a45fab198a9\") " pod="openshift-dns/dns-default-8lh8v" Jul 06 15:18:20.652583 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:20.652442 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9da83319-4207-4990-ad20-a322c1f750ec-cert\") pod \"ingress-canary-n5z5d\" (UID: \"9da83319-4207-4990-ad20-a322c1f750ec\") " pod="openshift-ingress-canary/ingress-canary-n5z5d" Jul 06 15:18:20.752751 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:20.752714 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9da83319-4207-4990-ad20-a322c1f750ec-cert\") pod \"ingress-canary-n5z5d\" (UID: \"9da83319-4207-4990-ad20-a322c1f750ec\") " pod="openshift-ingress-canary/ingress-canary-n5z5d" Jul 06 15:18:20.752905 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:20.752761 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ce904e6-fad6-4078-ab8b-1a45fab198a9-tmp-dir\") pod \"dns-default-8lh8v\" (UID: \"1ce904e6-fad6-4078-ab8b-1a45fab198a9\") " pod="openshift-dns/dns-default-8lh8v" Jul 06 15:18:20.752905 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:20.752820 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1ce904e6-fad6-4078-ab8b-1a45fab198a9-config-volume\") pod \"dns-default-8lh8v\" (UID: \"1ce904e6-fad6-4078-ab8b-1a45fab198a9\") " pod="openshift-dns/dns-default-8lh8v" Jul 06 15:18:20.752905 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:20.752848 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr28r\" (UniqueName: \"kubernetes.io/projected/9da83319-4207-4990-ad20-a322c1f750ec-kube-api-access-lr28r\") pod \"ingress-canary-n5z5d\" (UID: \"9da83319-4207-4990-ad20-a322c1f750ec\") " pod="openshift-ingress-canary/ingress-canary-n5z5d" Jul 06 15:18:20.752905 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:20.752878 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shnlk\" (UniqueName: \"kubernetes.io/projected/1ce904e6-fad6-4078-ab8b-1a45fab198a9-kube-api-access-shnlk\") pod \"dns-default-8lh8v\" (UID: \"1ce904e6-fad6-4078-ab8b-1a45fab198a9\") " pod="openshift-dns/dns-default-8lh8v" Jul 06 15:18:20.753102 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:20.752914 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ce904e6-fad6-4078-ab8b-1a45fab198a9-metrics-tls\") pod \"dns-default-8lh8v\" (UID: \"1ce904e6-fad6-4078-ab8b-1a45fab198a9\") " pod="openshift-dns/dns-default-8lh8v" Jul 06 15:18:20.753102 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:20.753009 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 15:18:20.753102 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:20.753084 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ce904e6-fad6-4078-ab8b-1a45fab198a9-metrics-tls podName:1ce904e6-fad6-4078-ab8b-1a45fab198a9 nodeName:}" failed. No retries permitted until 2026-07-06 15:18:21.253064568 +0000 UTC m=+47.866813104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1ce904e6-fad6-4078-ab8b-1a45fab198a9-metrics-tls") pod "dns-default-8lh8v" (UID: "1ce904e6-fad6-4078-ab8b-1a45fab198a9") : secret "dns-default-metrics-tls" not found Jul 06 15:18:20.753240 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:20.753151 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ce904e6-fad6-4078-ab8b-1a45fab198a9-tmp-dir\") pod \"dns-default-8lh8v\" (UID: \"1ce904e6-fad6-4078-ab8b-1a45fab198a9\") " pod="openshift-dns/dns-default-8lh8v" Jul 06 15:18:20.753451 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:20.753432 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1ce904e6-fad6-4078-ab8b-1a45fab198a9-config-volume\") pod \"dns-default-8lh8v\" (UID: \"1ce904e6-fad6-4078-ab8b-1a45fab198a9\") " pod="openshift-dns/dns-default-8lh8v" Jul 06 15:18:20.756882 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:20.756862 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9da83319-4207-4990-ad20-a322c1f750ec-cert\") pod \"ingress-canary-n5z5d\" (UID: \"9da83319-4207-4990-ad20-a322c1f750ec\") " pod="openshift-ingress-canary/ingress-canary-n5z5d" Jul 06 15:18:20.762608 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:20.762582 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shnlk\" (UniqueName: \"kubernetes.io/projected/1ce904e6-fad6-4078-ab8b-1a45fab198a9-kube-api-access-shnlk\") pod \"dns-default-8lh8v\" (UID: \"1ce904e6-fad6-4078-ab8b-1a45fab198a9\") " pod="openshift-dns/dns-default-8lh8v" Jul 06 15:18:20.764354 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:20.764338 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr28r\" (UniqueName: \"kubernetes.io/projected/9da83319-4207-4990-ad20-a322c1f750ec-kube-api-access-lr28r\") pod \"ingress-canary-n5z5d\" (UID: \"9da83319-4207-4990-ad20-a322c1f750ec\") " pod="openshift-ingress-canary/ingress-canary-n5z5d" Jul 06 15:18:20.831640 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:20.831603 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n5z5d" Jul 06 15:18:20.880222 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:20.880193 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:18:21.009680 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:21.009619 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n5z5d"] Jul 06 15:18:21.013996 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:18:21.013957 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9da83319_4207_4990_ad20_a322c1f750ec.slice/crio-d82b09189d4360a58229fa5c75007447abc46a5e5886ad3a63e517420e9177e9 WatchSource:0}: Error finding container d82b09189d4360a58229fa5c75007447abc46a5e5886ad3a63e517420e9177e9: Status 404 returned error can't find the container with id d82b09189d4360a58229fa5c75007447abc46a5e5886ad3a63e517420e9177e9 Jul 06 15:18:21.256072 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:21.255975 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ce904e6-fad6-4078-ab8b-1a45fab198a9-metrics-tls\") pod \"dns-default-8lh8v\" (UID: \"1ce904e6-fad6-4078-ab8b-1a45fab198a9\") " pod="openshift-dns/dns-default-8lh8v" Jul 06 15:18:21.256217 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:21.256101 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 15:18:21.256217 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:21.256158 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ce904e6-fad6-4078-ab8b-1a45fab198a9-metrics-tls podName:1ce904e6-fad6-4078-ab8b-1a45fab198a9 nodeName:}" failed. No retries permitted until 2026-07-06 15:18:22.256143334 +0000 UTC m=+48.869891850 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1ce904e6-fad6-4078-ab8b-1a45fab198a9-metrics-tls") pod "dns-default-8lh8v" (UID: "1ce904e6-fad6-4078-ab8b-1a45fab198a9") : secret "dns-default-metrics-tls" not found Jul 06 15:18:21.793462 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:21.793430 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:18:21.793919 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:21.793437 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:18:21.793919 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:21.793437 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:18:21.796848 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:21.796831 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 15:18:21.796944 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:21.796865 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 15:18:21.797891 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:21.797876 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gh99x\"" Jul 06 15:18:21.797989 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:21.797934 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-x2rt6\"" Jul 06 15:18:21.797989 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:21.797947 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 15:18:21.797989 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:21.797947 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 15:18:22.014446 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:22.014405 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n5z5d" event={"ID":"9da83319-4207-4990-ad20-a322c1f750ec","Type":"ContainerStarted","Data":"d82b09189d4360a58229fa5c75007447abc46a5e5886ad3a63e517420e9177e9"} Jul 06 15:18:22.265265 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:22.265230 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ce904e6-fad6-4078-ab8b-1a45fab198a9-metrics-tls\") pod \"dns-default-8lh8v\" (UID: \"1ce904e6-fad6-4078-ab8b-1a45fab198a9\") " pod="openshift-dns/dns-default-8lh8v" Jul 06 15:18:22.265450 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:22.265394 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 15:18:22.265523 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:22.265459 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ce904e6-fad6-4078-ab8b-1a45fab198a9-metrics-tls podName:1ce904e6-fad6-4078-ab8b-1a45fab198a9 nodeName:}" failed. No retries permitted until 2026-07-06 15:18:24.265442256 +0000 UTC m=+50.879190777 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1ce904e6-fad6-4078-ab8b-1a45fab198a9-metrics-tls") pod "dns-default-8lh8v" (UID: "1ce904e6-fad6-4078-ab8b-1a45fab198a9") : secret "dns-default-metrics-tls" not found Jul 06 15:18:24.283187 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:24.283151 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ce904e6-fad6-4078-ab8b-1a45fab198a9-metrics-tls\") pod \"dns-default-8lh8v\" (UID: \"1ce904e6-fad6-4078-ab8b-1a45fab198a9\") " pod="openshift-dns/dns-default-8lh8v" Jul 06 15:18:24.283616 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:24.283303 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 15:18:24.283616 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:24.283384 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ce904e6-fad6-4078-ab8b-1a45fab198a9-metrics-tls podName:1ce904e6-fad6-4078-ab8b-1a45fab198a9 nodeName:}" failed. No retries permitted until 2026-07-06 15:18:28.283362573 +0000 UTC m=+54.897111090 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1ce904e6-fad6-4078-ab8b-1a45fab198a9-metrics-tls") pod "dns-default-8lh8v" (UID: "1ce904e6-fad6-4078-ab8b-1a45fab198a9") : secret "dns-default-metrics-tls" not found Jul 06 15:18:25.024134 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:25.024103 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n5z5d" event={"ID":"9da83319-4207-4990-ad20-a322c1f750ec","Type":"ContainerStarted","Data":"cbf088e99476298a3eb0d415b879e5f2611a0389f5edb022d2cbe4bd54ea3b21"} Jul 06 15:18:25.045897 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:25.045846 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-n5z5d" podStartSLOduration=1.982540181 podStartE2EDuration="5.045831498s" totalImagesPullingTime="3.063291317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:18:25.045549124 +0000 UTC m=+51.659297660" watchObservedRunningTime="2026-07-06 15:18:25.045831498 +0000 UTC m=+51.659580034" Jul 06 15:18:28.311629 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:28.311585 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ce904e6-fad6-4078-ab8b-1a45fab198a9-metrics-tls\") pod \"dns-default-8lh8v\" (UID: \"1ce904e6-fad6-4078-ab8b-1a45fab198a9\") " pod="openshift-dns/dns-default-8lh8v" Jul 06 15:18:28.312078 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:28.311701 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 15:18:28.312078 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:28.311759 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ce904e6-fad6-4078-ab8b-1a45fab198a9-metrics-tls podName:1ce904e6-fad6-4078-ab8b-1a45fab198a9 nodeName:}" failed. No retries permitted until 2026-07-06 15:18:36.311746753 +0000 UTC m=+62.925495271 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1ce904e6-fad6-4078-ab8b-1a45fab198a9-metrics-tls") pod "dns-default-8lh8v" (UID: "1ce904e6-fad6-4078-ab8b-1a45fab198a9") : secret "dns-default-metrics-tls" not found Jul 06 15:18:36.361138 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:36.361107 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ce904e6-fad6-4078-ab8b-1a45fab198a9-metrics-tls\") pod \"dns-default-8lh8v\" (UID: \"1ce904e6-fad6-4078-ab8b-1a45fab198a9\") " pod="openshift-dns/dns-default-8lh8v" Jul 06 15:18:36.361622 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:36.361219 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 15:18:36.361622 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:36.361274 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ce904e6-fad6-4078-ab8b-1a45fab198a9-metrics-tls podName:1ce904e6-fad6-4078-ab8b-1a45fab198a9 nodeName:}" failed. No retries permitted until 2026-07-06 15:18:52.361256821 +0000 UTC m=+78.975005347 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1ce904e6-fad6-4078-ab8b-1a45fab198a9-metrics-tls") pod "dns-default-8lh8v" (UID: "1ce904e6-fad6-4078-ab8b-1a45fab198a9") : secret "dns-default-metrics-tls" not found Jul 06 15:18:39.380607 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:39.380571 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b-metrics-certs\") pod \"network-metrics-daemon-6x68d\" (UID: \"5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b\") " pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:18:39.384001 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:39.383983 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 15:18:39.391443 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:39.391431 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 06 15:18:39.391490 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:39.391480 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b-metrics-certs podName:5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b nodeName:}" failed. No retries permitted until 2026-07-06 15:19:43.391465338 +0000 UTC m=+130.005213852 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b-metrics-certs") pod "network-metrics-daemon-6x68d" (UID: "5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b") : secret "metrics-daemon-secret" not found Jul 06 15:18:39.481239 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:39.481209 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkznd\" (UniqueName: \"kubernetes.io/projected/e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac-kube-api-access-wkznd\") pod \"network-check-target-v9nsl\" (UID: \"e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac\") " pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:18:39.481332 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:39.481248 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9aaa45f1-1a99-4e94-96f3-a12ddbe62e83-original-pull-secret\") pod \"global-pull-secret-syncer-l77mc\" (UID: \"9aaa45f1-1a99-4e94-96f3-a12ddbe62e83\") " pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:18:39.484970 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:39.484955 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 15:18:39.485254 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:39.485243 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 15:18:39.493670 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:39.493651 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9aaa45f1-1a99-4e94-96f3-a12ddbe62e83-original-pull-secret\") pod \"global-pull-secret-syncer-l77mc\" (UID: \"9aaa45f1-1a99-4e94-96f3-a12ddbe62e83\") " pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:18:39.494656 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:39.494643 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 15:18:39.504606 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:39.504582 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkznd\" (UniqueName: \"kubernetes.io/projected/e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac-kube-api-access-wkznd\") pod \"network-check-target-v9nsl\" (UID: \"e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac\") " pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:18:39.512554 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:39.512539 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-x2rt6\"" Jul 06 15:18:39.514639 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:39.514626 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l77mc" Jul 06 15:18:39.520399 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:39.520385 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:18:39.677888 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:39.677865 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l77mc"] Jul 06 15:18:39.680315 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:18:39.680287 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aaa45f1_1a99_4e94_96f3_a12ddbe62e83.slice/crio-3728d8c82fc48801512a5e545e2d0eb2fe5c848977773c3776552730cbc42367 WatchSource:0}: Error finding container 3728d8c82fc48801512a5e545e2d0eb2fe5c848977773c3776552730cbc42367: Status 404 returned error can't find the container with id 3728d8c82fc48801512a5e545e2d0eb2fe5c848977773c3776552730cbc42367 Jul 06 15:18:39.691602 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:39.691574 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v9nsl"] Jul 06 15:18:39.695279 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:18:39.695249 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode834b8fc_fb20_4ccb_98b7_3b2c5e1fdcac.slice/crio-e7c4d517e8e7f43b92dd9eb6ddcd15783a83cb5f197e7a3b30beaa15719b9376 WatchSource:0}: Error finding container e7c4d517e8e7f43b92dd9eb6ddcd15783a83cb5f197e7a3b30beaa15719b9376: Status 404 returned error can't find the container with id e7c4d517e8e7f43b92dd9eb6ddcd15783a83cb5f197e7a3b30beaa15719b9376 Jul 06 15:18:40.057251 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:40.057171 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v9nsl" event={"ID":"e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac","Type":"ContainerStarted","Data":"e7c4d517e8e7f43b92dd9eb6ddcd15783a83cb5f197e7a3b30beaa15719b9376"} Jul 06 15:18:40.058263 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:40.058232 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l77mc" event={"ID":"9aaa45f1-1a99-4e94-96f3-a12ddbe62e83","Type":"ContainerStarted","Data":"3728d8c82fc48801512a5e545e2d0eb2fe5c848977773c3776552730cbc42367"} Jul 06 15:18:45.008204 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:45.008176 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9hxpw" Jul 06 15:18:45.154724 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:45.154680 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:18:45.221303 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:45.221268 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:18:46.074793 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:46.074756 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v9nsl" event={"ID":"e834b8fc-fb20-4ccb-98b7-3b2c5e1fdcac","Type":"ContainerStarted","Data":"dda611d82f721d639f2e8389571349d034c2a66378bc4d20a9745c13f9f1bf9f"} Jul 06 15:18:46.075277 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:46.074851 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:18:46.076236 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:46.076212 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l77mc" event={"ID":"9aaa45f1-1a99-4e94-96f3-a12ddbe62e83","Type":"ContainerStarted","Data":"93d8ea27735dc10e7aa7915ae350fcdd6e7c66946863a4d8e0d02b5187ea1731"} Jul 06 15:18:46.092042 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:46.091996 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-v9nsl" podStartSLOduration=65.459907986 podStartE2EDuration="1m11.091982481s" totalImagesPullingTime="5.632074495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:17:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:18:46.091623307 +0000 UTC m=+72.705371854" watchObservedRunningTime="2026-07-06 15:18:46.091982481 +0000 UTC m=+72.705731016" Jul 06 15:18:46.109767 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:46.109718 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-l77mc" podStartSLOduration=65.46215491 podStartE2EDuration="1m11.109703209s" totalImagesPullingTime="5.647548299s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:17:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:18:46.109421989 +0000 UTC m=+72.723170525" watchObservedRunningTime="2026-07-06 15:18:46.109703209 +0000 UTC m=+72.723451745" Jul 06 15:18:52.365427 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:52.365372 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ce904e6-fad6-4078-ab8b-1a45fab198a9-metrics-tls\") pod \"dns-default-8lh8v\" (UID: \"1ce904e6-fad6-4078-ab8b-1a45fab198a9\") " pod="openshift-dns/dns-default-8lh8v" Jul 06 15:18:52.365917 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:52.365536 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 15:18:52.365917 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:52.365604 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ce904e6-fad6-4078-ab8b-1a45fab198a9-metrics-tls podName:1ce904e6-fad6-4078-ab8b-1a45fab198a9 nodeName:}" failed. No retries permitted until 2026-07-06 15:19:24.365588044 +0000 UTC m=+110.979336558 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1ce904e6-fad6-4078-ab8b-1a45fab198a9-metrics-tls") pod "dns-default-8lh8v" (UID: "1ce904e6-fad6-4078-ab8b-1a45fab198a9") : secret "dns-default-metrics-tls" not found Jul 06 15:18:56.327213 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.327090 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-9ktzq"] Jul 06 15:18:56.328904 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.328889 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-9ktzq" Jul 06 15:18:56.331758 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.331738 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 06 15:18:56.332155 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.332131 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 15:18:56.332243 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.332191 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 06 15:18:56.332290 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.332271 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 15:18:56.334254 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.333781 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-fvb8l\"" Jul 06 15:18:56.339230 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.339210 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 06 15:18:56.348072 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.348047 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-9ktzq"] Jul 06 15:18:56.393636 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.393605 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9f53ff9d-d4c6-4b93-8eed-39e727850e02-snapshots\") pod \"insights-operator-5c847dcfc4-9ktzq\" (UID: \"9f53ff9d-d4c6-4b93-8eed-39e727850e02\") " pod="openshift-insights/insights-operator-5c847dcfc4-9ktzq" Jul 06 15:18:56.393636 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.393643 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f53ff9d-d4c6-4b93-8eed-39e727850e02-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-9ktzq\" (UID: \"9f53ff9d-d4c6-4b93-8eed-39e727850e02\") " pod="openshift-insights/insights-operator-5c847dcfc4-9ktzq" Jul 06 15:18:56.393892 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.393661 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f53ff9d-d4c6-4b93-8eed-39e727850e02-serving-cert\") pod \"insights-operator-5c847dcfc4-9ktzq\" (UID: \"9f53ff9d-d4c6-4b93-8eed-39e727850e02\") " pod="openshift-insights/insights-operator-5c847dcfc4-9ktzq" Jul 06 15:18:56.393892 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.393679 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bth24\" (UniqueName: \"kubernetes.io/projected/9f53ff9d-d4c6-4b93-8eed-39e727850e02-kube-api-access-bth24\") pod \"insights-operator-5c847dcfc4-9ktzq\" (UID: \"9f53ff9d-d4c6-4b93-8eed-39e727850e02\") " pod="openshift-insights/insights-operator-5c847dcfc4-9ktzq" Jul 06 15:18:56.393892 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.393698 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f53ff9d-d4c6-4b93-8eed-39e727850e02-tmp\") pod \"insights-operator-5c847dcfc4-9ktzq\" (UID: \"9f53ff9d-d4c6-4b93-8eed-39e727850e02\") " pod="openshift-insights/insights-operator-5c847dcfc4-9ktzq" Jul 06 15:18:56.393892 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.393757 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f53ff9d-d4c6-4b93-8eed-39e727850e02-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-9ktzq\" (UID: \"9f53ff9d-d4c6-4b93-8eed-39e727850e02\") " pod="openshift-insights/insights-operator-5c847dcfc4-9ktzq" Jul 06 15:18:56.437308 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.437279 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-55896fdd7f-l5sg2"] Jul 06 15:18:56.439020 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.439005 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-55896fdd7f-l5sg2" Jul 06 15:18:56.441892 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.441872 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 06 15:18:56.442053 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.442036 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 06 15:18:56.442303 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.442282 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 06 15:18:56.442402 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.442350 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 15:18:56.442402 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.442360 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vpwgw\"" Jul 06 15:18:56.442614 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.442598 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 06 15:18:56.442860 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.442845 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 15:18:56.459558 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.459538 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-55896fdd7f-l5sg2"] Jul 06 15:18:56.494421 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.494389 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9f53ff9d-d4c6-4b93-8eed-39e727850e02-snapshots\") pod \"insights-operator-5c847dcfc4-9ktzq\" (UID: \"9f53ff9d-d4c6-4b93-8eed-39e727850e02\") " pod="openshift-insights/insights-operator-5c847dcfc4-9ktzq" Jul 06 15:18:56.494600 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.494427 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgcsg\" (UniqueName: \"kubernetes.io/projected/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-kube-api-access-hgcsg\") pod \"router-default-55896fdd7f-l5sg2\" (UID: \"d00f67b5-9c7a-41f4-bd9e-2160fcd8641a\") " pod="openshift-ingress/router-default-55896fdd7f-l5sg2" Jul 06 15:18:56.494600 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.494460 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f53ff9d-d4c6-4b93-8eed-39e727850e02-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-9ktzq\" (UID: \"9f53ff9d-d4c6-4b93-8eed-39e727850e02\") " pod="openshift-insights/insights-operator-5c847dcfc4-9ktzq" Jul 06 15:18:56.494600 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.494524 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f53ff9d-d4c6-4b93-8eed-39e727850e02-serving-cert\") pod \"insights-operator-5c847dcfc4-9ktzq\" (UID: \"9f53ff9d-d4c6-4b93-8eed-39e727850e02\") " pod="openshift-insights/insights-operator-5c847dcfc4-9ktzq" Jul 06 15:18:56.494600 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.494553 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-default-certificate\") pod \"router-default-55896fdd7f-l5sg2\" (UID: \"d00f67b5-9c7a-41f4-bd9e-2160fcd8641a\") " pod="openshift-ingress/router-default-55896fdd7f-l5sg2" Jul 06 15:18:56.494600 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.494580 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bth24\" (UniqueName: \"kubernetes.io/projected/9f53ff9d-d4c6-4b93-8eed-39e727850e02-kube-api-access-bth24\") pod \"insights-operator-5c847dcfc4-9ktzq\" (UID: \"9f53ff9d-d4c6-4b93-8eed-39e727850e02\") " pod="openshift-insights/insights-operator-5c847dcfc4-9ktzq" Jul 06 15:18:56.494600 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.494598 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-stats-auth\") pod \"router-default-55896fdd7f-l5sg2\" (UID: \"d00f67b5-9c7a-41f4-bd9e-2160fcd8641a\") " pod="openshift-ingress/router-default-55896fdd7f-l5sg2" Jul 06 15:18:56.494869 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.494645 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f53ff9d-d4c6-4b93-8eed-39e727850e02-tmp\") pod \"insights-operator-5c847dcfc4-9ktzq\" (UID: \"9f53ff9d-d4c6-4b93-8eed-39e727850e02\") " pod="openshift-insights/insights-operator-5c847dcfc4-9ktzq" Jul 06 15:18:56.494869 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.494699 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f53ff9d-d4c6-4b93-8eed-39e727850e02-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-9ktzq\" (UID: \"9f53ff9d-d4c6-4b93-8eed-39e727850e02\") " pod="openshift-insights/insights-operator-5c847dcfc4-9ktzq" Jul 06 15:18:56.494869 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.494724 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-service-ca-bundle\") pod \"router-default-55896fdd7f-l5sg2\" (UID: \"d00f67b5-9c7a-41f4-bd9e-2160fcd8641a\") " pod="openshift-ingress/router-default-55896fdd7f-l5sg2" Jul 06 15:18:56.494869 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.494784 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-metrics-certs\") pod \"router-default-55896fdd7f-l5sg2\" (UID: \"d00f67b5-9c7a-41f4-bd9e-2160fcd8641a\") " pod="openshift-ingress/router-default-55896fdd7f-l5sg2" Jul 06 15:18:56.495105 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.495082 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f53ff9d-d4c6-4b93-8eed-39e727850e02-tmp\") pod \"insights-operator-5c847dcfc4-9ktzq\" (UID: \"9f53ff9d-d4c6-4b93-8eed-39e727850e02\") " pod="openshift-insights/insights-operator-5c847dcfc4-9ktzq" Jul 06 15:18:56.495105 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.495099 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9f53ff9d-d4c6-4b93-8eed-39e727850e02-snapshots\") pod \"insights-operator-5c847dcfc4-9ktzq\" (UID: \"9f53ff9d-d4c6-4b93-8eed-39e727850e02\") " pod="openshift-insights/insights-operator-5c847dcfc4-9ktzq" Jul 06 15:18:56.495285 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.495263 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f53ff9d-d4c6-4b93-8eed-39e727850e02-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-9ktzq\" (UID: \"9f53ff9d-d4c6-4b93-8eed-39e727850e02\") " pod="openshift-insights/insights-operator-5c847dcfc4-9ktzq" Jul 06 15:18:56.495324 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.495310 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f53ff9d-d4c6-4b93-8eed-39e727850e02-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-9ktzq\" (UID: \"9f53ff9d-d4c6-4b93-8eed-39e727850e02\") " pod="openshift-insights/insights-operator-5c847dcfc4-9ktzq" Jul 06 15:18:56.496823 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.496805 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f53ff9d-d4c6-4b93-8eed-39e727850e02-serving-cert\") pod \"insights-operator-5c847dcfc4-9ktzq\" (UID: \"9f53ff9d-d4c6-4b93-8eed-39e727850e02\") " pod="openshift-insights/insights-operator-5c847dcfc4-9ktzq" Jul 06 15:18:56.503244 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.503220 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bth24\" (UniqueName: \"kubernetes.io/projected/9f53ff9d-d4c6-4b93-8eed-39e727850e02-kube-api-access-bth24\") pod \"insights-operator-5c847dcfc4-9ktzq\" (UID: \"9f53ff9d-d4c6-4b93-8eed-39e727850e02\") " pod="openshift-insights/insights-operator-5c847dcfc4-9ktzq" Jul 06 15:18:56.595849 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.595759 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-default-certificate\") pod \"router-default-55896fdd7f-l5sg2\" (UID: \"d00f67b5-9c7a-41f4-bd9e-2160fcd8641a\") " pod="openshift-ingress/router-default-55896fdd7f-l5sg2" Jul 06 15:18:56.595849 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.595813 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-stats-auth\") pod \"router-default-55896fdd7f-l5sg2\" (UID: \"d00f67b5-9c7a-41f4-bd9e-2160fcd8641a\") " pod="openshift-ingress/router-default-55896fdd7f-l5sg2" Jul 06 15:18:56.595849 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.595851 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-service-ca-bundle\") pod \"router-default-55896fdd7f-l5sg2\" (UID: \"d00f67b5-9c7a-41f4-bd9e-2160fcd8641a\") " pod="openshift-ingress/router-default-55896fdd7f-l5sg2" Jul 06 15:18:56.596105 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.595902 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-metrics-certs\") pod \"router-default-55896fdd7f-l5sg2\" (UID: \"d00f67b5-9c7a-41f4-bd9e-2160fcd8641a\") " pod="openshift-ingress/router-default-55896fdd7f-l5sg2" Jul 06 15:18:56.596105 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.595941 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgcsg\" (UniqueName: \"kubernetes.io/projected/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-kube-api-access-hgcsg\") pod \"router-default-55896fdd7f-l5sg2\" (UID: \"d00f67b5-9c7a-41f4-bd9e-2160fcd8641a\") " pod="openshift-ingress/router-default-55896fdd7f-l5sg2" Jul 06 15:18:56.596105 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:56.596049 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-service-ca-bundle podName:d00f67b5-9c7a-41f4-bd9e-2160fcd8641a nodeName:}" failed. No retries permitted until 2026-07-06 15:18:57.096024417 +0000 UTC m=+83.709772931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-service-ca-bundle") pod "router-default-55896fdd7f-l5sg2" (UID: "d00f67b5-9c7a-41f4-bd9e-2160fcd8641a") : configmap references non-existent config key: service-ca.crt Jul 06 15:18:56.596105 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:56.596085 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 15:18:56.596281 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:56.596151 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-metrics-certs podName:d00f67b5-9c7a-41f4-bd9e-2160fcd8641a nodeName:}" failed. No retries permitted until 2026-07-06 15:18:57.096133207 +0000 UTC m=+83.709881721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-metrics-certs") pod "router-default-55896fdd7f-l5sg2" (UID: "d00f67b5-9c7a-41f4-bd9e-2160fcd8641a") : secret "router-metrics-certs-default" not found Jul 06 15:18:56.598316 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.598289 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-default-certificate\") pod \"router-default-55896fdd7f-l5sg2\" (UID: \"d00f67b5-9c7a-41f4-bd9e-2160fcd8641a\") " pod="openshift-ingress/router-default-55896fdd7f-l5sg2" Jul 06 15:18:56.598428 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.598408 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-stats-auth\") pod \"router-default-55896fdd7f-l5sg2\" (UID: \"d00f67b5-9c7a-41f4-bd9e-2160fcd8641a\") " pod="openshift-ingress/router-default-55896fdd7f-l5sg2" Jul 06 15:18:56.606997 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.606975 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgcsg\" (UniqueName: \"kubernetes.io/projected/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-kube-api-access-hgcsg\") pod \"router-default-55896fdd7f-l5sg2\" (UID: \"d00f67b5-9c7a-41f4-bd9e-2160fcd8641a\") " pod="openshift-ingress/router-default-55896fdd7f-l5sg2" Jul 06 15:18:56.609726 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.609703 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:18:56.639129 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.639098 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-9ktzq" Jul 06 15:18:56.738689 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.738649 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:18:56.772930 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:56.772902 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-9ktzq"] Jul 06 15:18:56.776143 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:18:56.776116 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f53ff9d_d4c6_4b93_8eed_39e727850e02.slice/crio-d33a39c4eef26ad5f0c46212ff4bb221681905e363a01c82e25aee6fe85bdf3a WatchSource:0}: Error finding container d33a39c4eef26ad5f0c46212ff4bb221681905e363a01c82e25aee6fe85bdf3a: Status 404 returned error can't find the container with id d33a39c4eef26ad5f0c46212ff4bb221681905e363a01c82e25aee6fe85bdf3a Jul 06 15:18:57.098765 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:57.098732 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-service-ca-bundle\") pod \"router-default-55896fdd7f-l5sg2\" (UID: \"d00f67b5-9c7a-41f4-bd9e-2160fcd8641a\") " pod="openshift-ingress/router-default-55896fdd7f-l5sg2" Jul 06 15:18:57.098940 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:57.098785 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-metrics-certs\") pod \"router-default-55896fdd7f-l5sg2\" (UID: \"d00f67b5-9c7a-41f4-bd9e-2160fcd8641a\") " pod="openshift-ingress/router-default-55896fdd7f-l5sg2" Jul 06 15:18:57.098940 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:57.098903 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 15:18:57.098940 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:57.098918 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-service-ca-bundle podName:d00f67b5-9c7a-41f4-bd9e-2160fcd8641a nodeName:}" failed. No retries permitted until 2026-07-06 15:18:58.098898849 +0000 UTC m=+84.712647382 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-service-ca-bundle") pod "router-default-55896fdd7f-l5sg2" (UID: "d00f67b5-9c7a-41f4-bd9e-2160fcd8641a") : configmap references non-existent config key: service-ca.crt Jul 06 15:18:57.098940 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:57.098941 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-metrics-certs podName:d00f67b5-9c7a-41f4-bd9e-2160fcd8641a nodeName:}" failed. No retries permitted until 2026-07-06 15:18:58.098930902 +0000 UTC m=+84.712679420 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-metrics-certs") pod "router-default-55896fdd7f-l5sg2" (UID: "d00f67b5-9c7a-41f4-bd9e-2160fcd8641a") : secret "router-metrics-certs-default" not found Jul 06 15:18:57.103780 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:57.103756 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-9ktzq" event={"ID":"9f53ff9d-d4c6-4b93-8eed-39e727850e02","Type":"ContainerStarted","Data":"d33a39c4eef26ad5f0c46212ff4bb221681905e363a01c82e25aee6fe85bdf3a"} Jul 06 15:18:58.104713 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:58.104679 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-service-ca-bundle\") pod \"router-default-55896fdd7f-l5sg2\" (UID: \"d00f67b5-9c7a-41f4-bd9e-2160fcd8641a\") " pod="openshift-ingress/router-default-55896fdd7f-l5sg2" Jul 06 15:18:58.105301 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:58.104747 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-metrics-certs\") pod \"router-default-55896fdd7f-l5sg2\" (UID: \"d00f67b5-9c7a-41f4-bd9e-2160fcd8641a\") " pod="openshift-ingress/router-default-55896fdd7f-l5sg2" Jul 06 15:18:58.105301 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:58.104861 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 15:18:58.105301 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:58.104884 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-service-ca-bundle podName:d00f67b5-9c7a-41f4-bd9e-2160fcd8641a nodeName:}" failed. No retries permitted until 2026-07-06 15:19:00.104858236 +0000 UTC m=+86.718606754 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-service-ca-bundle") pod "router-default-55896fdd7f-l5sg2" (UID: "d00f67b5-9c7a-41f4-bd9e-2160fcd8641a") : configmap references non-existent config key: service-ca.crt Jul 06 15:18:58.105301 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:18:58.104909 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-metrics-certs podName:d00f67b5-9c7a-41f4-bd9e-2160fcd8641a nodeName:}" failed. No retries permitted until 2026-07-06 15:19:00.10489896 +0000 UTC m=+86.718647474 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-metrics-certs") pod "router-default-55896fdd7f-l5sg2" (UID: "d00f67b5-9c7a-41f4-bd9e-2160fcd8641a") : secret "router-metrics-certs-default" not found Jul 06 15:18:59.110663 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:59.110626 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-9ktzq" event={"ID":"9f53ff9d-d4c6-4b93-8eed-39e727850e02","Type":"ContainerStarted","Data":"e48986d805b8d0bd8099ac022e982fb2dba001c5eed431371507cbff2c7d7fb1"} Jul 06 15:18:59.132999 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:18:59.132944 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-9ktzq" podStartSLOduration=0.989744837 podStartE2EDuration="3.132927701s" totalImagesPullingTime="2.143182864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:18:59.132216636 +0000 UTC m=+85.745965173" watchObservedRunningTime="2026-07-06 15:18:59.132927701 +0000 UTC m=+85.746676234" Jul 06 15:19:00.009765 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:00.009726 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9hm78"] Jul 06 15:19:00.012224 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:00.012199 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9hm78" Jul 06 15:19:00.015827 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:00.015803 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rqfh6\"" Jul 06 15:19:00.015987 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:00.015967 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 15:19:00.016931 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:00.016912 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 15:19:00.031153 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:00.031127 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9hm78"] Jul 06 15:19:00.109114 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:00.109076 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:00.122165 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:00.122139 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/92299e93-9bb8-452f-ba3a-523beff0880a-crio-socket\") pod \"insights-runtime-extractor-9hm78\" (UID: \"92299e93-9bb8-452f-ba3a-523beff0880a\") " pod="openshift-insights/insights-runtime-extractor-9hm78" Jul 06 15:19:00.122660 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:00.122176 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92299e93-9bb8-452f-ba3a-523beff0880a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9hm78\" (UID: \"92299e93-9bb8-452f-ba3a-523beff0880a\") " pod="openshift-insights/insights-runtime-extractor-9hm78" Jul 06 15:19:00.122660 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:00.122250 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92299e93-9bb8-452f-ba3a-523beff0880a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9hm78\" (UID: \"92299e93-9bb8-452f-ba3a-523beff0880a\") " pod="openshift-insights/insights-runtime-extractor-9hm78" Jul 06 15:19:00.122660 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:00.122304 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/92299e93-9bb8-452f-ba3a-523beff0880a-data-volume\") pod \"insights-runtime-extractor-9hm78\" (UID: \"92299e93-9bb8-452f-ba3a-523beff0880a\") " pod="openshift-insights/insights-runtime-extractor-9hm78" Jul 06 15:19:00.122660 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:00.122322 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx97t\" (UniqueName: \"kubernetes.io/projected/92299e93-9bb8-452f-ba3a-523beff0880a-kube-api-access-kx97t\") pod \"insights-runtime-extractor-9hm78\" (UID: \"92299e93-9bb8-452f-ba3a-523beff0880a\") " pod="openshift-insights/insights-runtime-extractor-9hm78" Jul 06 15:19:00.122660 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:00.122399 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-service-ca-bundle\") pod \"router-default-55896fdd7f-l5sg2\" (UID: \"d00f67b5-9c7a-41f4-bd9e-2160fcd8641a\") " pod="openshift-ingress/router-default-55896fdd7f-l5sg2" Jul 06 15:19:00.122660 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:00.122465 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-metrics-certs\") pod \"router-default-55896fdd7f-l5sg2\" (UID: \"d00f67b5-9c7a-41f4-bd9e-2160fcd8641a\") " pod="openshift-ingress/router-default-55896fdd7f-l5sg2" Jul 06 15:19:00.122660 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:00.122513 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-service-ca-bundle podName:d00f67b5-9c7a-41f4-bd9e-2160fcd8641a nodeName:}" failed. No retries permitted until 2026-07-06 15:19:04.122479202 +0000 UTC m=+90.736227715 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-service-ca-bundle") pod "router-default-55896fdd7f-l5sg2" (UID: "d00f67b5-9c7a-41f4-bd9e-2160fcd8641a") : configmap references non-existent config key: service-ca.crt Jul 06 15:19:00.122660 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:00.122593 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 15:19:00.122660 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:00.122647 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-metrics-certs podName:d00f67b5-9c7a-41f4-bd9e-2160fcd8641a nodeName:}" failed. No retries permitted until 2026-07-06 15:19:04.122633001 +0000 UTC m=+90.736381516 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-metrics-certs") pod "router-default-55896fdd7f-l5sg2" (UID: "d00f67b5-9c7a-41f4-bd9e-2160fcd8641a") : secret "router-metrics-certs-default" not found Jul 06 15:19:00.181514 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:00.181473 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:00.223701 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:00.223667 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/92299e93-9bb8-452f-ba3a-523beff0880a-data-volume\") pod \"insights-runtime-extractor-9hm78\" (UID: \"92299e93-9bb8-452f-ba3a-523beff0880a\") " pod="openshift-insights/insights-runtime-extractor-9hm78" Jul 06 15:19:00.223701 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:00.223700 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx97t\" (UniqueName: \"kubernetes.io/projected/92299e93-9bb8-452f-ba3a-523beff0880a-kube-api-access-kx97t\") pod \"insights-runtime-extractor-9hm78\" (UID: \"92299e93-9bb8-452f-ba3a-523beff0880a\") " pod="openshift-insights/insights-runtime-extractor-9hm78" Jul 06 15:19:00.224032 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:00.223990 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/92299e93-9bb8-452f-ba3a-523beff0880a-crio-socket\") pod \"insights-runtime-extractor-9hm78\" (UID: \"92299e93-9bb8-452f-ba3a-523beff0880a\") " pod="openshift-insights/insights-runtime-extractor-9hm78" Jul 06 15:19:00.224122 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:00.224042 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92299e93-9bb8-452f-ba3a-523beff0880a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9hm78\" (UID: \"92299e93-9bb8-452f-ba3a-523beff0880a\") " pod="openshift-insights/insights-runtime-extractor-9hm78" Jul 06 15:19:00.224122 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:00.224059 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/92299e93-9bb8-452f-ba3a-523beff0880a-data-volume\") pod \"insights-runtime-extractor-9hm78\" (UID: \"92299e93-9bb8-452f-ba3a-523beff0880a\") " pod="openshift-insights/insights-runtime-extractor-9hm78" Jul 06 15:19:00.224122 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:00.224066 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92299e93-9bb8-452f-ba3a-523beff0880a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9hm78\" (UID: \"92299e93-9bb8-452f-ba3a-523beff0880a\") " pod="openshift-insights/insights-runtime-extractor-9hm78" Jul 06 15:19:00.224122 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:00.224059 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/92299e93-9bb8-452f-ba3a-523beff0880a-crio-socket\") pod \"insights-runtime-extractor-9hm78\" (UID: \"92299e93-9bb8-452f-ba3a-523beff0880a\") " pod="openshift-insights/insights-runtime-extractor-9hm78" Jul 06 15:19:00.224122 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:00.224116 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 15:19:00.224315 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:00.224140 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 15:19:00.224315 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:00.224194 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/92299e93-9bb8-452f-ba3a-523beff0880a-kube-rbac-proxy-cm podName:92299e93-9bb8-452f-ba3a-523beff0880a nodeName:}" failed. No retries permitted until 2026-07-06 15:19:00.724174977 +0000 UTC m=+87.337923493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/92299e93-9bb8-452f-ba3a-523beff0880a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9hm78" (UID: "92299e93-9bb8-452f-ba3a-523beff0880a") : configmap "kube-rbac-proxy" not found Jul 06 15:19:00.224315 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:00.224207 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92299e93-9bb8-452f-ba3a-523beff0880a-insights-runtime-extractor-tls podName:92299e93-9bb8-452f-ba3a-523beff0880a nodeName:}" failed. No retries permitted until 2026-07-06 15:19:00.72420157 +0000 UTC m=+87.337950084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/92299e93-9bb8-452f-ba3a-523beff0880a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9hm78" (UID: "92299e93-9bb8-452f-ba3a-523beff0880a") : secret "insights-runtime-extractor-tls" not found Jul 06 15:19:00.234789 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:00.234768 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx97t\" (UniqueName: \"kubernetes.io/projected/92299e93-9bb8-452f-ba3a-523beff0880a-kube-api-access-kx97t\") pod \"insights-runtime-extractor-9hm78\" (UID: \"92299e93-9bb8-452f-ba3a-523beff0880a\") " pod="openshift-insights/insights-runtime-extractor-9hm78" Jul 06 15:19:00.726998 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:00.726961 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92299e93-9bb8-452f-ba3a-523beff0880a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9hm78\" (UID: \"92299e93-9bb8-452f-ba3a-523beff0880a\") " pod="openshift-insights/insights-runtime-extractor-9hm78" Jul 06 15:19:00.726998 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:00.726997 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92299e93-9bb8-452f-ba3a-523beff0880a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9hm78\" (UID: \"92299e93-9bb8-452f-ba3a-523beff0880a\") " pod="openshift-insights/insights-runtime-extractor-9hm78" Jul 06 15:19:00.727268 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:00.727088 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 15:19:00.727268 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:00.727111 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 15:19:00.727268 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:00.727152 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92299e93-9bb8-452f-ba3a-523beff0880a-insights-runtime-extractor-tls podName:92299e93-9bb8-452f-ba3a-523beff0880a nodeName:}" failed. No retries permitted until 2026-07-06 15:19:01.727140056 +0000 UTC m=+88.340888581 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/92299e93-9bb8-452f-ba3a-523beff0880a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9hm78" (UID: "92299e93-9bb8-452f-ba3a-523beff0880a") : secret "insights-runtime-extractor-tls" not found Jul 06 15:19:00.727268 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:00.727164 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/92299e93-9bb8-452f-ba3a-523beff0880a-kube-rbac-proxy-cm podName:92299e93-9bb8-452f-ba3a-523beff0880a nodeName:}" failed. No retries permitted until 2026-07-06 15:19:01.727158671 +0000 UTC m=+88.340907185 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/92299e93-9bb8-452f-ba3a-523beff0880a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9hm78" (UID: "92299e93-9bb8-452f-ba3a-523beff0880a") : configmap "kube-rbac-proxy" not found Jul 06 15:19:01.734421 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:01.734378 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92299e93-9bb8-452f-ba3a-523beff0880a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9hm78\" (UID: \"92299e93-9bb8-452f-ba3a-523beff0880a\") " pod="openshift-insights/insights-runtime-extractor-9hm78" Jul 06 15:19:01.734421 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:01.734422 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92299e93-9bb8-452f-ba3a-523beff0880a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9hm78\" (UID: \"92299e93-9bb8-452f-ba3a-523beff0880a\") " pod="openshift-insights/insights-runtime-extractor-9hm78" Jul 06 15:19:01.734920 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:01.734532 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 15:19:01.734920 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:01.734577 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 15:19:01.734920 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:01.734594 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/92299e93-9bb8-452f-ba3a-523beff0880a-kube-rbac-proxy-cm podName:92299e93-9bb8-452f-ba3a-523beff0880a nodeName:}" failed. No retries permitted until 2026-07-06 15:19:03.734578389 +0000 UTC m=+90.348326903 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/92299e93-9bb8-452f-ba3a-523beff0880a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9hm78" (UID: "92299e93-9bb8-452f-ba3a-523beff0880a") : configmap "kube-rbac-proxy" not found Jul 06 15:19:01.734920 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:01.734635 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92299e93-9bb8-452f-ba3a-523beff0880a-insights-runtime-extractor-tls podName:92299e93-9bb8-452f-ba3a-523beff0880a nodeName:}" failed. No retries permitted until 2026-07-06 15:19:03.734619964 +0000 UTC m=+90.348368481 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/92299e93-9bb8-452f-ba3a-523beff0880a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9hm78" (UID: "92299e93-9bb8-452f-ba3a-523beff0880a") : secret "insights-runtime-extractor-tls" not found Jul 06 15:19:02.207343 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:02.207315 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mgd8j_65d43ca0-dc12-4e4f-8a7d-a42c0f10a651/dns-node-resolver/0.log" Jul 06 15:19:03.206853 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:03.206824 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fbvqs_9a7b5582-0829-4212-8ff0-6379e5eb5306/node-ca/0.log" Jul 06 15:19:03.751069 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:03.751029 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92299e93-9bb8-452f-ba3a-523beff0880a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9hm78\" (UID: \"92299e93-9bb8-452f-ba3a-523beff0880a\") " pod="openshift-insights/insights-runtime-extractor-9hm78" Jul 06 15:19:03.751069 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:03.751071 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92299e93-9bb8-452f-ba3a-523beff0880a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9hm78\" (UID: \"92299e93-9bb8-452f-ba3a-523beff0880a\") " pod="openshift-insights/insights-runtime-extractor-9hm78" Jul 06 15:19:03.751329 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:03.751183 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 15:19:03.751329 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:03.751190 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 15:19:03.751329 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:03.751235 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92299e93-9bb8-452f-ba3a-523beff0880a-insights-runtime-extractor-tls podName:92299e93-9bb8-452f-ba3a-523beff0880a nodeName:}" failed. No retries permitted until 2026-07-06 15:19:07.751221415 +0000 UTC m=+94.364969930 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/92299e93-9bb8-452f-ba3a-523beff0880a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9hm78" (UID: "92299e93-9bb8-452f-ba3a-523beff0880a") : secret "insights-runtime-extractor-tls" not found Jul 06 15:19:03.751329 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:03.751267 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/92299e93-9bb8-452f-ba3a-523beff0880a-kube-rbac-proxy-cm podName:92299e93-9bb8-452f-ba3a-523beff0880a nodeName:}" failed. No retries permitted until 2026-07-06 15:19:07.751250396 +0000 UTC m=+94.364998910 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/92299e93-9bb8-452f-ba3a-523beff0880a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9hm78" (UID: "92299e93-9bb8-452f-ba3a-523beff0880a") : configmap "kube-rbac-proxy" not found Jul 06 15:19:03.808631 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:03.808600 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n5z5d_9da83319-4207-4990-ad20-a322c1f750ec/serve-healthcheck-canary/0.log" Jul 06 15:19:04.154363 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:04.154325 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-service-ca-bundle\") pod \"router-default-55896fdd7f-l5sg2\" (UID: \"d00f67b5-9c7a-41f4-bd9e-2160fcd8641a\") " pod="openshift-ingress/router-default-55896fdd7f-l5sg2" Jul 06 15:19:04.154536 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:04.154377 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-metrics-certs\") pod \"router-default-55896fdd7f-l5sg2\" (UID: \"d00f67b5-9c7a-41f4-bd9e-2160fcd8641a\") " pod="openshift-ingress/router-default-55896fdd7f-l5sg2" Jul 06 15:19:04.154536 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:04.154524 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 15:19:04.154634 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:04.154526 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-service-ca-bundle podName:d00f67b5-9c7a-41f4-bd9e-2160fcd8641a nodeName:}" failed. No retries permitted until 2026-07-06 15:19:12.154492395 +0000 UTC m=+98.768240912 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-service-ca-bundle") pod "router-default-55896fdd7f-l5sg2" (UID: "d00f67b5-9c7a-41f4-bd9e-2160fcd8641a") : configmap references non-existent config key: service-ca.crt Jul 06 15:19:04.154634 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:04.154576 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-metrics-certs podName:d00f67b5-9c7a-41f4-bd9e-2160fcd8641a nodeName:}" failed. No retries permitted until 2026-07-06 15:19:12.154565255 +0000 UTC m=+98.768313772 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-metrics-certs") pod "router-default-55896fdd7f-l5sg2" (UID: "d00f67b5-9c7a-41f4-bd9e-2160fcd8641a") : secret "router-metrics-certs-default" not found Jul 06 15:19:06.359648 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.359613 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7s7kr"] Jul 06 15:19:06.361675 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.361661 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-mlw9w"] Jul 06 15:19:06.361795 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.361777 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7s7kr" Jul 06 15:19:06.363455 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.363440 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jv8z5"] Jul 06 15:19:06.363602 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.363584 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-mlw9w" Jul 06 15:19:06.364592 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.364571 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-87dt7\"" Jul 06 15:19:06.364722 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.364657 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 06 15:19:06.364722 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.364700 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 06 15:19:06.364832 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.364728 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 06 15:19:06.364930 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.364911 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 06 15:19:06.365270 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.365257 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jv8z5" Jul 06 15:19:06.370722 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.370533 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 06 15:19:06.370890 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.370872 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 06 15:19:06.370967 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.370878 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 06 15:19:06.370967 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.370917 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 06 15:19:06.371088 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.370921 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 06 15:19:06.371221 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.371141 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-pdbjc\"" Jul 06 15:19:06.371221 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.371186 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 06 15:19:06.371221 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.371191 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 06 15:19:06.372342 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.372299 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 06 15:19:06.372342 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.372313 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-rd652\"" Jul 06 15:19:06.375059 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.375042 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 06 15:19:06.379813 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.379795 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7s7kr"] Jul 06 15:19:06.380948 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.380929 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jv8z5"] Jul 06 15:19:06.381797 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.381778 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-mlw9w"] Jul 06 15:19:06.471979 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.471947 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/91e0c855-5b32-4ca5-bab6-8e3aae20c1e0-config\") pod \"console-operator-68b757fb67-mlw9w\" (UID: \"91e0c855-5b32-4ca5-bab6-8e3aae20c1e0\") " pod="openshift-console-operator/console-operator-68b757fb67-mlw9w" Jul 06 15:19:06.472168 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.471991 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4b9bcef7-bd88-424e-b6cb-1125adbb047d-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-jv8z5\" (UID: \"4b9bcef7-bd88-424e-b6cb-1125adbb047d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jv8z5" Jul 06 15:19:06.472168 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.472011 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3f6843dc-43c3-44f3-9d07-6ac8a695b490-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-7s7kr\" (UID: \"3f6843dc-43c3-44f3-9d07-6ac8a695b490\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7s7kr" Jul 06 15:19:06.472168 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.472087 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8pdh\" (UniqueName: \"kubernetes.io/projected/4b9bcef7-bd88-424e-b6cb-1125adbb047d-kube-api-access-z8pdh\") pod \"kube-storage-version-migrator-operator-846bf668f7-jv8z5\" (UID: \"4b9bcef7-bd88-424e-b6cb-1125adbb047d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jv8z5" Jul 06 15:19:06.472168 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.472137 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fltb4\" (UniqueName: \"kubernetes.io/projected/91e0c855-5b32-4ca5-bab6-8e3aae20c1e0-kube-api-access-fltb4\") pod \"console-operator-68b757fb67-mlw9w\" (UID: \"91e0c855-5b32-4ca5-bab6-8e3aae20c1e0\") " pod="openshift-console-operator/console-operator-68b757fb67-mlw9w" Jul 06 15:19:06.472332 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.472176 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3f6843dc-43c3-44f3-9d07-6ac8a695b490-config\") pod \"service-ca-operator-5d46d9ffd4-7s7kr\" (UID: \"3f6843dc-43c3-44f3-9d07-6ac8a695b490\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7s7kr" Jul 06 15:19:06.472332 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.472198 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/91e0c855-5b32-4ca5-bab6-8e3aae20c1e0-serving-cert\") pod \"console-operator-68b757fb67-mlw9w\" (UID: \"91e0c855-5b32-4ca5-bab6-8e3aae20c1e0\") " pod="openshift-console-operator/console-operator-68b757fb67-mlw9w" Jul 06 15:19:06.472332 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.472222 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdz7h\" (UniqueName: \"kubernetes.io/projected/3f6843dc-43c3-44f3-9d07-6ac8a695b490-kube-api-access-cdz7h\") pod \"service-ca-operator-5d46d9ffd4-7s7kr\" (UID: \"3f6843dc-43c3-44f3-9d07-6ac8a695b490\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7s7kr" Jul 06 15:19:06.472332 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.472237 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4b9bcef7-bd88-424e-b6cb-1125adbb047d-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-jv8z5\" (UID: \"4b9bcef7-bd88-424e-b6cb-1125adbb047d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jv8z5" Jul 06 15:19:06.472332 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.472268 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/91e0c855-5b32-4ca5-bab6-8e3aae20c1e0-trusted-ca\") pod \"console-operator-68b757fb67-mlw9w\" (UID: \"91e0c855-5b32-4ca5-bab6-8e3aae20c1e0\") " pod="openshift-console-operator/console-operator-68b757fb67-mlw9w" Jul 06 15:19:06.573256 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.573207 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4b9bcef7-bd88-424e-b6cb-1125adbb047d-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-jv8z5\" (UID: \"4b9bcef7-bd88-424e-b6cb-1125adbb047d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jv8z5" Jul 06 15:19:06.573256 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.573263 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3f6843dc-43c3-44f3-9d07-6ac8a695b490-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-7s7kr\" (UID: \"3f6843dc-43c3-44f3-9d07-6ac8a695b490\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7s7kr" Jul 06 15:19:06.573488 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.573314 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8pdh\" (UniqueName: \"kubernetes.io/projected/4b9bcef7-bd88-424e-b6cb-1125adbb047d-kube-api-access-z8pdh\") pod \"kube-storage-version-migrator-operator-846bf668f7-jv8z5\" (UID: \"4b9bcef7-bd88-424e-b6cb-1125adbb047d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jv8z5" Jul 06 15:19:06.573488 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.573370 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fltb4\" (UniqueName: \"kubernetes.io/projected/91e0c855-5b32-4ca5-bab6-8e3aae20c1e0-kube-api-access-fltb4\") pod \"console-operator-68b757fb67-mlw9w\" (UID: \"91e0c855-5b32-4ca5-bab6-8e3aae20c1e0\") " pod="openshift-console-operator/console-operator-68b757fb67-mlw9w" Jul 06 15:19:06.573488 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.573405 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3f6843dc-43c3-44f3-9d07-6ac8a695b490-config\") pod \"service-ca-operator-5d46d9ffd4-7s7kr\" (UID: \"3f6843dc-43c3-44f3-9d07-6ac8a695b490\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7s7kr" Jul 06 15:19:06.573488 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.573437 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/91e0c855-5b32-4ca5-bab6-8e3aae20c1e0-serving-cert\") pod \"console-operator-68b757fb67-mlw9w\" (UID: \"91e0c855-5b32-4ca5-bab6-8e3aae20c1e0\") " pod="openshift-console-operator/console-operator-68b757fb67-mlw9w" Jul 06 15:19:06.573488 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.573481 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdz7h\" (UniqueName: \"kubernetes.io/projected/3f6843dc-43c3-44f3-9d07-6ac8a695b490-kube-api-access-cdz7h\") pod \"service-ca-operator-5d46d9ffd4-7s7kr\" (UID: \"3f6843dc-43c3-44f3-9d07-6ac8a695b490\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7s7kr" Jul 06 15:19:06.573764 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.573528 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4b9bcef7-bd88-424e-b6cb-1125adbb047d-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-jv8z5\" (UID: \"4b9bcef7-bd88-424e-b6cb-1125adbb047d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jv8z5" Jul 06 15:19:06.573764 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.573557 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/91e0c855-5b32-4ca5-bab6-8e3aae20c1e0-trusted-ca\") pod \"console-operator-68b757fb67-mlw9w\" (UID: \"91e0c855-5b32-4ca5-bab6-8e3aae20c1e0\") " pod="openshift-console-operator/console-operator-68b757fb67-mlw9w" Jul 06 15:19:06.573764 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.573597 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/91e0c855-5b32-4ca5-bab6-8e3aae20c1e0-config\") pod \"console-operator-68b757fb67-mlw9w\" (UID: \"91e0c855-5b32-4ca5-bab6-8e3aae20c1e0\") " pod="openshift-console-operator/console-operator-68b757fb67-mlw9w" Jul 06 15:19:06.574097 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.574068 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3f6843dc-43c3-44f3-9d07-6ac8a695b490-config\") pod \"service-ca-operator-5d46d9ffd4-7s7kr\" (UID: \"3f6843dc-43c3-44f3-9d07-6ac8a695b490\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7s7kr" Jul 06 15:19:06.574187 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.574147 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4b9bcef7-bd88-424e-b6cb-1125adbb047d-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-jv8z5\" (UID: \"4b9bcef7-bd88-424e-b6cb-1125adbb047d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jv8z5" Jul 06 15:19:06.574720 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.574694 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/91e0c855-5b32-4ca5-bab6-8e3aae20c1e0-trusted-ca\") pod \"console-operator-68b757fb67-mlw9w\" (UID: \"91e0c855-5b32-4ca5-bab6-8e3aae20c1e0\") " pod="openshift-console-operator/console-operator-68b757fb67-mlw9w" Jul 06 15:19:06.574720 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.574709 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/91e0c855-5b32-4ca5-bab6-8e3aae20c1e0-config\") pod \"console-operator-68b757fb67-mlw9w\" (UID: \"91e0c855-5b32-4ca5-bab6-8e3aae20c1e0\") " pod="openshift-console-operator/console-operator-68b757fb67-mlw9w" Jul 06 15:19:06.575682 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.575657 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4b9bcef7-bd88-424e-b6cb-1125adbb047d-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-jv8z5\" (UID: \"4b9bcef7-bd88-424e-b6cb-1125adbb047d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jv8z5" Jul 06 15:19:06.575972 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.575953 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3f6843dc-43c3-44f3-9d07-6ac8a695b490-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-7s7kr\" (UID: \"3f6843dc-43c3-44f3-9d07-6ac8a695b490\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7s7kr" Jul 06 15:19:06.576038 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.576026 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/91e0c855-5b32-4ca5-bab6-8e3aae20c1e0-serving-cert\") pod \"console-operator-68b757fb67-mlw9w\" (UID: \"91e0c855-5b32-4ca5-bab6-8e3aae20c1e0\") " pod="openshift-console-operator/console-operator-68b757fb67-mlw9w" Jul 06 15:19:06.582426 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.582408 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8pdh\" (UniqueName: \"kubernetes.io/projected/4b9bcef7-bd88-424e-b6cb-1125adbb047d-kube-api-access-z8pdh\") pod \"kube-storage-version-migrator-operator-846bf668f7-jv8z5\" (UID: \"4b9bcef7-bd88-424e-b6cb-1125adbb047d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jv8z5" Jul 06 15:19:06.582555 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.582520 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fltb4\" (UniqueName: \"kubernetes.io/projected/91e0c855-5b32-4ca5-bab6-8e3aae20c1e0-kube-api-access-fltb4\") pod \"console-operator-68b757fb67-mlw9w\" (UID: \"91e0c855-5b32-4ca5-bab6-8e3aae20c1e0\") " pod="openshift-console-operator/console-operator-68b757fb67-mlw9w" Jul 06 15:19:06.582715 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.582692 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdz7h\" (UniqueName: \"kubernetes.io/projected/3f6843dc-43c3-44f3-9d07-6ac8a695b490-kube-api-access-cdz7h\") pod \"service-ca-operator-5d46d9ffd4-7s7kr\" (UID: \"3f6843dc-43c3-44f3-9d07-6ac8a695b490\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7s7kr" Jul 06 15:19:06.674085 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.673989 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7s7kr" Jul 06 15:19:06.680153 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.680135 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-mlw9w" Jul 06 15:19:06.686113 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.686080 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jv8z5" Jul 06 15:19:06.817004 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:06.816894 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7s7kr"] Jul 06 15:19:06.819565 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:19:06.819534 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f6843dc_43c3_44f3_9d07_6ac8a695b490.slice/crio-a6802ccf55bf9719c29add8175fd4315dd16b97417f68e722e534ca0238cd938 WatchSource:0}: Error finding container a6802ccf55bf9719c29add8175fd4315dd16b97417f68e722e534ca0238cd938: Status 404 returned error can't find the container with id a6802ccf55bf9719c29add8175fd4315dd16b97417f68e722e534ca0238cd938 Jul 06 15:19:07.040328 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:07.040252 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jv8z5"] Jul 06 15:19:07.043073 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:19:07.043043 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b9bcef7_bd88_424e_b6cb_1125adbb047d.slice/crio-4a99673a78bd71dc1a2bea51ab457e20e8d80de7796525ffe2a392da745d0a51 WatchSource:0}: Error finding container 4a99673a78bd71dc1a2bea51ab457e20e8d80de7796525ffe2a392da745d0a51: Status 404 returned error can't find the container with id 4a99673a78bd71dc1a2bea51ab457e20e8d80de7796525ffe2a392da745d0a51 Jul 06 15:19:07.043175 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:07.043158 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-mlw9w"] Jul 06 15:19:07.046221 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:19:07.046195 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e0c855_5b32_4ca5_bab6_8e3aae20c1e0.slice/crio-8dc21205d86726cb6089cd1929d576a68351ab31c98ad4423bb29796c3c956ef WatchSource:0}: Error finding container 8dc21205d86726cb6089cd1929d576a68351ab31c98ad4423bb29796c3c956ef: Status 404 returned error can't find the container with id 8dc21205d86726cb6089cd1929d576a68351ab31c98ad4423bb29796c3c956ef Jul 06 15:19:07.130055 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:07.130021 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-mlw9w" event={"ID":"91e0c855-5b32-4ca5-bab6-8e3aae20c1e0","Type":"ContainerStarted","Data":"8dc21205d86726cb6089cd1929d576a68351ab31c98ad4423bb29796c3c956ef"} Jul 06 15:19:07.131031 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:07.131002 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jv8z5" event={"ID":"4b9bcef7-bd88-424e-b6cb-1125adbb047d","Type":"ContainerStarted","Data":"4a99673a78bd71dc1a2bea51ab457e20e8d80de7796525ffe2a392da745d0a51"} Jul 06 15:19:07.131918 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:07.131899 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7s7kr" event={"ID":"3f6843dc-43c3-44f3-9d07-6ac8a695b490","Type":"ContainerStarted","Data":"a6802ccf55bf9719c29add8175fd4315dd16b97417f68e722e534ca0238cd938"} Jul 06 15:19:07.782840 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:07.782802 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92299e93-9bb8-452f-ba3a-523beff0880a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9hm78\" (UID: \"92299e93-9bb8-452f-ba3a-523beff0880a\") " pod="openshift-insights/insights-runtime-extractor-9hm78" Jul 06 15:19:07.782840 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:07.782854 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92299e93-9bb8-452f-ba3a-523beff0880a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9hm78\" (UID: \"92299e93-9bb8-452f-ba3a-523beff0880a\") " pod="openshift-insights/insights-runtime-extractor-9hm78" Jul 06 15:19:07.783306 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:07.782994 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 15:19:07.783306 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:07.783009 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 15:19:07.783306 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:07.783073 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/92299e93-9bb8-452f-ba3a-523beff0880a-kube-rbac-proxy-cm podName:92299e93-9bb8-452f-ba3a-523beff0880a nodeName:}" failed. No retries permitted until 2026-07-06 15:19:15.783056077 +0000 UTC m=+102.396804600 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/92299e93-9bb8-452f-ba3a-523beff0880a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9hm78" (UID: "92299e93-9bb8-452f-ba3a-523beff0880a") : configmap "kube-rbac-proxy" not found Jul 06 15:19:07.783306 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:07.783092 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92299e93-9bb8-452f-ba3a-523beff0880a-insights-runtime-extractor-tls podName:92299e93-9bb8-452f-ba3a-523beff0880a nodeName:}" failed. No retries permitted until 2026-07-06 15:19:15.783083771 +0000 UTC m=+102.396832285 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/92299e93-9bb8-452f-ba3a-523beff0880a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9hm78" (UID: "92299e93-9bb8-452f-ba3a-523beff0880a") : secret "insights-runtime-extractor-tls" not found Jul 06 15:19:08.756369 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:08.756335 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:08.860001 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:08.859960 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:11.145493 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:11.145457 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7s7kr" event={"ID":"3f6843dc-43c3-44f3-9d07-6ac8a695b490","Type":"ContainerStarted","Data":"af33b71f1522f1be4a9ebc979be0ed430539f530cd63d048ec3f683575e79a6a"} Jul 06 15:19:11.147169 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:11.147142 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-mlw9w_91e0c855-5b32-4ca5-bab6-8e3aae20c1e0/console-operator/0.log" Jul 06 15:19:11.147298 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:11.147177 2570 generic.go:356] "Generic (PLEG): container finished" podID="91e0c855-5b32-4ca5-bab6-8e3aae20c1e0" containerID="c2b4bbd547fafcdd25b5479651038f3673fd71ca34165b3d31aa790ed41fb2f4" exitCode=255 Jul 06 15:19:11.147298 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:11.147278 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-mlw9w" event={"ID":"91e0c855-5b32-4ca5-bab6-8e3aae20c1e0","Type":"ContainerDied","Data":"c2b4bbd547fafcdd25b5479651038f3673fd71ca34165b3d31aa790ed41fb2f4"} Jul 06 15:19:11.147475 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:11.147459 2570 scope.go:117] "RemoveContainer" containerID="c2b4bbd547fafcdd25b5479651038f3673fd71ca34165b3d31aa790ed41fb2f4" Jul 06 15:19:11.148850 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:11.148831 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jv8z5" event={"ID":"4b9bcef7-bd88-424e-b6cb-1125adbb047d","Type":"ContainerStarted","Data":"d17c5676876ede92c907aadb3bf5dc1ab5656f02d1a87c2aeba15c7c85ffd563"} Jul 06 15:19:11.165382 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:11.165342 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7s7kr" podStartSLOduration=1.606381906 podStartE2EDuration="5.165325953s" totalImagesPullingTime="3.558944047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:19:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:19:11.164914313 +0000 UTC m=+97.778662852" watchObservedRunningTime="2026-07-06 15:19:11.165325953 +0000 UTC m=+97.779074489" Jul 06 15:19:11.240446 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:11.240390 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jv8z5" podStartSLOduration=1.908761889 podStartE2EDuration="5.240367409s" totalImagesPullingTime="3.33160552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:19:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:19:11.239901937 +0000 UTC m=+97.853650475" watchObservedRunningTime="2026-07-06 15:19:11.240367409 +0000 UTC m=+97.854115946" Jul 06 15:19:12.086627 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:12.086578 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:12.089670 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:12.089646 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-jg2tk"] Jul 06 15:19:12.092173 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:12.092157 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-jg2tk" Jul 06 15:19:12.095105 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:12.095079 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 06 15:19:12.095229 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:12.095080 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-pscfl\"" Jul 06 15:19:12.096165 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:12.096146 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 06 15:19:12.104095 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:12.104076 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-jg2tk"] Jul 06 15:19:12.155252 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:12.155226 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-mlw9w_91e0c855-5b32-4ca5-bab6-8e3aae20c1e0/console-operator/1.log" Jul 06 15:19:12.155699 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:12.155679 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-mlw9w_91e0c855-5b32-4ca5-bab6-8e3aae20c1e0/console-operator/0.log" Jul 06 15:19:12.155742 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:12.155714 2570 generic.go:356] "Generic (PLEG): container finished" podID="91e0c855-5b32-4ca5-bab6-8e3aae20c1e0" containerID="3ce60f5db53c7c25764dc790829343ac26115a84aebfa9bacfcd8f11faab3c7e" exitCode=255 Jul 06 15:19:12.155820 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:12.155794 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-mlw9w" event={"ID":"91e0c855-5b32-4ca5-bab6-8e3aae20c1e0","Type":"ContainerDied","Data":"3ce60f5db53c7c25764dc790829343ac26115a84aebfa9bacfcd8f11faab3c7e"} Jul 06 15:19:12.155950 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:12.155856 2570 scope.go:117] "RemoveContainer" containerID="c2b4bbd547fafcdd25b5479651038f3673fd71ca34165b3d31aa790ed41fb2f4" Jul 06 15:19:12.156180 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:12.156161 2570 scope.go:117] "RemoveContainer" containerID="3ce60f5db53c7c25764dc790829343ac26115a84aebfa9bacfcd8f11faab3c7e" Jul 06 15:19:12.156571 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:12.156348 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-mlw9w_openshift-console-operator(91e0c855-5b32-4ca5-bab6-8e3aae20c1e0)\"" pod="openshift-console-operator/console-operator-68b757fb67-mlw9w" podUID="91e0c855-5b32-4ca5-bab6-8e3aae20c1e0" Jul 06 15:19:12.183285 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:12.183253 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:12.222440 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:12.222410 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj5h9\" (UniqueName: \"kubernetes.io/projected/b8c9d558-02e2-4906-8480-62d7ac3bd357-kube-api-access-bj5h9\") pod \"migrator-d57d58cc9-jg2tk\" (UID: \"b8c9d558-02e2-4906-8480-62d7ac3bd357\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-jg2tk" Jul 06 15:19:12.222644 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:12.222483 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-service-ca-bundle\") pod \"router-default-55896fdd7f-l5sg2\" (UID: \"d00f67b5-9c7a-41f4-bd9e-2160fcd8641a\") " pod="openshift-ingress/router-default-55896fdd7f-l5sg2" Jul 06 15:19:12.222644 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:12.222576 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-metrics-certs\") pod \"router-default-55896fdd7f-l5sg2\" (UID: \"d00f67b5-9c7a-41f4-bd9e-2160fcd8641a\") " pod="openshift-ingress/router-default-55896fdd7f-l5sg2" Jul 06 15:19:12.222774 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:12.222668 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-service-ca-bundle podName:d00f67b5-9c7a-41f4-bd9e-2160fcd8641a nodeName:}" failed. No retries permitted until 2026-07-06 15:19:28.222614793 +0000 UTC m=+114.836363307 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-service-ca-bundle") pod "router-default-55896fdd7f-l5sg2" (UID: "d00f67b5-9c7a-41f4-bd9e-2160fcd8641a") : configmap references non-existent config key: service-ca.crt Jul 06 15:19:12.222774 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:12.222685 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 15:19:12.222956 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:12.222930 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-metrics-certs podName:d00f67b5-9c7a-41f4-bd9e-2160fcd8641a nodeName:}" failed. No retries permitted until 2026-07-06 15:19:28.222900511 +0000 UTC m=+114.836649038 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-metrics-certs") pod "router-default-55896fdd7f-l5sg2" (UID: "d00f67b5-9c7a-41f4-bd9e-2160fcd8641a") : secret "router-metrics-certs-default" not found Jul 06 15:19:12.279762 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:12.279729 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:12.324224 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:12.324194 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj5h9\" (UniqueName: \"kubernetes.io/projected/b8c9d558-02e2-4906-8480-62d7ac3bd357-kube-api-access-bj5h9\") pod \"migrator-d57d58cc9-jg2tk\" (UID: \"b8c9d558-02e2-4906-8480-62d7ac3bd357\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-jg2tk" Jul 06 15:19:12.333161 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:12.333129 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj5h9\" (UniqueName: \"kubernetes.io/projected/b8c9d558-02e2-4906-8480-62d7ac3bd357-kube-api-access-bj5h9\") pod \"migrator-d57d58cc9-jg2tk\" (UID: \"b8c9d558-02e2-4906-8480-62d7ac3bd357\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-jg2tk" Jul 06 15:19:12.380899 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:12.380818 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:12.401621 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:12.401594 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-jg2tk" Jul 06 15:19:12.525807 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:12.525776 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-jg2tk"] Jul 06 15:19:12.528790 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:19:12.528759 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c9d558_02e2_4906_8480_62d7ac3bd357.slice/crio-ef726d9fbde16d8f9d7e41f740dc0579f7b5dda15c532336c6206419ed9446d9 WatchSource:0}: Error finding container ef726d9fbde16d8f9d7e41f740dc0579f7b5dda15c532336c6206419ed9446d9: Status 404 returned error can't find the container with id ef726d9fbde16d8f9d7e41f740dc0579f7b5dda15c532336c6206419ed9446d9 Jul 06 15:19:12.538552 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:12.538532 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:12.602827 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:12.602782 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:13.011789 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:13.011762 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:13.114671 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:13.114632 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:13.161663 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:13.161635 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-mlw9w_91e0c855-5b32-4ca5-bab6-8e3aae20c1e0/console-operator/1.log" Jul 06 15:19:13.162125 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:13.162105 2570 scope.go:117] "RemoveContainer" containerID="3ce60f5db53c7c25764dc790829343ac26115a84aebfa9bacfcd8f11faab3c7e" Jul 06 15:19:13.162313 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:13.162290 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-mlw9w_openshift-console-operator(91e0c855-5b32-4ca5-bab6-8e3aae20c1e0)\"" pod="openshift-console-operator/console-operator-68b757fb67-mlw9w" podUID="91e0c855-5b32-4ca5-bab6-8e3aae20c1e0" Jul 06 15:19:13.162775 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:13.162743 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-jg2tk" event={"ID":"b8c9d558-02e2-4906-8480-62d7ac3bd357","Type":"ContainerStarted","Data":"ef726d9fbde16d8f9d7e41f740dc0579f7b5dda15c532336c6206419ed9446d9"} Jul 06 15:19:13.259997 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:13.259960 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:14.170320 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:14.170287 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-jg2tk" event={"ID":"b8c9d558-02e2-4906-8480-62d7ac3bd357","Type":"ContainerStarted","Data":"5139cc6d7a944f7d6ea49c74ea74c7a1452fbcda920ecffb710978faf7cd3a27"} Jul 06 15:19:14.170719 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:14.170345 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-jg2tk" event={"ID":"b8c9d558-02e2-4906-8480-62d7ac3bd357","Type":"ContainerStarted","Data":"cf5db0b41a82c7fc2ad40537c8e1eefebcf59966da4afc70b33d7973c3076bef"} Jul 06 15:19:14.186974 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:14.186884 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-jg2tk" podStartSLOduration=0.780546485 podStartE2EDuration="2.186866228s" totalImagesPullingTime="1.406319743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:19:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:19:14.186671268 +0000 UTC m=+100.800419805" watchObservedRunningTime="2026-07-06 15:19:14.186866228 +0000 UTC m=+100.800614766" Jul 06 15:19:14.266106 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:14.266067 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:14.367224 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:14.367181 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:14.652819 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:14.652782 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:14.720918 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:14.720873 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:15.788950 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:15.788912 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:15.855187 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:15.855154 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92299e93-9bb8-452f-ba3a-523beff0880a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9hm78\" (UID: \"92299e93-9bb8-452f-ba3a-523beff0880a\") " pod="openshift-insights/insights-runtime-extractor-9hm78" Jul 06 15:19:15.855339 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:15.855321 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92299e93-9bb8-452f-ba3a-523beff0880a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9hm78\" (UID: \"92299e93-9bb8-452f-ba3a-523beff0880a\") " pod="openshift-insights/insights-runtime-extractor-9hm78" Jul 06 15:19:15.855436 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:15.855423 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 15:19:15.855521 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:15.855479 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92299e93-9bb8-452f-ba3a-523beff0880a-insights-runtime-extractor-tls podName:92299e93-9bb8-452f-ba3a-523beff0880a nodeName:}" failed. No retries permitted until 2026-07-06 15:19:31.855462319 +0000 UTC m=+118.469210851 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/92299e93-9bb8-452f-ba3a-523beff0880a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9hm78" (UID: "92299e93-9bb8-452f-ba3a-523beff0880a") : secret "insights-runtime-extractor-tls" not found Jul 06 15:19:15.855776 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:15.855751 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92299e93-9bb8-452f-ba3a-523beff0880a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9hm78\" (UID: \"92299e93-9bb8-452f-ba3a-523beff0880a\") " pod="openshift-insights/insights-runtime-extractor-9hm78" Jul 06 15:19:16.680458 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:16.680408 2570 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-mlw9w" Jul 06 15:19:16.680655 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:16.680470 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-mlw9w" Jul 06 15:19:16.680881 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:16.680867 2570 scope.go:117] "RemoveContainer" containerID="3ce60f5db53c7c25764dc790829343ac26115a84aebfa9bacfcd8f11faab3c7e" Jul 06 15:19:16.681024 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:16.681006 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-mlw9w_openshift-console-operator(91e0c855-5b32-4ca5-bab6-8e3aae20c1e0)\"" pod="openshift-console-operator/console-operator-68b757fb67-mlw9w" podUID="91e0c855-5b32-4ca5-bab6-8e3aae20c1e0" Jul 06 15:19:17.081094 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:17.081066 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-v9nsl" Jul 06 15:19:19.036441 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:19.036404 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:19.136147 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:19.136081 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:19.236808 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:19.236777 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:19.338958 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:19.338929 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:19.465226 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:19.465193 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:19.743994 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:19.743912 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:19.809541 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:19.809490 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:19.886348 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:19.886313 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:20.090825 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:20.090795 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:21.290658 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:21.290607 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:21.488421 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:21.488387 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:21.693793 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:21.693766 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:21.881898 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:21.881863 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:22.085611 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:22.085575 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:22.805249 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:22.805223 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:22.905215 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:22.905184 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:23.283201 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:23.283160 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:23.484079 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:23.484050 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:23.681916 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:23.681882 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:23.747006 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:23.746971 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:23.880444 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:23.880407 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:24.080870 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:24.080834 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:24.423261 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:24.423178 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ce904e6-fad6-4078-ab8b-1a45fab198a9-metrics-tls\") pod \"dns-default-8lh8v\" (UID: \"1ce904e6-fad6-4078-ab8b-1a45fab198a9\") " pod="openshift-dns/dns-default-8lh8v" Jul 06 15:19:24.425404 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:24.425378 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ce904e6-fad6-4078-ab8b-1a45fab198a9-metrics-tls\") pod \"dns-default-8lh8v\" (UID: \"1ce904e6-fad6-4078-ab8b-1a45fab198a9\") " pod="openshift-dns/dns-default-8lh8v" Jul 06 15:19:24.728032 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:24.727945 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8l7xg\"" Jul 06 15:19:24.733635 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:24.733618 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8lh8v" Jul 06 15:19:24.861300 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:24.861271 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8lh8v"] Jul 06 15:19:24.864945 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:19:24.864921 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce904e6_fad6_4078_ab8b_1a45fab198a9.slice/crio-0ac7d2ee7492147890d8c70d0dbe7102b9927b10c2f933d9d1e3fd1b204b64e4 WatchSource:0}: Error finding container 0ac7d2ee7492147890d8c70d0dbe7102b9927b10c2f933d9d1e3fd1b204b64e4: Status 404 returned error can't find the container with id 0ac7d2ee7492147890d8c70d0dbe7102b9927b10c2f933d9d1e3fd1b204b64e4 Jul 06 15:19:25.124062 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:25.124030 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:25.205878 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:25.205843 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8lh8v" event={"ID":"1ce904e6-fad6-4078-ab8b-1a45fab198a9","Type":"ContainerStarted","Data":"0ac7d2ee7492147890d8c70d0dbe7102b9927b10c2f933d9d1e3fd1b204b64e4"} Jul 06 15:19:25.206847 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:25.206824 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:25.492277 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:25.492185 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:25.680620 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:25.680593 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:25.885863 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:25.885825 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:26.086582 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:26.086543 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:26.880144 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:26.880109 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:26.989406 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:26.989374 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:27.216450 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:27.216412 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8lh8v" event={"ID":"1ce904e6-fad6-4078-ab8b-1a45fab198a9","Type":"ContainerStarted","Data":"d752f56a8dae5f2952620c24cfb936bf88a967671afc3295febecff18626ce10"} Jul 06 15:19:27.485159 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:27.485066 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:27.719951 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:27.719920 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:28.091127 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:28.091096 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:28.220691 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:28.220657 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8lh8v" event={"ID":"1ce904e6-fad6-4078-ab8b-1a45fab198a9","Type":"ContainerStarted","Data":"0e7d94296f9321bb72d52de3f2cae6d5246e7196264ac89150e9e1020f02a35a"} Jul 06 15:19:28.220861 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:28.220759 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8lh8v" Jul 06 15:19:28.243048 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:28.243000 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8lh8v" podStartSLOduration=66.008879208 podStartE2EDuration="1m8.242982541s" totalImagesPullingTime="2.234103333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:19:28.242344396 +0000 UTC m=+114.856092932" watchObservedRunningTime="2026-07-06 15:19:28.242982541 +0000 UTC m=+114.856731080" Jul 06 15:19:28.256756 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:28.256728 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-service-ca-bundle\") pod \"router-default-55896fdd7f-l5sg2\" (UID: \"d00f67b5-9c7a-41f4-bd9e-2160fcd8641a\") " pod="openshift-ingress/router-default-55896fdd7f-l5sg2" Jul 06 15:19:28.256891 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:28.256773 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-metrics-certs\") pod \"router-default-55896fdd7f-l5sg2\" (UID: \"d00f67b5-9c7a-41f4-bd9e-2160fcd8641a\") " pod="openshift-ingress/router-default-55896fdd7f-l5sg2" Jul 06 15:19:28.257585 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:28.257561 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-service-ca-bundle\") pod \"router-default-55896fdd7f-l5sg2\" (UID: \"d00f67b5-9c7a-41f4-bd9e-2160fcd8641a\") " pod="openshift-ingress/router-default-55896fdd7f-l5sg2" Jul 06 15:19:28.259013 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:28.258996 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d00f67b5-9c7a-41f4-bd9e-2160fcd8641a-metrics-certs\") pod \"router-default-55896fdd7f-l5sg2\" (UID: \"d00f67b5-9c7a-41f4-bd9e-2160fcd8641a\") " pod="openshift-ingress/router-default-55896fdd7f-l5sg2" Jul 06 15:19:28.296893 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:28.296858 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:28.548158 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:28.548072 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-55896fdd7f-l5sg2" Jul 06 15:19:28.685676 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:28.685647 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-55896fdd7f-l5sg2"] Jul 06 15:19:28.689542 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:19:28.689494 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd00f67b5_9c7a_41f4_bd9e_2160fcd8641a.slice/crio-8401ae5b9b00332f75a3d3197c48e71c657ba6cb13e4e743660e200d95ee2601 WatchSource:0}: Error finding container 8401ae5b9b00332f75a3d3197c48e71c657ba6cb13e4e743660e200d95ee2601: Status 404 returned error can't find the container with id 8401ae5b9b00332f75a3d3197c48e71c657ba6cb13e4e743660e200d95ee2601 Jul 06 15:19:28.882467 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:28.882436 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:29.225733 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:29.225639 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-55896fdd7f-l5sg2" event={"ID":"d00f67b5-9c7a-41f4-bd9e-2160fcd8641a","Type":"ContainerStarted","Data":"384ef6d0b0a569da489e601f2e3465ff9a707a1ff3ca256db29c807072c7e832"} Jul 06 15:19:29.225733 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:29.225698 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-55896fdd7f-l5sg2" event={"ID":"d00f67b5-9c7a-41f4-bd9e-2160fcd8641a","Type":"ContainerStarted","Data":"8401ae5b9b00332f75a3d3197c48e71c657ba6cb13e4e743660e200d95ee2601"} Jul 06 15:19:29.251168 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:29.251114 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-55896fdd7f-l5sg2" podStartSLOduration=33.251096777 podStartE2EDuration="33.251096777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:18:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:19:29.250895737 +0000 UTC m=+115.864644275" watchObservedRunningTime="2026-07-06 15:19:29.251096777 +0000 UTC m=+115.864845314" Jul 06 15:19:29.291859 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:29.291823 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:29.499491 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:29.499408 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:29.548990 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:29.548955 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-55896fdd7f-l5sg2" Jul 06 15:19:29.551450 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:29.551425 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-55896fdd7f-l5sg2" Jul 06 15:19:29.688660 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:29.688635 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:29.794103 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:29.794028 2570 scope.go:117] "RemoveContainer" containerID="3ce60f5db53c7c25764dc790829343ac26115a84aebfa9bacfcd8f11faab3c7e" Jul 06 15:19:30.230671 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:30.230638 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-mlw9w_91e0c855-5b32-4ca5-bab6-8e3aae20c1e0/console-operator/2.log" Jul 06 15:19:30.231122 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:30.231055 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-mlw9w_91e0c855-5b32-4ca5-bab6-8e3aae20c1e0/console-operator/1.log" Jul 06 15:19:30.231122 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:30.231084 2570 generic.go:356] "Generic (PLEG): container finished" podID="91e0c855-5b32-4ca5-bab6-8e3aae20c1e0" containerID="277d1ced7b6e19f97050eea3d954e8cdde6d45dc00127883dfbc753f3959ea60" exitCode=255 Jul 06 15:19:30.231194 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:30.231163 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-mlw9w" event={"ID":"91e0c855-5b32-4ca5-bab6-8e3aae20c1e0","Type":"ContainerDied","Data":"277d1ced7b6e19f97050eea3d954e8cdde6d45dc00127883dfbc753f3959ea60"} Jul 06 15:19:30.231231 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:30.231213 2570 scope.go:117] "RemoveContainer" containerID="3ce60f5db53c7c25764dc790829343ac26115a84aebfa9bacfcd8f11faab3c7e" Jul 06 15:19:30.231427 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:30.231404 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-55896fdd7f-l5sg2" Jul 06 15:19:30.231594 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:30.231573 2570 scope.go:117] "RemoveContainer" containerID="277d1ced7b6e19f97050eea3d954e8cdde6d45dc00127883dfbc753f3959ea60" Jul 06 15:19:30.231764 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:30.231737 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-mlw9w_openshift-console-operator(91e0c855-5b32-4ca5-bab6-8e3aae20c1e0)\"" pod="openshift-console-operator/console-operator-68b757fb67-mlw9w" podUID="91e0c855-5b32-4ca5-bab6-8e3aae20c1e0" Jul 06 15:19:30.232791 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:30.232775 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-55896fdd7f-l5sg2" Jul 06 15:19:31.235308 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:31.235280 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-mlw9w_91e0c855-5b32-4ca5-bab6-8e3aae20c1e0/console-operator/2.log" Jul 06 15:19:31.602851 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:31.602818 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:31.700319 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:31.700258 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:31.849735 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:31.849702 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:31.887828 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:31.887753 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92299e93-9bb8-452f-ba3a-523beff0880a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9hm78\" (UID: \"92299e93-9bb8-452f-ba3a-523beff0880a\") " pod="openshift-insights/insights-runtime-extractor-9hm78" Jul 06 15:19:31.889965 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:31.889941 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92299e93-9bb8-452f-ba3a-523beff0880a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9hm78\" (UID: \"92299e93-9bb8-452f-ba3a-523beff0880a\") " pod="openshift-insights/insights-runtime-extractor-9hm78" Jul 06 15:19:32.122183 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:32.122142 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9hm78" Jul 06 15:19:32.245050 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:32.245021 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9hm78"] Jul 06 15:19:32.247698 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:19:32.247676 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92299e93_9bb8_452f_ba3a_523beff0880a.slice/crio-4fefa6ddafe1ea28823f085b74dc295cb0cd5906f08f221e6772703ee934645b WatchSource:0}: Error finding container 4fefa6ddafe1ea28823f085b74dc295cb0cd5906f08f221e6772703ee934645b: Status 404 returned error can't find the container with id 4fefa6ddafe1ea28823f085b74dc295cb0cd5906f08f221e6772703ee934645b Jul 06 15:19:32.436421 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:32.436322 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:33.243142 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:33.243101 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9hm78" event={"ID":"92299e93-9bb8-452f-ba3a-523beff0880a","Type":"ContainerStarted","Data":"4fefa6ddafe1ea28823f085b74dc295cb0cd5906f08f221e6772703ee934645b"} Jul 06 15:19:34.738601 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:34.738538 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:34.799407 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:34.799370 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:35.836825 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:35.836789 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-l2s4b"] Jul 06 15:19:35.842022 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:35.841996 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-l2s4b" Jul 06 15:19:35.855248 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:35.855222 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-8d4tk\"" Jul 06 15:19:35.868542 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:35.868519 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-l2s4b"] Jul 06 15:19:36.019920 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.019828 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcxhb\" (UniqueName: \"kubernetes.io/projected/42ec6d80-3b4d-45d6-aef1-afc442857841-kube-api-access-bcxhb\") pod \"network-check-source-989f8b8dc-l2s4b\" (UID: \"42ec6d80-3b4d-45d6-aef1-afc442857841\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-l2s4b" Jul 06 15:19:36.020925 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.020888 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:36.041435 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.041403 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-695c8f4bdb-jm29j"] Jul 06 15:19:36.043729 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.043708 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-695c8f4bdb-jm29j" Jul 06 15:19:36.048338 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.048312 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 15:19:36.048542 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.048518 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7t2f5\"" Jul 06 15:19:36.048634 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.048557 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 15:19:36.048634 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.048566 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 15:19:36.055212 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.055191 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 15:19:36.068020 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.067993 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-695c8f4bdb-jm29j"] Jul 06 15:19:36.121008 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.120969 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcxhb\" (UniqueName: \"kubernetes.io/projected/42ec6d80-3b4d-45d6-aef1-afc442857841-kube-api-access-bcxhb\") pod \"network-check-source-989f8b8dc-l2s4b\" (UID: \"42ec6d80-3b4d-45d6-aef1-afc442857841\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-l2s4b" Jul 06 15:19:36.140583 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.140557 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcxhb\" (UniqueName: \"kubernetes.io/projected/42ec6d80-3b4d-45d6-aef1-afc442857841-kube-api-access-bcxhb\") pod \"network-check-source-989f8b8dc-l2s4b\" (UID: \"42ec6d80-3b4d-45d6-aef1-afc442857841\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-l2s4b" Jul 06 15:19:36.152305 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.152269 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-l2s4b" Jul 06 15:19:36.179555 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.179523 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:36.221813 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.221770 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f993611f-b1fa-4b2f-8173-2e78ca6dde08-ca-trust-extracted\") pod \"image-registry-695c8f4bdb-jm29j\" (UID: \"f993611f-b1fa-4b2f-8173-2e78ca6dde08\") " pod="openshift-image-registry/image-registry-695c8f4bdb-jm29j" Jul 06 15:19:36.221967 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.221828 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f993611f-b1fa-4b2f-8173-2e78ca6dde08-image-registry-private-configuration\") pod \"image-registry-695c8f4bdb-jm29j\" (UID: \"f993611f-b1fa-4b2f-8173-2e78ca6dde08\") " pod="openshift-image-registry/image-registry-695c8f4bdb-jm29j" Jul 06 15:19:36.221967 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.221889 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f993611f-b1fa-4b2f-8173-2e78ca6dde08-trusted-ca\") pod \"image-registry-695c8f4bdb-jm29j\" (UID: \"f993611f-b1fa-4b2f-8173-2e78ca6dde08\") " pod="openshift-image-registry/image-registry-695c8f4bdb-jm29j" Jul 06 15:19:36.221967 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.221917 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f993611f-b1fa-4b2f-8173-2e78ca6dde08-registry-certificates\") pod \"image-registry-695c8f4bdb-jm29j\" (UID: \"f993611f-b1fa-4b2f-8173-2e78ca6dde08\") " pod="openshift-image-registry/image-registry-695c8f4bdb-jm29j" Jul 06 15:19:36.221967 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.221960 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jvtl\" (UniqueName: \"kubernetes.io/projected/f993611f-b1fa-4b2f-8173-2e78ca6dde08-kube-api-access-9jvtl\") pod \"image-registry-695c8f4bdb-jm29j\" (UID: \"f993611f-b1fa-4b2f-8173-2e78ca6dde08\") " pod="openshift-image-registry/image-registry-695c8f4bdb-jm29j" Jul 06 15:19:36.222136 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.221992 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f993611f-b1fa-4b2f-8173-2e78ca6dde08-registry-tls\") pod \"image-registry-695c8f4bdb-jm29j\" (UID: \"f993611f-b1fa-4b2f-8173-2e78ca6dde08\") " pod="openshift-image-registry/image-registry-695c8f4bdb-jm29j" Jul 06 15:19:36.222136 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.222018 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f993611f-b1fa-4b2f-8173-2e78ca6dde08-bound-sa-token\") pod \"image-registry-695c8f4bdb-jm29j\" (UID: \"f993611f-b1fa-4b2f-8173-2e78ca6dde08\") " pod="openshift-image-registry/image-registry-695c8f4bdb-jm29j" Jul 06 15:19:36.222136 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.222058 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f993611f-b1fa-4b2f-8173-2e78ca6dde08-installation-pull-secrets\") pod \"image-registry-695c8f4bdb-jm29j\" (UID: \"f993611f-b1fa-4b2f-8173-2e78ca6dde08\") " pod="openshift-image-registry/image-registry-695c8f4bdb-jm29j" Jul 06 15:19:36.256106 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.256069 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9hm78" event={"ID":"92299e93-9bb8-452f-ba3a-523beff0880a","Type":"ContainerStarted","Data":"d1a44e79f5599ba0d4c17eadec91c01f4a599c7dd2f271658b105beba6c0546e"} Jul 06 15:19:36.276860 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.276790 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-l2s4b"] Jul 06 15:19:36.279825 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:19:36.279786 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ec6d80_3b4d_45d6_aef1_afc442857841.slice/crio-81d6c5fb0cbc86af655bd6e4aa32547f8a029e4300b6b35e3c9474caf56f6951 WatchSource:0}: Error finding container 81d6c5fb0cbc86af655bd6e4aa32547f8a029e4300b6b35e3c9474caf56f6951: Status 404 returned error can't find the container with id 81d6c5fb0cbc86af655bd6e4aa32547f8a029e4300b6b35e3c9474caf56f6951 Jul 06 15:19:36.322954 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.322927 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f993611f-b1fa-4b2f-8173-2e78ca6dde08-registry-certificates\") pod \"image-registry-695c8f4bdb-jm29j\" (UID: \"f993611f-b1fa-4b2f-8173-2e78ca6dde08\") " pod="openshift-image-registry/image-registry-695c8f4bdb-jm29j" Jul 06 15:19:36.323097 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.322962 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jvtl\" (UniqueName: \"kubernetes.io/projected/f993611f-b1fa-4b2f-8173-2e78ca6dde08-kube-api-access-9jvtl\") pod \"image-registry-695c8f4bdb-jm29j\" (UID: \"f993611f-b1fa-4b2f-8173-2e78ca6dde08\") " pod="openshift-image-registry/image-registry-695c8f4bdb-jm29j" Jul 06 15:19:36.323097 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.322979 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f993611f-b1fa-4b2f-8173-2e78ca6dde08-registry-tls\") pod \"image-registry-695c8f4bdb-jm29j\" (UID: \"f993611f-b1fa-4b2f-8173-2e78ca6dde08\") " pod="openshift-image-registry/image-registry-695c8f4bdb-jm29j" Jul 06 15:19:36.323097 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.322996 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f993611f-b1fa-4b2f-8173-2e78ca6dde08-bound-sa-token\") pod \"image-registry-695c8f4bdb-jm29j\" (UID: \"f993611f-b1fa-4b2f-8173-2e78ca6dde08\") " pod="openshift-image-registry/image-registry-695c8f4bdb-jm29j" Jul 06 15:19:36.323097 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.323022 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f993611f-b1fa-4b2f-8173-2e78ca6dde08-installation-pull-secrets\") pod \"image-registry-695c8f4bdb-jm29j\" (UID: \"f993611f-b1fa-4b2f-8173-2e78ca6dde08\") " pod="openshift-image-registry/image-registry-695c8f4bdb-jm29j" Jul 06 15:19:36.323321 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.323098 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f993611f-b1fa-4b2f-8173-2e78ca6dde08-ca-trust-extracted\") pod \"image-registry-695c8f4bdb-jm29j\" (UID: \"f993611f-b1fa-4b2f-8173-2e78ca6dde08\") " pod="openshift-image-registry/image-registry-695c8f4bdb-jm29j" Jul 06 15:19:36.323321 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.323121 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f993611f-b1fa-4b2f-8173-2e78ca6dde08-image-registry-private-configuration\") pod \"image-registry-695c8f4bdb-jm29j\" (UID: \"f993611f-b1fa-4b2f-8173-2e78ca6dde08\") " pod="openshift-image-registry/image-registry-695c8f4bdb-jm29j" Jul 06 15:19:36.323321 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.323169 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f993611f-b1fa-4b2f-8173-2e78ca6dde08-trusted-ca\") pod \"image-registry-695c8f4bdb-jm29j\" (UID: \"f993611f-b1fa-4b2f-8173-2e78ca6dde08\") " pod="openshift-image-registry/image-registry-695c8f4bdb-jm29j" Jul 06 15:19:36.323548 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.323523 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f993611f-b1fa-4b2f-8173-2e78ca6dde08-ca-trust-extracted\") pod \"image-registry-695c8f4bdb-jm29j\" (UID: \"f993611f-b1fa-4b2f-8173-2e78ca6dde08\") " pod="openshift-image-registry/image-registry-695c8f4bdb-jm29j" Jul 06 15:19:36.323824 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.323790 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f993611f-b1fa-4b2f-8173-2e78ca6dde08-registry-certificates\") pod \"image-registry-695c8f4bdb-jm29j\" (UID: \"f993611f-b1fa-4b2f-8173-2e78ca6dde08\") " pod="openshift-image-registry/image-registry-695c8f4bdb-jm29j" Jul 06 15:19:36.324275 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.324249 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f993611f-b1fa-4b2f-8173-2e78ca6dde08-trusted-ca\") pod \"image-registry-695c8f4bdb-jm29j\" (UID: \"f993611f-b1fa-4b2f-8173-2e78ca6dde08\") " pod="openshift-image-registry/image-registry-695c8f4bdb-jm29j" Jul 06 15:19:36.325449 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.325427 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f993611f-b1fa-4b2f-8173-2e78ca6dde08-image-registry-private-configuration\") pod \"image-registry-695c8f4bdb-jm29j\" (UID: \"f993611f-b1fa-4b2f-8173-2e78ca6dde08\") " pod="openshift-image-registry/image-registry-695c8f4bdb-jm29j" Jul 06 15:19:36.325545 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.325522 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f993611f-b1fa-4b2f-8173-2e78ca6dde08-registry-tls\") pod \"image-registry-695c8f4bdb-jm29j\" (UID: \"f993611f-b1fa-4b2f-8173-2e78ca6dde08\") " pod="openshift-image-registry/image-registry-695c8f4bdb-jm29j" Jul 06 15:19:36.325583 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.325560 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f993611f-b1fa-4b2f-8173-2e78ca6dde08-installation-pull-secrets\") pod \"image-registry-695c8f4bdb-jm29j\" (UID: \"f993611f-b1fa-4b2f-8173-2e78ca6dde08\") " pod="openshift-image-registry/image-registry-695c8f4bdb-jm29j" Jul 06 15:19:36.332520 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.332478 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f993611f-b1fa-4b2f-8173-2e78ca6dde08-bound-sa-token\") pod \"image-registry-695c8f4bdb-jm29j\" (UID: \"f993611f-b1fa-4b2f-8173-2e78ca6dde08\") " pod="openshift-image-registry/image-registry-695c8f4bdb-jm29j" Jul 06 15:19:36.332769 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.332751 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jvtl\" (UniqueName: \"kubernetes.io/projected/f993611f-b1fa-4b2f-8173-2e78ca6dde08-kube-api-access-9jvtl\") pod \"image-registry-695c8f4bdb-jm29j\" (UID: \"f993611f-b1fa-4b2f-8173-2e78ca6dde08\") " pod="openshift-image-registry/image-registry-695c8f4bdb-jm29j" Jul 06 15:19:36.352629 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.352598 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-695c8f4bdb-jm29j" Jul 06 15:19:36.492152 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.492113 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-695c8f4bdb-jm29j"] Jul 06 15:19:36.523737 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:19:36.523703 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf993611f_b1fa_4b2f_8173_2e78ca6dde08.slice/crio-410e85f979a00e830324a3fae72deab2fa4b89fed5314bc1d8d68ec79593faa7 WatchSource:0}: Error finding container 410e85f979a00e830324a3fae72deab2fa4b89fed5314bc1d8d68ec79593faa7: Status 404 returned error can't find the container with id 410e85f979a00e830324a3fae72deab2fa4b89fed5314bc1d8d68ec79593faa7 Jul 06 15:19:36.680389 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.680353 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-mlw9w" Jul 06 15:19:36.680560 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.680429 2570 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-mlw9w" Jul 06 15:19:36.680841 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:36.680827 2570 scope.go:117] "RemoveContainer" containerID="277d1ced7b6e19f97050eea3d954e8cdde6d45dc00127883dfbc753f3959ea60" Jul 06 15:19:36.681034 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:36.681013 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-mlw9w_openshift-console-operator(91e0c855-5b32-4ca5-bab6-8e3aae20c1e0)\"" pod="openshift-console-operator/console-operator-68b757fb67-mlw9w" podUID="91e0c855-5b32-4ca5-bab6-8e3aae20c1e0" Jul 06 15:19:37.261888 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:37.261851 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-l2s4b" event={"ID":"42ec6d80-3b4d-45d6-aef1-afc442857841","Type":"ContainerStarted","Data":"ccb9a1243aff6eee312e9a179f3b719430703b7dd22e2592ea0f011446ad8365"} Jul 06 15:19:37.262373 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:37.261914 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-l2s4b" event={"ID":"42ec6d80-3b4d-45d6-aef1-afc442857841","Type":"ContainerStarted","Data":"81d6c5fb0cbc86af655bd6e4aa32547f8a029e4300b6b35e3c9474caf56f6951"} Jul 06 15:19:37.264160 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:37.264132 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9hm78" event={"ID":"92299e93-9bb8-452f-ba3a-523beff0880a","Type":"ContainerStarted","Data":"8a411a180dff5679fa443c3f8a05db77c88be73c68fa3794559b06f3f7e8400a"} Jul 06 15:19:37.265930 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:37.265903 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-695c8f4bdb-jm29j" event={"ID":"f993611f-b1fa-4b2f-8173-2e78ca6dde08","Type":"ContainerStarted","Data":"bb96eed96e153b46aa1db09fa109b67aceb82fa33db82880dc536ca6fa755b15"} Jul 06 15:19:37.266050 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:37.265947 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-695c8f4bdb-jm29j" event={"ID":"f993611f-b1fa-4b2f-8173-2e78ca6dde08","Type":"ContainerStarted","Data":"410e85f979a00e830324a3fae72deab2fa4b89fed5314bc1d8d68ec79593faa7"} Jul 06 15:19:37.266120 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:37.266068 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-695c8f4bdb-jm29j" Jul 06 15:19:37.301569 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:37.301493 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-l2s4b" podStartSLOduration=2.301470885 podStartE2EDuration="2.301470885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:19:37.300483327 +0000 UTC m=+123.914231864" watchObservedRunningTime="2026-07-06 15:19:37.301470885 +0000 UTC m=+123.915219423" Jul 06 15:19:37.332887 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:37.332831 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-695c8f4bdb-jm29j" podStartSLOduration=1.332813752 podStartE2EDuration="1.332813752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:19:37.331272801 +0000 UTC m=+123.945021338" watchObservedRunningTime="2026-07-06 15:19:37.332813752 +0000 UTC m=+123.946562287" Jul 06 15:19:38.228464 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:38.228432 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8lh8v" Jul 06 15:19:38.866678 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:38.866644 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:38.955719 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:38.955687 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:39.028224 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:39.028193 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:39.128353 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:39.128264 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:40.278518 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:40.278474 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9hm78" event={"ID":"92299e93-9bb8-452f-ba3a-523beff0880a","Type":"ContainerStarted","Data":"af933f6f2d441d3b25d1e17873ec121c69640ec0d46b6ff6bbf2745c77cee2b9"} Jul 06 15:19:40.298997 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:40.298949 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9hm78" podStartSLOduration=34.382187231 podStartE2EDuration="41.298933366s" totalImagesPullingTime="6.916746135s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:18:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:19:40.297585959 +0000 UTC m=+126.911334495" watchObservedRunningTime="2026-07-06 15:19:40.298933366 +0000 UTC m=+126.912681904" Jul 06 15:19:41.842382 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:41.842335 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:43.007076 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:43.007018 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:43.446042 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:43.446001 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:43.477703 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:43.477665 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b-metrics-certs\") pod \"network-metrics-daemon-6x68d\" (UID: \"5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b\") " pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:19:43.480082 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:43.480058 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b-metrics-certs\") pod \"network-metrics-daemon-6x68d\" (UID: \"5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b\") " pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:19:43.705695 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:43.705613 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gh99x\"" Jul 06 15:19:43.713965 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:43.713940 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6x68d" Jul 06 15:19:43.841797 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:43.841724 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6x68d"] Jul 06 15:19:43.844211 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:19:43.844180 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fec1fd4_1e4c_4f7b_bb04_fd5948be7f2b.slice/crio-fbf4a7432a1225ecfaabac39fec6a51e94b1b58dca3f810cf696285d23499370 WatchSource:0}: Error finding container fbf4a7432a1225ecfaabac39fec6a51e94b1b58dca3f810cf696285d23499370: Status 404 returned error can't find the container with id fbf4a7432a1225ecfaabac39fec6a51e94b1b58dca3f810cf696285d23499370 Jul 06 15:19:44.293089 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:44.293054 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6x68d" event={"ID":"5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b","Type":"ContainerStarted","Data":"fbf4a7432a1225ecfaabac39fec6a51e94b1b58dca3f810cf696285d23499370"} Jul 06 15:19:45.298553 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:45.298435 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6x68d" event={"ID":"5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b","Type":"ContainerStarted","Data":"074fcaa804870858a5eb30897c1fe48d75b393721343dd73eb567f3d9de37681"} Jul 06 15:19:45.298553 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:45.298487 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6x68d" event={"ID":"5fec1fd4-1e4c-4f7b-bb04-fd5948be7f2b","Type":"ContainerStarted","Data":"69dcc4e287c19552c048cfe687608e2478edc9e03836e0d99d8aa85554e99f30"} Jul 06 15:19:45.318800 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:45.318748 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6x68d" podStartSLOduration=129.169462153 podStartE2EDuration="2m10.31873227s" totalImagesPullingTime="1.149270117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:17:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:19:45.317725893 +0000 UTC m=+131.931474429" watchObservedRunningTime="2026-07-06 15:19:45.31873227 +0000 UTC m=+131.932480808" Jul 06 15:19:45.418541 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:45.418494 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:45.525116 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:45.525080 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:47.376224 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.376184 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:47.482695 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.482660 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:47.580010 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.579975 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:47.618877 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.618844 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bwbvt"] Jul 06 15:19:47.621024 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.621008 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bwbvt" Jul 06 15:19:47.623753 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.623721 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-v7t7r\"" Jul 06 15:19:47.623753 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.623721 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 15:19:47.623982 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.623772 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 15:19:47.623982 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.623782 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 15:19:47.626664 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.625516 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 15:19:47.626664 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.625616 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 15:19:47.629147 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.626884 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 15:19:47.678955 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.678916 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:47.710059 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.710020 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8351bb23-9cd0-412e-8bef-d36f1eeff370-node-exporter-textfile\") pod \"node-exporter-bwbvt\" (UID: \"8351bb23-9cd0-412e-8bef-d36f1eeff370\") " pod="openshift-monitoring/node-exporter-bwbvt" Jul 06 15:19:47.710212 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.710096 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8351bb23-9cd0-412e-8bef-d36f1eeff370-node-exporter-tls\") pod \"node-exporter-bwbvt\" (UID: \"8351bb23-9cd0-412e-8bef-d36f1eeff370\") " pod="openshift-monitoring/node-exporter-bwbvt" Jul 06 15:19:47.710212 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.710114 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q2r6\" (UniqueName: \"kubernetes.io/projected/8351bb23-9cd0-412e-8bef-d36f1eeff370-kube-api-access-4q2r6\") pod \"node-exporter-bwbvt\" (UID: \"8351bb23-9cd0-412e-8bef-d36f1eeff370\") " pod="openshift-monitoring/node-exporter-bwbvt" Jul 06 15:19:47.710212 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.710135 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8351bb23-9cd0-412e-8bef-d36f1eeff370-root\") pod \"node-exporter-bwbvt\" (UID: \"8351bb23-9cd0-412e-8bef-d36f1eeff370\") " pod="openshift-monitoring/node-exporter-bwbvt" Jul 06 15:19:47.710212 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.710156 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8351bb23-9cd0-412e-8bef-d36f1eeff370-node-exporter-accelerators-collector-config\") pod \"node-exporter-bwbvt\" (UID: \"8351bb23-9cd0-412e-8bef-d36f1eeff370\") " pod="openshift-monitoring/node-exporter-bwbvt" Jul 06 15:19:47.710383 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.710229 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8351bb23-9cd0-412e-8bef-d36f1eeff370-metrics-client-ca\") pod \"node-exporter-bwbvt\" (UID: \"8351bb23-9cd0-412e-8bef-d36f1eeff370\") " pod="openshift-monitoring/node-exporter-bwbvt" Jul 06 15:19:47.710383 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.710277 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8351bb23-9cd0-412e-8bef-d36f1eeff370-node-exporter-wtmp\") pod \"node-exporter-bwbvt\" (UID: \"8351bb23-9cd0-412e-8bef-d36f1eeff370\") " pod="openshift-monitoring/node-exporter-bwbvt" Jul 06 15:19:47.710383 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.710313 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8351bb23-9cd0-412e-8bef-d36f1eeff370-sys\") pod \"node-exporter-bwbvt\" (UID: \"8351bb23-9cd0-412e-8bef-d36f1eeff370\") " pod="openshift-monitoring/node-exporter-bwbvt" Jul 06 15:19:47.710383 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.710331 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8351bb23-9cd0-412e-8bef-d36f1eeff370-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bwbvt\" (UID: \"8351bb23-9cd0-412e-8bef-d36f1eeff370\") " pod="openshift-monitoring/node-exporter-bwbvt" Jul 06 15:19:47.777693 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.777655 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:47.811043 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.811008 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8351bb23-9cd0-412e-8bef-d36f1eeff370-sys\") pod \"node-exporter-bwbvt\" (UID: \"8351bb23-9cd0-412e-8bef-d36f1eeff370\") " pod="openshift-monitoring/node-exporter-bwbvt" Jul 06 15:19:47.811218 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.811054 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8351bb23-9cd0-412e-8bef-d36f1eeff370-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bwbvt\" (UID: \"8351bb23-9cd0-412e-8bef-d36f1eeff370\") " pod="openshift-monitoring/node-exporter-bwbvt" Jul 06 15:19:47.811218 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.811089 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8351bb23-9cd0-412e-8bef-d36f1eeff370-node-exporter-textfile\") pod \"node-exporter-bwbvt\" (UID: \"8351bb23-9cd0-412e-8bef-d36f1eeff370\") " pod="openshift-monitoring/node-exporter-bwbvt" Jul 06 15:19:47.811218 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.811122 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8351bb23-9cd0-412e-8bef-d36f1eeff370-sys\") pod \"node-exporter-bwbvt\" (UID: \"8351bb23-9cd0-412e-8bef-d36f1eeff370\") " pod="openshift-monitoring/node-exporter-bwbvt" Jul 06 15:19:47.811218 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.811143 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8351bb23-9cd0-412e-8bef-d36f1eeff370-node-exporter-tls\") pod \"node-exporter-bwbvt\" (UID: \"8351bb23-9cd0-412e-8bef-d36f1eeff370\") " pod="openshift-monitoring/node-exporter-bwbvt" Jul 06 15:19:47.811218 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.811166 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q2r6\" (UniqueName: \"kubernetes.io/projected/8351bb23-9cd0-412e-8bef-d36f1eeff370-kube-api-access-4q2r6\") pod \"node-exporter-bwbvt\" (UID: \"8351bb23-9cd0-412e-8bef-d36f1eeff370\") " pod="openshift-monitoring/node-exporter-bwbvt" Jul 06 15:19:47.811218 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.811194 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8351bb23-9cd0-412e-8bef-d36f1eeff370-root\") pod \"node-exporter-bwbvt\" (UID: \"8351bb23-9cd0-412e-8bef-d36f1eeff370\") " pod="openshift-monitoring/node-exporter-bwbvt" Jul 06 15:19:47.811475 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.811251 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8351bb23-9cd0-412e-8bef-d36f1eeff370-root\") pod \"node-exporter-bwbvt\" (UID: \"8351bb23-9cd0-412e-8bef-d36f1eeff370\") " pod="openshift-monitoring/node-exporter-bwbvt" Jul 06 15:19:47.811475 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.811286 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8351bb23-9cd0-412e-8bef-d36f1eeff370-node-exporter-accelerators-collector-config\") pod \"node-exporter-bwbvt\" (UID: \"8351bb23-9cd0-412e-8bef-d36f1eeff370\") " pod="openshift-monitoring/node-exporter-bwbvt" Jul 06 15:19:47.811475 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.811319 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8351bb23-9cd0-412e-8bef-d36f1eeff370-metrics-client-ca\") pod \"node-exporter-bwbvt\" (UID: \"8351bb23-9cd0-412e-8bef-d36f1eeff370\") " pod="openshift-monitoring/node-exporter-bwbvt" Jul 06 15:19:47.811475 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.811346 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8351bb23-9cd0-412e-8bef-d36f1eeff370-node-exporter-wtmp\") pod \"node-exporter-bwbvt\" (UID: \"8351bb23-9cd0-412e-8bef-d36f1eeff370\") " pod="openshift-monitoring/node-exporter-bwbvt" Jul 06 15:19:47.811665 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.811488 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8351bb23-9cd0-412e-8bef-d36f1eeff370-node-exporter-textfile\") pod \"node-exporter-bwbvt\" (UID: \"8351bb23-9cd0-412e-8bef-d36f1eeff370\") " pod="openshift-monitoring/node-exporter-bwbvt" Jul 06 15:19:47.811665 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.811491 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8351bb23-9cd0-412e-8bef-d36f1eeff370-node-exporter-wtmp\") pod \"node-exporter-bwbvt\" (UID: \"8351bb23-9cd0-412e-8bef-d36f1eeff370\") " pod="openshift-monitoring/node-exporter-bwbvt" Jul 06 15:19:47.811956 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.811938 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8351bb23-9cd0-412e-8bef-d36f1eeff370-node-exporter-accelerators-collector-config\") pod \"node-exporter-bwbvt\" (UID: \"8351bb23-9cd0-412e-8bef-d36f1eeff370\") " pod="openshift-monitoring/node-exporter-bwbvt" Jul 06 15:19:47.812052 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.812032 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8351bb23-9cd0-412e-8bef-d36f1eeff370-metrics-client-ca\") pod \"node-exporter-bwbvt\" (UID: \"8351bb23-9cd0-412e-8bef-d36f1eeff370\") " pod="openshift-monitoring/node-exporter-bwbvt" Jul 06 15:19:47.813486 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.813467 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8351bb23-9cd0-412e-8bef-d36f1eeff370-node-exporter-tls\") pod \"node-exporter-bwbvt\" (UID: \"8351bb23-9cd0-412e-8bef-d36f1eeff370\") " pod="openshift-monitoring/node-exporter-bwbvt" Jul 06 15:19:47.813598 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.813579 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8351bb23-9cd0-412e-8bef-d36f1eeff370-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bwbvt\" (UID: \"8351bb23-9cd0-412e-8bef-d36f1eeff370\") " pod="openshift-monitoring/node-exporter-bwbvt" Jul 06 15:19:47.819973 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.819948 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q2r6\" (UniqueName: \"kubernetes.io/projected/8351bb23-9cd0-412e-8bef-d36f1eeff370-kube-api-access-4q2r6\") pod \"node-exporter-bwbvt\" (UID: \"8351bb23-9cd0-412e-8bef-d36f1eeff370\") " pod="openshift-monitoring/node-exporter-bwbvt" Jul 06 15:19:47.879427 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.879337 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:47.935372 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.935343 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bwbvt" Jul 06 15:19:47.945320 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:19:47.945296 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8351bb23_9cd0_412e_8bef_d36f1eeff370.slice/crio-62bffc61a8c67b37b3a72653b5df6076944637ca41a86ba74105299cad424da8 WatchSource:0}: Error finding container 62bffc61a8c67b37b3a72653b5df6076944637ca41a86ba74105299cad424da8: Status 404 returned error can't find the container with id 62bffc61a8c67b37b3a72653b5df6076944637ca41a86ba74105299cad424da8 Jul 06 15:19:47.979035 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:47.978997 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:48.086521 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:48.086467 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:48.188259 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:48.188175 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:48.308001 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:48.307967 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bwbvt" event={"ID":"8351bb23-9cd0-412e-8bef-d36f1eeff370","Type":"ContainerStarted","Data":"62bffc61a8c67b37b3a72653b5df6076944637ca41a86ba74105299cad424da8"} Jul 06 15:19:48.539564 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:48.539480 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:48.621747 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:48.621713 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:48.793958 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:48.793882 2570 scope.go:117] "RemoveContainer" containerID="277d1ced7b6e19f97050eea3d954e8cdde6d45dc00127883dfbc753f3959ea60" Jul 06 15:19:48.794110 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:48.794061 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-mlw9w_openshift-console-operator(91e0c855-5b32-4ca5-bab6-8e3aae20c1e0)\"" pod="openshift-console-operator/console-operator-68b757fb67-mlw9w" podUID="91e0c855-5b32-4ca5-bab6-8e3aae20c1e0" Jul 06 15:19:49.129648 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:49.129615 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:49.312270 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:49.312238 2570 generic.go:356] "Generic (PLEG): container finished" podID="8351bb23-9cd0-412e-8bef-d36f1eeff370" containerID="68d8647a2956ae474e7a99b92a7a58ee80488bf20f52e48308669f338f0a3935" exitCode=0 Jul 06 15:19:49.312430 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:49.312312 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bwbvt" event={"ID":"8351bb23-9cd0-412e-8bef-d36f1eeff370","Type":"ContainerDied","Data":"68d8647a2956ae474e7a99b92a7a58ee80488bf20f52e48308669f338f0a3935"} Jul 06 15:19:50.317324 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:50.317294 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bwbvt" event={"ID":"8351bb23-9cd0-412e-8bef-d36f1eeff370","Type":"ContainerStarted","Data":"c4e6d7c08c8d3f34ebbbb98a93b5bfc3e82d85cdc26322f2a53a6dfcd6c96814"} Jul 06 15:19:50.317715 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:50.317341 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bwbvt" event={"ID":"8351bb23-9cd0-412e-8bef-d36f1eeff370","Type":"ContainerStarted","Data":"402a0bbafc23790d023173765b50aa227bb2e2e005e20ccbecbe045fe40ceb69"} Jul 06 15:19:50.340216 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:50.340168 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bwbvt" podStartSLOduration=2.055558844 podStartE2EDuration="3.340152941s" totalImagesPullingTime="1.218400952s" totalInitContainerRuntime="66.193145ms" isStatefulPod=true podCreationTimestamp="2026-07-06 15:19:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:19:50.338533319 +0000 UTC m=+136.952281854" watchObservedRunningTime="2026-07-06 15:19:50.340152941 +0000 UTC m=+136.953901476" Jul 06 15:19:50.403284 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:50.403247 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:50.494378 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:50.494347 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:51.032387 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:51.032353 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:51.835591 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:51.835557 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-785fdbc95c-q4q4j"] Jul 06 15:19:51.837973 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:51.837954 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-785fdbc95c-q4q4j" Jul 06 15:19:51.840728 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:51.840696 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 06 15:19:51.840872 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:51.840857 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 06 15:19:51.842009 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:51.841990 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 06 15:19:51.842100 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:51.842054 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 06 15:19:51.842100 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:51.842078 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9ln4amltt5nqj\"" Jul 06 15:19:51.842363 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:51.842350 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-fpfwb\"" Jul 06 15:19:51.848234 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:51.848200 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-785fdbc95c-q4q4j"] Jul 06 15:19:51.944258 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:51.944224 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ffb012d-1795-42fb-b6c8-db9419cedd19-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-785fdbc95c-q4q4j\" (UID: \"8ffb012d-1795-42fb-b6c8-db9419cedd19\") " pod="openshift-monitoring/metrics-server-785fdbc95c-q4q4j" Jul 06 15:19:51.944258 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:51.944259 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p68sm\" (UniqueName: \"kubernetes.io/projected/8ffb012d-1795-42fb-b6c8-db9419cedd19-kube-api-access-p68sm\") pod \"metrics-server-785fdbc95c-q4q4j\" (UID: \"8ffb012d-1795-42fb-b6c8-db9419cedd19\") " pod="openshift-monitoring/metrics-server-785fdbc95c-q4q4j" Jul 06 15:19:51.944534 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:51.944288 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8ffb012d-1795-42fb-b6c8-db9419cedd19-secret-metrics-server-client-certs\") pod \"metrics-server-785fdbc95c-q4q4j\" (UID: \"8ffb012d-1795-42fb-b6c8-db9419cedd19\") " pod="openshift-monitoring/metrics-server-785fdbc95c-q4q4j" Jul 06 15:19:51.944534 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:51.944305 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8ffb012d-1795-42fb-b6c8-db9419cedd19-audit-log\") pod \"metrics-server-785fdbc95c-q4q4j\" (UID: \"8ffb012d-1795-42fb-b6c8-db9419cedd19\") " pod="openshift-monitoring/metrics-server-785fdbc95c-q4q4j" Jul 06 15:19:51.944534 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:51.944323 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8ffb012d-1795-42fb-b6c8-db9419cedd19-metrics-server-audit-profiles\") pod \"metrics-server-785fdbc95c-q4q4j\" (UID: \"8ffb012d-1795-42fb-b6c8-db9419cedd19\") " pod="openshift-monitoring/metrics-server-785fdbc95c-q4q4j" Jul 06 15:19:51.944534 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:51.944364 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8ffb012d-1795-42fb-b6c8-db9419cedd19-client-ca-bundle\") pod \"metrics-server-785fdbc95c-q4q4j\" (UID: \"8ffb012d-1795-42fb-b6c8-db9419cedd19\") " pod="openshift-monitoring/metrics-server-785fdbc95c-q4q4j" Jul 06 15:19:51.944534 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:51.944460 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8ffb012d-1795-42fb-b6c8-db9419cedd19-secret-metrics-server-tls\") pod \"metrics-server-785fdbc95c-q4q4j\" (UID: \"8ffb012d-1795-42fb-b6c8-db9419cedd19\") " pod="openshift-monitoring/metrics-server-785fdbc95c-q4q4j" Jul 06 15:19:52.045335 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:52.045297 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8ffb012d-1795-42fb-b6c8-db9419cedd19-secret-metrics-server-tls\") pod \"metrics-server-785fdbc95c-q4q4j\" (UID: \"8ffb012d-1795-42fb-b6c8-db9419cedd19\") " pod="openshift-monitoring/metrics-server-785fdbc95c-q4q4j" Jul 06 15:19:52.045557 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:52.045343 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ffb012d-1795-42fb-b6c8-db9419cedd19-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-785fdbc95c-q4q4j\" (UID: \"8ffb012d-1795-42fb-b6c8-db9419cedd19\") " pod="openshift-monitoring/metrics-server-785fdbc95c-q4q4j" Jul 06 15:19:52.045557 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:52.045363 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p68sm\" (UniqueName: \"kubernetes.io/projected/8ffb012d-1795-42fb-b6c8-db9419cedd19-kube-api-access-p68sm\") pod \"metrics-server-785fdbc95c-q4q4j\" (UID: \"8ffb012d-1795-42fb-b6c8-db9419cedd19\") " pod="openshift-monitoring/metrics-server-785fdbc95c-q4q4j" Jul 06 15:19:52.045557 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:52.045386 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8ffb012d-1795-42fb-b6c8-db9419cedd19-secret-metrics-server-client-certs\") pod \"metrics-server-785fdbc95c-q4q4j\" (UID: \"8ffb012d-1795-42fb-b6c8-db9419cedd19\") " pod="openshift-monitoring/metrics-server-785fdbc95c-q4q4j" Jul 06 15:19:52.045557 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:52.045414 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8ffb012d-1795-42fb-b6c8-db9419cedd19-audit-log\") pod \"metrics-server-785fdbc95c-q4q4j\" (UID: \"8ffb012d-1795-42fb-b6c8-db9419cedd19\") " pod="openshift-monitoring/metrics-server-785fdbc95c-q4q4j" Jul 06 15:19:52.045768 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:52.045590 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8ffb012d-1795-42fb-b6c8-db9419cedd19-metrics-server-audit-profiles\") pod \"metrics-server-785fdbc95c-q4q4j\" (UID: \"8ffb012d-1795-42fb-b6c8-db9419cedd19\") " pod="openshift-monitoring/metrics-server-785fdbc95c-q4q4j" Jul 06 15:19:52.045768 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:52.045653 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8ffb012d-1795-42fb-b6c8-db9419cedd19-client-ca-bundle\") pod \"metrics-server-785fdbc95c-q4q4j\" (UID: \"8ffb012d-1795-42fb-b6c8-db9419cedd19\") " pod="openshift-monitoring/metrics-server-785fdbc95c-q4q4j" Jul 06 15:19:52.045922 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:52.045865 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8ffb012d-1795-42fb-b6c8-db9419cedd19-audit-log\") pod \"metrics-server-785fdbc95c-q4q4j\" (UID: \"8ffb012d-1795-42fb-b6c8-db9419cedd19\") " pod="openshift-monitoring/metrics-server-785fdbc95c-q4q4j" Jul 06 15:19:52.046222 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:52.046193 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ffb012d-1795-42fb-b6c8-db9419cedd19-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-785fdbc95c-q4q4j\" (UID: \"8ffb012d-1795-42fb-b6c8-db9419cedd19\") " pod="openshift-monitoring/metrics-server-785fdbc95c-q4q4j" Jul 06 15:19:52.046532 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:52.046496 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8ffb012d-1795-42fb-b6c8-db9419cedd19-metrics-server-audit-profiles\") pod \"metrics-server-785fdbc95c-q4q4j\" (UID: \"8ffb012d-1795-42fb-b6c8-db9419cedd19\") " pod="openshift-monitoring/metrics-server-785fdbc95c-q4q4j" Jul 06 15:19:52.047958 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:52.047938 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8ffb012d-1795-42fb-b6c8-db9419cedd19-secret-metrics-server-client-certs\") pod \"metrics-server-785fdbc95c-q4q4j\" (UID: \"8ffb012d-1795-42fb-b6c8-db9419cedd19\") " pod="openshift-monitoring/metrics-server-785fdbc95c-q4q4j" Jul 06 15:19:52.048051 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:52.047978 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8ffb012d-1795-42fb-b6c8-db9419cedd19-client-ca-bundle\") pod \"metrics-server-785fdbc95c-q4q4j\" (UID: \"8ffb012d-1795-42fb-b6c8-db9419cedd19\") " pod="openshift-monitoring/metrics-server-785fdbc95c-q4q4j" Jul 06 15:19:52.048051 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:52.048017 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8ffb012d-1795-42fb-b6c8-db9419cedd19-secret-metrics-server-tls\") pod \"metrics-server-785fdbc95c-q4q4j\" (UID: \"8ffb012d-1795-42fb-b6c8-db9419cedd19\") " pod="openshift-monitoring/metrics-server-785fdbc95c-q4q4j" Jul 06 15:19:52.058796 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:52.058768 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p68sm\" (UniqueName: \"kubernetes.io/projected/8ffb012d-1795-42fb-b6c8-db9419cedd19-kube-api-access-p68sm\") pod \"metrics-server-785fdbc95c-q4q4j\" (UID: \"8ffb012d-1795-42fb-b6c8-db9419cedd19\") " pod="openshift-monitoring/metrics-server-785fdbc95c-q4q4j" Jul 06 15:19:52.147297 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:52.146969 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-785fdbc95c-q4q4j" Jul 06 15:19:52.287352 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:52.287306 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:52.288174 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:52.288154 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-4x8sx"] Jul 06 15:19:52.290804 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:52.290790 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4x8sx" Jul 06 15:19:52.293681 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:52.293662 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-6kchn\"" Jul 06 15:19:52.293796 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:52.293667 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 06 15:19:52.299797 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:52.299778 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-785fdbc95c-q4q4j"] Jul 06 15:19:52.302191 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:19:52.302170 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ffb012d_1795_42fb_b6c8_db9419cedd19.slice/crio-0c81437cf49b4055c5ff990b035e49d490f1f8453b8dd7870b66d7c91ea4278b WatchSource:0}: Error finding container 0c81437cf49b4055c5ff990b035e49d490f1f8453b8dd7870b66d7c91ea4278b: Status 404 returned error can't find the container with id 0c81437cf49b4055c5ff990b035e49d490f1f8453b8dd7870b66d7c91ea4278b Jul 06 15:19:52.306038 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:52.306017 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-4x8sx"] Jul 06 15:19:52.324768 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:52.324736 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-785fdbc95c-q4q4j" event={"ID":"8ffb012d-1795-42fb-b6c8-db9419cedd19","Type":"ContainerStarted","Data":"0c81437cf49b4055c5ff990b035e49d490f1f8453b8dd7870b66d7c91ea4278b"} Jul 06 15:19:52.349355 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:52.349317 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr4k7\" (UniqueName: \"kubernetes.io/projected/278d8a2b-9af9-43dd-aeaf-0db49545ee23-kube-api-access-rr4k7\") pod \"monitoring-plugin-d5dcff7cf-4x8sx\" (UID: \"278d8a2b-9af9-43dd-aeaf-0db49545ee23\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4x8sx" Jul 06 15:19:52.349535 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:52.349374 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/278d8a2b-9af9-43dd-aeaf-0db49545ee23-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-4x8sx\" (UID: \"278d8a2b-9af9-43dd-aeaf-0db49545ee23\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4x8sx" Jul 06 15:19:52.450715 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:52.450633 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr4k7\" (UniqueName: \"kubernetes.io/projected/278d8a2b-9af9-43dd-aeaf-0db49545ee23-kube-api-access-rr4k7\") pod \"monitoring-plugin-d5dcff7cf-4x8sx\" (UID: \"278d8a2b-9af9-43dd-aeaf-0db49545ee23\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4x8sx" Jul 06 15:19:52.450715 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:52.450686 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/278d8a2b-9af9-43dd-aeaf-0db49545ee23-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-4x8sx\" (UID: \"278d8a2b-9af9-43dd-aeaf-0db49545ee23\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4x8sx" Jul 06 15:19:52.450905 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:52.450796 2570 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 06 15:19:52.450905 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:19:52.450854 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/278d8a2b-9af9-43dd-aeaf-0db49545ee23-monitoring-plugin-cert podName:278d8a2b-9af9-43dd-aeaf-0db49545ee23 nodeName:}" failed. No retries permitted until 2026-07-06 15:19:52.950836973 +0000 UTC m=+139.564585488 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/278d8a2b-9af9-43dd-aeaf-0db49545ee23-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-4x8sx" (UID: "278d8a2b-9af9-43dd-aeaf-0db49545ee23") : secret "monitoring-plugin-cert" not found Jul 06 15:19:52.472212 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:52.472179 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr4k7\" (UniqueName: \"kubernetes.io/projected/278d8a2b-9af9-43dd-aeaf-0db49545ee23-kube-api-access-rr4k7\") pod \"monitoring-plugin-d5dcff7cf-4x8sx\" (UID: \"278d8a2b-9af9-43dd-aeaf-0db49545ee23\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4x8sx" Jul 06 15:19:52.520804 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:52.520760 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:52.954095 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:52.954062 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/278d8a2b-9af9-43dd-aeaf-0db49545ee23-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-4x8sx\" (UID: \"278d8a2b-9af9-43dd-aeaf-0db49545ee23\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4x8sx" Jul 06 15:19:52.956948 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:52.956917 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/278d8a2b-9af9-43dd-aeaf-0db49545ee23-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-4x8sx\" (UID: \"278d8a2b-9af9-43dd-aeaf-0db49545ee23\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4x8sx" Jul 06 15:19:53.201027 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.200990 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4x8sx" Jul 06 15:19:53.644003 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.643973 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-4x8sx"] Jul 06 15:19:53.647278 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:19:53.647251 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod278d8a2b_9af9_43dd_aeaf_0db49545ee23.slice/crio-cf1096e5cea984d58c5b818e5c5caacc7a2303cab383485a57f6a94f744d66ef WatchSource:0}: Error finding container cf1096e5cea984d58c5b818e5c5caacc7a2303cab383485a57f6a94f744d66ef: Status 404 returned error can't find the container with id cf1096e5cea984d58c5b818e5c5caacc7a2303cab383485a57f6a94f744d66ef Jul 06 15:19:53.810629 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.810539 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:53.823723 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.823700 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 15:19:53.826341 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.826322 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.839454 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.839433 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 15:19:53.839657 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.839639 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 15:19:53.839901 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.839889 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 15:19:53.840452 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.840439 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 15:19:53.842580 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.842565 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 15:19:53.842640 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.842630 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 15:19:53.842962 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.842946 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 15:19:53.842962 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.842957 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 15:19:53.843086 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.842958 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 15:19:53.843086 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.843010 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 15:19:53.844135 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.844118 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c8of7rodm035b\"" Jul 06 15:19:53.844234 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.844219 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 15:19:53.844290 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.844238 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-s2zzx\"" Jul 06 15:19:53.847482 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.847466 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 15:19:53.850242 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.850226 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 15:19:53.854612 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.854591 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 15:19:53.863312 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.863285 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.863404 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.863322 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.863404 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.863354 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.863404 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.863391 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.863545 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.863438 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.863545 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.863459 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.863545 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.863539 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-config-out\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.863696 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.863566 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrn4w\" (UniqueName: \"kubernetes.io/projected/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-kube-api-access-qrn4w\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.863696 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.863667 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.863793 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.863698 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-web-config\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.863793 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.863759 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-config\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.863891 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.863789 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.863891 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.863817 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.863891 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.863851 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.863891 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.863884 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.864021 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.863937 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.864021 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.863968 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.864021 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.864009 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.864124 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.864039 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.864124 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.864067 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.887856 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.887832 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 15:19:53.965404 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.965369 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.965404 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.965409 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.965935 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.965433 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.965935 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.965450 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.965935 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.965474 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.965935 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.965496 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.965935 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.965644 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.965935 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.965694 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.965935 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.965726 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.965935 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.965756 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.965935 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.965782 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.965935 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.965814 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-config-out\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.965935 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.965836 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrn4w\" (UniqueName: \"kubernetes.io/projected/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-kube-api-access-qrn4w\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.965935 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.965893 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.965935 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.965905 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.965935 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.965925 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-web-config\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.966637 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.965994 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-config\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.966637 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.966022 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.966637 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.966058 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.966637 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.966117 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.966637 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.966151 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.966637 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.966361 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.967033 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.967010 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.968942 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.968913 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.970175 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.969151 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.970175 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.969395 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-web-config\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.970175 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.969475 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.970175 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.969531 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.970175 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.969681 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.970175 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.969974 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.970543 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.970464 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.970727 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.970701 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.970987 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.970967 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.972593 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.972544 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.972593 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.972558 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.973020 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.972995 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.974344 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.974320 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-config-out\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.975740 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.975719 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-config\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.976356 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.976336 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:53.977708 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:53.977691 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrn4w\" (UniqueName: \"kubernetes.io/projected/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-kube-api-access-qrn4w\") pod \"prometheus-k8s-0\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:54.135275 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:54.135234 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:19:54.284089 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:54.284065 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 15:19:54.285057 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:19:54.285016 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccb599aa_5195_41dc_9d9e_d61c6e8f5192.slice/crio-871c81e351cf3ee575ed9d03e876b8d1195426caa8349e9aa1ad3bb0c1881b8c WatchSource:0}: Error finding container 871c81e351cf3ee575ed9d03e876b8d1195426caa8349e9aa1ad3bb0c1881b8c: Status 404 returned error can't find the container with id 871c81e351cf3ee575ed9d03e876b8d1195426caa8349e9aa1ad3bb0c1881b8c Jul 06 15:19:54.334274 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:54.334234 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-785fdbc95c-q4q4j" event={"ID":"8ffb012d-1795-42fb-b6c8-db9419cedd19","Type":"ContainerStarted","Data":"d8d271ca4670dde06af612924b259d187bee3e8bb3c6bb76e848701fd176e166"} Jul 06 15:19:54.335480 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:54.335456 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb599aa-5195-41dc-9d9e-d61c6e8f5192","Type":"ContainerStarted","Data":"871c81e351cf3ee575ed9d03e876b8d1195426caa8349e9aa1ad3bb0c1881b8c"} Jul 06 15:19:54.336544 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:54.336520 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4x8sx" event={"ID":"278d8a2b-9af9-43dd-aeaf-0db49545ee23","Type":"ContainerStarted","Data":"cf1096e5cea984d58c5b818e5c5caacc7a2303cab383485a57f6a94f744d66ef"} Jul 06 15:19:54.353875 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:54.353823 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-785fdbc95c-q4q4j" podStartSLOduration=2.115375251 podStartE2EDuration="3.353809028s" totalImagesPullingTime="1.238433777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:19:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:19:54.352938871 +0000 UTC m=+140.966687426" watchObservedRunningTime="2026-07-06 15:19:54.353809028 +0000 UTC m=+140.967557565" Jul 06 15:19:56.347020 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:56.346990 2570 generic.go:356] "Generic (PLEG): container finished" podID="ccb599aa-5195-41dc-9d9e-d61c6e8f5192" containerID="e6bcbbe1681c3e1489c8fc98d1887a510106fa0bca351626b20c863028ea1e95" exitCode=0 Jul 06 15:19:56.347454 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:56.347056 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb599aa-5195-41dc-9d9e-d61c6e8f5192","Type":"ContainerDied","Data":"e6bcbbe1681c3e1489c8fc98d1887a510106fa0bca351626b20c863028ea1e95"} Jul 06 15:19:56.349129 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:56.349039 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4x8sx" event={"ID":"278d8a2b-9af9-43dd-aeaf-0db49545ee23","Type":"ContainerStarted","Data":"ca41739e31c80c8583b2d93ece4ca6b11c7770fea33f3938c6c7a1c127735660"} Jul 06 15:19:56.349425 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:56.349366 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4x8sx" Jul 06 15:19:56.358804 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:56.358783 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4x8sx" Jul 06 15:19:56.398931 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:56.398865 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4x8sx" podStartSLOduration=2.672632818 podStartE2EDuration="4.398846522s" totalImagesPullingTime="1.726213704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:19:56.398464311 +0000 UTC m=+143.012212847" watchObservedRunningTime="2026-07-06 15:19:56.398846522 +0000 UTC m=+143.012595059" Jul 06 15:19:56.960460 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:56.960422 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:57.048911 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:57.048868 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:58.275550 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:58.275521 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-695c8f4bdb-jm29j" Jul 06 15:19:58.434032 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:58.433993 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:58.535879 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:58.535788 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:19:58.597107 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:19:58.597072 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:00.247787 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:00.247759 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:00.356746 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:00.356710 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:00.370679 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:00.370644 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb599aa-5195-41dc-9d9e-d61c6e8f5192","Type":"ContainerStarted","Data":"37ad5eec076a3d3f073ebb53c62ec7e80365120cfb48607ad55f96219057f0dc"} Jul 06 15:20:00.370839 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:00.370735 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb599aa-5195-41dc-9d9e-d61c6e8f5192","Type":"ContainerStarted","Data":"12b0afb1be06da616d71bc212e4d34dd0c3ea4ea7c2b69d4133b2a4d4ab74c3e"} Jul 06 15:20:01.793931 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:01.793899 2570 scope.go:117] "RemoveContainer" containerID="277d1ced7b6e19f97050eea3d954e8cdde6d45dc00127883dfbc753f3959ea60" Jul 06 15:20:02.381726 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:02.381702 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-mlw9w_91e0c855-5b32-4ca5-bab6-8e3aae20c1e0/console-operator/2.log" Jul 06 15:20:02.381916 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:02.381824 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-mlw9w" event={"ID":"91e0c855-5b32-4ca5-bab6-8e3aae20c1e0","Type":"ContainerStarted","Data":"a48523e625a5807fa0e7273bb5164a7bfbe614aaeb4b750b4316e68217613ae5"} Jul 06 15:20:02.382694 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:02.382627 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-mlw9w" Jul 06 15:20:02.386366 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:02.386343 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb599aa-5195-41dc-9d9e-d61c6e8f5192","Type":"ContainerStarted","Data":"5a6f8b693c228b08cf94db09b1ae412554308a6e5c0b264b1d2faf2944af4513"} Jul 06 15:20:02.386517 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:02.386399 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb599aa-5195-41dc-9d9e-d61c6e8f5192","Type":"ContainerStarted","Data":"0a78e183f96534e00da776ccec4ae67d0cb3b2a8e64ad59712ef973d6ae6ced8"} Jul 06 15:20:02.386517 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:02.386427 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb599aa-5195-41dc-9d9e-d61c6e8f5192","Type":"ContainerStarted","Data":"e9c2161500a390fab09658e0514ee2a5a48782fac89ef70b1a94a9a83310a159"} Jul 06 15:20:02.386517 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:02.386459 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb599aa-5195-41dc-9d9e-d61c6e8f5192","Type":"ContainerStarted","Data":"3faeb35a7f4300c805c8c020edce95a4d7ada0aa49f9f2ac0e0ba89fa0f91716"} Jul 06 15:20:02.401792 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:02.401742 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-mlw9w" podStartSLOduration=53.06691379 podStartE2EDuration="56.40172805s" totalImagesPullingTime="3.33481426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:19:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:20:02.401287251 +0000 UTC m=+149.015035788" watchObservedRunningTime="2026-07-06 15:20:02.40172805 +0000 UTC m=+149.015476587" Jul 06 15:20:02.431057 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:02.430995 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.695802432 podStartE2EDuration="9.430976345s" totalImagesPullingTime="6.716225436s" totalInitContainerRuntime="18.948477ms" isStatefulPod=false podCreationTimestamp="2026-07-06 15:19:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:20:02.428830093 +0000 UTC m=+149.042578624" watchObservedRunningTime="2026-07-06 15:20:02.430976345 +0000 UTC m=+149.044724915" Jul 06 15:20:02.793253 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:02.793170 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-mlw9w" Jul 06 15:20:03.070011 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:03.069933 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:03.222773 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:03.222738 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:03.318392 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:03.318358 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:03.470747 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:03.470714 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:03.687734 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:03.687695 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:03.757272 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:03.757184 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:04.086262 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:04.086225 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:04.136164 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:04.136137 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:20:04.494005 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:04.493910 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:04.569552 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:04.569519 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:04.711824 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:04.711792 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:04.789223 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:04.789139 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:04.894052 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:04.894012 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:05.053350 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:05.053272 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:05.238437 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:05.238401 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:09.925244 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:09.925208 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:12.147771 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:12.147735 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-785fdbc95c-q4q4j" Jul 06 15:20:12.148133 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:12.147790 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-785fdbc95c-q4q4j" Jul 06 15:20:13.320848 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:13.320811 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:13.422316 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:13.422272 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:13.523673 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:13.523636 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:19.792393 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:19.792352 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:19.876639 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:19.876588 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:20.125103 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:20.125072 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:20.219319 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:20.219286 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:21.459854 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:21.459822 2570 generic.go:356] "Generic (PLEG): container finished" podID="4b9bcef7-bd88-424e-b6cb-1125adbb047d" containerID="d17c5676876ede92c907aadb3bf5dc1ab5656f02d1a87c2aeba15c7c85ffd563" exitCode=0 Jul 06 15:20:21.460341 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:21.459900 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jv8z5" event={"ID":"4b9bcef7-bd88-424e-b6cb-1125adbb047d","Type":"ContainerDied","Data":"d17c5676876ede92c907aadb3bf5dc1ab5656f02d1a87c2aeba15c7c85ffd563"} Jul 06 15:20:21.460341 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:21.460275 2570 scope.go:117] "RemoveContainer" containerID="d17c5676876ede92c907aadb3bf5dc1ab5656f02d1a87c2aeba15c7c85ffd563" Jul 06 15:20:21.938612 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:21.938577 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:22.009226 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:22.009197 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:22.465172 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:22.465141 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jv8z5" event={"ID":"4b9bcef7-bd88-424e-b6cb-1125adbb047d","Type":"ContainerStarted","Data":"eef992de8751ce24ff6d1f3b0f419c808d6c22861a5dc1faecdd9b371c94f4ef"} Jul 06 15:20:27.484382 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:27.484333 2570 generic.go:356] "Generic (PLEG): container finished" podID="3f6843dc-43c3-44f3-9d07-6ac8a695b490" containerID="af33b71f1522f1be4a9ebc979be0ed430539f530cd63d048ec3f683575e79a6a" exitCode=0 Jul 06 15:20:27.484824 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:27.484410 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7s7kr" event={"ID":"3f6843dc-43c3-44f3-9d07-6ac8a695b490","Type":"ContainerDied","Data":"af33b71f1522f1be4a9ebc979be0ed430539f530cd63d048ec3f683575e79a6a"} Jul 06 15:20:27.484824 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:27.484777 2570 scope.go:117] "RemoveContainer" containerID="af33b71f1522f1be4a9ebc979be0ed430539f530cd63d048ec3f683575e79a6a" Jul 06 15:20:28.489931 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:28.489898 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-7s7kr" event={"ID":"3f6843dc-43c3-44f3-9d07-6ac8a695b490","Type":"ContainerStarted","Data":"334a60b02f2d0b7b5ccc28378fb40a0e778fa721825e1f2117188abcaa597cd0"} Jul 06 15:20:29.385947 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:29.385904 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:30.173681 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:30.173653 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:30.299391 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:30.299350 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:30.395323 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:30.395294 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:32.153521 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:32.153471 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-785fdbc95c-q4q4j" Jul 06 15:20:32.157311 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:32.157285 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-785fdbc95c-q4q4j" Jul 06 15:20:33.340480 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:33.340446 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:33.462993 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:33.462958 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:33.850333 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:33.850292 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:33.950974 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:33.950944 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:34.514492 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:34.514462 2570 generic.go:356] "Generic (PLEG): container finished" podID="9f53ff9d-d4c6-4b93-8eed-39e727850e02" containerID="e48986d805b8d0bd8099ac022e982fb2dba001c5eed431371507cbff2c7d7fb1" exitCode=0 Jul 06 15:20:34.514897 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:34.514528 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-9ktzq" event={"ID":"9f53ff9d-d4c6-4b93-8eed-39e727850e02","Type":"ContainerDied","Data":"e48986d805b8d0bd8099ac022e982fb2dba001c5eed431371507cbff2c7d7fb1"} Jul 06 15:20:34.514897 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:34.514885 2570 scope.go:117] "RemoveContainer" containerID="e48986d805b8d0bd8099ac022e982fb2dba001c5eed431371507cbff2c7d7fb1" Jul 06 15:20:35.519968 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:35.519934 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-9ktzq" event={"ID":"9f53ff9d-d4c6-4b93-8eed-39e727850e02","Type":"ContainerStarted","Data":"d84f29df031414a823deae839550c9f2e6e2c269473fadc81d31336778afb514"} Jul 06 15:20:49.762997 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:49.762963 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:49.840287 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:49.840248 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:51.935689 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:51.935654 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:20:54.136359 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:54.136319 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:20:54.152018 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:54.151988 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:20:54.604181 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:20:54.604155 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:03.809207 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:03.809176 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:21:03.877615 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:03.877581 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:21:05.491282 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:05.491224 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:21:05.653717 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:05.653677 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:21:12.519978 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.519948 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 15:21:12.520523 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.520241 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ccb599aa-5195-41dc-9d9e-d61c6e8f5192" containerName="prometheus" containerID="cri-o://12b0afb1be06da616d71bc212e4d34dd0c3ea4ea7c2b69d4133b2a4d4ab74c3e" gracePeriod=600 Jul 06 15:21:12.520523 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.520263 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ccb599aa-5195-41dc-9d9e-d61c6e8f5192" containerName="kube-rbac-proxy" containerID="cri-o://0a78e183f96534e00da776ccec4ae67d0cb3b2a8e64ad59712ef973d6ae6ced8" gracePeriod=600 Jul 06 15:21:12.520523 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.520285 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ccb599aa-5195-41dc-9d9e-d61c6e8f5192" containerName="thanos-sidecar" containerID="cri-o://3faeb35a7f4300c805c8c020edce95a4d7ada0aa49f9f2ac0e0ba89fa0f91716" gracePeriod=600 Jul 06 15:21:12.520523 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.520321 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ccb599aa-5195-41dc-9d9e-d61c6e8f5192" containerName="kube-rbac-proxy-thanos" containerID="cri-o://5a6f8b693c228b08cf94db09b1ae412554308a6e5c0b264b1d2faf2944af4513" gracePeriod=600 Jul 06 15:21:12.520523 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.520322 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ccb599aa-5195-41dc-9d9e-d61c6e8f5192" containerName="config-reloader" containerID="cri-o://37ad5eec076a3d3f073ebb53c62ec7e80365120cfb48607ad55f96219057f0dc" gracePeriod=600 Jul 06 15:21:12.520523 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.520293 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ccb599aa-5195-41dc-9d9e-d61c6e8f5192" containerName="kube-rbac-proxy-web" containerID="cri-o://e9c2161500a390fab09658e0514ee2a5a48782fac89ef70b1a94a9a83310a159" gracePeriod=600 Jul 06 15:21:12.657821 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.657787 2570 generic.go:356] "Generic (PLEG): container finished" podID="ccb599aa-5195-41dc-9d9e-d61c6e8f5192" containerID="5a6f8b693c228b08cf94db09b1ae412554308a6e5c0b264b1d2faf2944af4513" exitCode=0 Jul 06 15:21:12.657821 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.657820 2570 generic.go:356] "Generic (PLEG): container finished" podID="ccb599aa-5195-41dc-9d9e-d61c6e8f5192" containerID="0a78e183f96534e00da776ccec4ae67d0cb3b2a8e64ad59712ef973d6ae6ced8" exitCode=0 Jul 06 15:21:12.657821 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.657829 2570 generic.go:356] "Generic (PLEG): container finished" podID="ccb599aa-5195-41dc-9d9e-d61c6e8f5192" containerID="e9c2161500a390fab09658e0514ee2a5a48782fac89ef70b1a94a9a83310a159" exitCode=0 Jul 06 15:21:12.658039 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.657837 2570 generic.go:356] "Generic (PLEG): container finished" podID="ccb599aa-5195-41dc-9d9e-d61c6e8f5192" containerID="3faeb35a7f4300c805c8c020edce95a4d7ada0aa49f9f2ac0e0ba89fa0f91716" exitCode=0 Jul 06 15:21:12.658039 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.657845 2570 generic.go:356] "Generic (PLEG): container finished" podID="ccb599aa-5195-41dc-9d9e-d61c6e8f5192" containerID="37ad5eec076a3d3f073ebb53c62ec7e80365120cfb48607ad55f96219057f0dc" exitCode=0 Jul 06 15:21:12.658039 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.657853 2570 generic.go:356] "Generic (PLEG): container finished" podID="ccb599aa-5195-41dc-9d9e-d61c6e8f5192" containerID="12b0afb1be06da616d71bc212e4d34dd0c3ea4ea7c2b69d4133b2a4d4ab74c3e" exitCode=0 Jul 06 15:21:12.658039 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.657855 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb599aa-5195-41dc-9d9e-d61c6e8f5192","Type":"ContainerDied","Data":"5a6f8b693c228b08cf94db09b1ae412554308a6e5c0b264b1d2faf2944af4513"} Jul 06 15:21:12.658039 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.657918 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb599aa-5195-41dc-9d9e-d61c6e8f5192","Type":"ContainerDied","Data":"0a78e183f96534e00da776ccec4ae67d0cb3b2a8e64ad59712ef973d6ae6ced8"} Jul 06 15:21:12.658039 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.657944 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb599aa-5195-41dc-9d9e-d61c6e8f5192","Type":"ContainerDied","Data":"e9c2161500a390fab09658e0514ee2a5a48782fac89ef70b1a94a9a83310a159"} Jul 06 15:21:12.658039 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.657967 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb599aa-5195-41dc-9d9e-d61c6e8f5192","Type":"ContainerDied","Data":"3faeb35a7f4300c805c8c020edce95a4d7ada0aa49f9f2ac0e0ba89fa0f91716"} Jul 06 15:21:12.658039 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.657993 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb599aa-5195-41dc-9d9e-d61c6e8f5192","Type":"ContainerDied","Data":"37ad5eec076a3d3f073ebb53c62ec7e80365120cfb48607ad55f96219057f0dc"} Jul 06 15:21:12.658039 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.658017 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb599aa-5195-41dc-9d9e-d61c6e8f5192","Type":"ContainerDied","Data":"12b0afb1be06da616d71bc212e4d34dd0c3ea4ea7c2b69d4133b2a4d4ab74c3e"} Jul 06 15:21:12.756827 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.756798 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:12.857528 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.857480 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-kube-rbac-proxy\") pod \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " Jul 06 15:21:12.857693 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.857577 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-prometheus-k8s-rulefiles-1\") pod \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " Jul 06 15:21:12.857693 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.857617 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " Jul 06 15:21:12.857693 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.857648 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrn4w\" (UniqueName: \"kubernetes.io/projected/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-kube-api-access-qrn4w\") pod \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " Jul 06 15:21:12.857693 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.857676 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-thanos-prometheus-http-client-file\") pod \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " Jul 06 15:21:12.859055 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.857707 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-prometheus-trusted-ca-bundle\") pod \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " Jul 06 15:21:12.859055 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.857738 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-configmap-kubelet-serving-ca-bundle\") pod \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " Jul 06 15:21:12.859055 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.857766 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-prometheus-k8s-rulefiles-0\") pod \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " Jul 06 15:21:12.859055 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.857798 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-configmap-metrics-client-ca\") pod \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " Jul 06 15:21:12.859055 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.857836 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-grpc-tls\") pod \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " Jul 06 15:21:12.859055 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.857869 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-prometheus-k8s-db\") pod \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " Jul 06 15:21:12.859055 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.857895 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-configmap-serving-certs-ca-bundle\") pod \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " Jul 06 15:21:12.859055 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.857923 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-prometheus-k8s-rulefiles-2\") pod \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " Jul 06 15:21:12.859055 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.857920 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "ccb599aa-5195-41dc-9d9e-d61c6e8f5192" (UID: "ccb599aa-5195-41dc-9d9e-d61c6e8f5192"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:21:12.859055 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.857956 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-config\") pod \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " Jul 06 15:21:12.859055 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.857983 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " Jul 06 15:21:12.859055 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.858044 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-tls-assets\") pod \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " Jul 06 15:21:12.859055 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.858146 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-prometheus-k8s-tls\") pod \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " Jul 06 15:21:12.859055 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.858234 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-metrics-client-certs\") pod \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " Jul 06 15:21:12.859055 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.858290 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-config-out\") pod \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " Jul 06 15:21:12.859055 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.858335 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-web-config\") pod \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\" (UID: \"ccb599aa-5195-41dc-9d9e-d61c6e8f5192\") " Jul 06 15:21:12.859055 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.858618 2570 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:21:12.860136 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.859197 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "ccb599aa-5195-41dc-9d9e-d61c6e8f5192" (UID: "ccb599aa-5195-41dc-9d9e-d61c6e8f5192"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:21:12.860136 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.859636 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "ccb599aa-5195-41dc-9d9e-d61c6e8f5192" (UID: "ccb599aa-5195-41dc-9d9e-d61c6e8f5192"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:21:12.860136 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.859725 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "ccb599aa-5195-41dc-9d9e-d61c6e8f5192" (UID: "ccb599aa-5195-41dc-9d9e-d61c6e8f5192"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:21:12.860136 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.859914 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "ccb599aa-5195-41dc-9d9e-d61c6e8f5192" (UID: "ccb599aa-5195-41dc-9d9e-d61c6e8f5192"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:21:12.860136 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.860115 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "ccb599aa-5195-41dc-9d9e-d61c6e8f5192" (UID: "ccb599aa-5195-41dc-9d9e-d61c6e8f5192"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:21:12.860643 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.860616 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "ccb599aa-5195-41dc-9d9e-d61c6e8f5192" (UID: "ccb599aa-5195-41dc-9d9e-d61c6e8f5192"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:21:12.860997 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.860969 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-kube-api-access-qrn4w" (OuterVolumeSpecName: "kube-api-access-qrn4w") pod "ccb599aa-5195-41dc-9d9e-d61c6e8f5192" (UID: "ccb599aa-5195-41dc-9d9e-d61c6e8f5192"). InnerVolumeSpecName "kube-api-access-qrn4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:21:12.861092 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.860992 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "ccb599aa-5195-41dc-9d9e-d61c6e8f5192" (UID: "ccb599aa-5195-41dc-9d9e-d61c6e8f5192"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:21:12.862860 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.862834 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "ccb599aa-5195-41dc-9d9e-d61c6e8f5192" (UID: "ccb599aa-5195-41dc-9d9e-d61c6e8f5192"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:21:12.862973 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.862876 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "ccb599aa-5195-41dc-9d9e-d61c6e8f5192" (UID: "ccb599aa-5195-41dc-9d9e-d61c6e8f5192"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:21:12.863135 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.863107 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ccb599aa-5195-41dc-9d9e-d61c6e8f5192" (UID: "ccb599aa-5195-41dc-9d9e-d61c6e8f5192"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:21:12.863233 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.863134 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "ccb599aa-5195-41dc-9d9e-d61c6e8f5192" (UID: "ccb599aa-5195-41dc-9d9e-d61c6e8f5192"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:21:12.863233 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.863160 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "ccb599aa-5195-41dc-9d9e-d61c6e8f5192" (UID: "ccb599aa-5195-41dc-9d9e-d61c6e8f5192"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:21:12.863375 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.863350 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-config" (OuterVolumeSpecName: "config") pod "ccb599aa-5195-41dc-9d9e-d61c6e8f5192" (UID: "ccb599aa-5195-41dc-9d9e-d61c6e8f5192"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:21:12.863953 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.863928 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-config-out" (OuterVolumeSpecName: "config-out") pod "ccb599aa-5195-41dc-9d9e-d61c6e8f5192" (UID: "ccb599aa-5195-41dc-9d9e-d61c6e8f5192"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:21:12.864039 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.863961 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "ccb599aa-5195-41dc-9d9e-d61c6e8f5192" (UID: "ccb599aa-5195-41dc-9d9e-d61c6e8f5192"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:21:12.864333 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.864316 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "ccb599aa-5195-41dc-9d9e-d61c6e8f5192" (UID: "ccb599aa-5195-41dc-9d9e-d61c6e8f5192"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:21:12.864906 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.864883 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "ccb599aa-5195-41dc-9d9e-d61c6e8f5192" (UID: "ccb599aa-5195-41dc-9d9e-d61c6e8f5192"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:21:12.872533 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.872493 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-web-config" (OuterVolumeSpecName: "web-config") pod "ccb599aa-5195-41dc-9d9e-d61c6e8f5192" (UID: "ccb599aa-5195-41dc-9d9e-d61c6e8f5192"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:21:12.959092 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.959050 2570 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-web-config\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:21:12.959092 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.959088 2570 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-kube-rbac-proxy\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:21:12.959092 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.959100 2570 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:21:12.959092 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.959112 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrn4w\" (UniqueName: \"kubernetes.io/projected/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-kube-api-access-qrn4w\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:21:12.959350 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.959121 2570 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:21:12.959350 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.959130 2570 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:21:12.959350 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.959139 2570 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:21:12.959350 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.959147 2570 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:21:12.959350 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.959157 2570 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-configmap-metrics-client-ca\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:21:12.959350 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.959165 2570 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-grpc-tls\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:21:12.959350 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.959173 2570 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-prometheus-k8s-db\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:21:12.959350 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.959181 2570 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:21:12.959350 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.959190 2570 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:21:12.959350 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.959198 2570 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-config\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:21:12.959350 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.959207 2570 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:21:12.959350 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.959215 2570 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-tls-assets\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:21:12.959350 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.959223 2570 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:21:12.959350 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.959231 2570 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-secret-metrics-client-certs\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:21:12.959350 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:12.959239 2570 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ccb599aa-5195-41dc-9d9e-d61c6e8f5192-config-out\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:21:13.664839 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.664806 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccb599aa-5195-41dc-9d9e-d61c6e8f5192","Type":"ContainerDied","Data":"871c81e351cf3ee575ed9d03e876b8d1195426caa8349e9aa1ad3bb0c1881b8c"} Jul 06 15:21:13.665230 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.664846 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.665230 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.664898 2570 scope.go:117] "RemoveContainer" containerID="5a6f8b693c228b08cf94db09b1ae412554308a6e5c0b264b1d2faf2944af4513" Jul 06 15:21:13.673464 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.673278 2570 scope.go:117] "RemoveContainer" containerID="0a78e183f96534e00da776ccec4ae67d0cb3b2a8e64ad59712ef973d6ae6ced8" Jul 06 15:21:13.680140 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.680120 2570 scope.go:117] "RemoveContainer" containerID="e9c2161500a390fab09658e0514ee2a5a48782fac89ef70b1a94a9a83310a159" Jul 06 15:21:13.686871 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.686847 2570 scope.go:117] "RemoveContainer" containerID="3faeb35a7f4300c805c8c020edce95a4d7ada0aa49f9f2ac0e0ba89fa0f91716" Jul 06 15:21:13.693354 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.693327 2570 scope.go:117] "RemoveContainer" containerID="37ad5eec076a3d3f073ebb53c62ec7e80365120cfb48607ad55f96219057f0dc" Jul 06 15:21:13.698401 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.698375 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 15:21:13.702788 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.702745 2570 scope.go:117] "RemoveContainer" containerID="12b0afb1be06da616d71bc212e4d34dd0c3ea4ea7c2b69d4133b2a4d4ab74c3e" Jul 06 15:21:13.704657 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.704635 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 15:21:13.710906 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.710886 2570 scope.go:117] "RemoveContainer" containerID="e6bcbbe1681c3e1489c8fc98d1887a510106fa0bca351626b20c863028ea1e95" Jul 06 15:21:13.746329 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.746303 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 15:21:13.746584 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.746572 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb599aa-5195-41dc-9d9e-d61c6e8f5192" containerName="init-config-reloader" Jul 06 15:21:13.746624 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.746586 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb599aa-5195-41dc-9d9e-d61c6e8f5192" containerName="init-config-reloader" Jul 06 15:21:13.746624 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.746618 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb599aa-5195-41dc-9d9e-d61c6e8f5192" containerName="prometheus" Jul 06 15:21:13.746624 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.746624 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb599aa-5195-41dc-9d9e-d61c6e8f5192" containerName="prometheus" Jul 06 15:21:13.746711 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.746632 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb599aa-5195-41dc-9d9e-d61c6e8f5192" containerName="thanos-sidecar" Jul 06 15:21:13.746711 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.746637 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb599aa-5195-41dc-9d9e-d61c6e8f5192" containerName="thanos-sidecar" Jul 06 15:21:13.746711 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.746654 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb599aa-5195-41dc-9d9e-d61c6e8f5192" containerName="config-reloader" Jul 06 15:21:13.746711 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.746659 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb599aa-5195-41dc-9d9e-d61c6e8f5192" containerName="config-reloader" Jul 06 15:21:13.746711 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.746665 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb599aa-5195-41dc-9d9e-d61c6e8f5192" containerName="kube-rbac-proxy-thanos" Jul 06 15:21:13.746711 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.746670 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb599aa-5195-41dc-9d9e-d61c6e8f5192" containerName="kube-rbac-proxy-thanos" Jul 06 15:21:13.746711 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.746675 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb599aa-5195-41dc-9d9e-d61c6e8f5192" containerName="kube-rbac-proxy-web" Jul 06 15:21:13.746711 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.746680 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb599aa-5195-41dc-9d9e-d61c6e8f5192" containerName="kube-rbac-proxy-web" Jul 06 15:21:13.746711 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.746685 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb599aa-5195-41dc-9d9e-d61c6e8f5192" containerName="kube-rbac-proxy" Jul 06 15:21:13.746711 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.746690 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb599aa-5195-41dc-9d9e-d61c6e8f5192" containerName="kube-rbac-proxy" Jul 06 15:21:13.747009 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.746737 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccb599aa-5195-41dc-9d9e-d61c6e8f5192" containerName="prometheus" Jul 06 15:21:13.747009 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.746744 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccb599aa-5195-41dc-9d9e-d61c6e8f5192" containerName="kube-rbac-proxy" Jul 06 15:21:13.747009 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.746749 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccb599aa-5195-41dc-9d9e-d61c6e8f5192" containerName="thanos-sidecar" Jul 06 15:21:13.747009 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.746756 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccb599aa-5195-41dc-9d9e-d61c6e8f5192" containerName="kube-rbac-proxy-thanos" Jul 06 15:21:13.747009 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.746763 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccb599aa-5195-41dc-9d9e-d61c6e8f5192" containerName="kube-rbac-proxy-web" Jul 06 15:21:13.747009 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.746768 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccb599aa-5195-41dc-9d9e-d61c6e8f5192" containerName="config-reloader" Jul 06 15:21:13.750318 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.750301 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.753761 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.753741 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 15:21:13.753901 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.753882 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 15:21:13.754008 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.753989 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-s2zzx\"" Jul 06 15:21:13.754349 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.754332 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 15:21:13.754463 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.754337 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 15:21:13.754545 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.754533 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 15:21:13.754616 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.754601 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 15:21:13.759011 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.758994 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 15:21:13.759098 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.759036 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 15:21:13.759154 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.759095 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 15:21:13.759292 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.759278 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 15:21:13.759573 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.759560 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 15:21:13.759645 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.759615 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 15:21:13.759968 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.759950 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c8of7rodm035b\"" Jul 06 15:21:13.761698 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.761681 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 15:21:13.764145 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.764127 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-web-config\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.764234 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.764158 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-config\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.764234 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.764174 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.764234 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.764194 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.764234 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.764229 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.764389 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.764259 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.764389 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.764280 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.764389 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.764303 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.764389 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.764329 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.764389 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.764345 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqltg\" (UniqueName: \"kubernetes.io/projected/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-kube-api-access-pqltg\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.764389 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.764364 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.764389 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.764384 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.764636 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.764400 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.764636 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.764442 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.764636 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.764457 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.764636 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.764470 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.764636 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.764521 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.764636 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.764549 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-config-out\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.764636 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.764578 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.764636 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.764593 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.764910 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.764750 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 15:21:13.776229 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.775809 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 15:21:13.833338 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.833309 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:21:13.865434 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.865395 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.865434 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.865436 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.865685 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.865478 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-web-config\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.865685 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.865540 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-config\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.865685 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.865562 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.865685 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.865590 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.865685 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.865611 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.865685 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.865646 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.865685 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.865678 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.866007 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.865705 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.866007 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.865733 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.866007 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.865759 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqltg\" (UniqueName: \"kubernetes.io/projected/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-kube-api-access-pqltg\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.866460 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.866429 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.867145 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.866610 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.867145 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.866687 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.867145 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.866728 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.867145 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.866757 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.867145 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.866790 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.867145 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.866817 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.867145 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.866820 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.867145 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.866878 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.867145 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.866922 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.867145 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.866954 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-config-out\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.867145 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.867055 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.867760 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.867727 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.868195 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.868166 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.868277 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.868226 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.868877 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.868853 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.869332 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.869305 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-config\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.869578 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.869557 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.870128 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.870086 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.870616 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.870596 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.870966 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.870945 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.870966 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.870960 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-web-config\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.871400 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.871381 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.871482 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.871438 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.871632 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.871616 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-config-out\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.871848 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.871829 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.872000 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.871984 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.879675 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.879649 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqltg\" (UniqueName: \"kubernetes.io/projected/48c083a8-ef73-46dc-8a62-d4748a1d4b4c-kube-api-access-pqltg\") pod \"prometheus-k8s-0\" (UID: \"48c083a8-ef73-46dc-8a62-d4748a1d4b4c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:13.932201 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:13.932164 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:21:14.059509 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:14.059476 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:14.221017 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:14.220991 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 15:21:14.223989 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:21:14.223957 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c083a8_ef73_46dc_8a62_d4748a1d4b4c.slice/crio-b966c307443a1464069a8c6c9ccc0c610342db5ac7f01de9c3e3ba830da0a3a5 WatchSource:0}: Error finding container b966c307443a1464069a8c6c9ccc0c610342db5ac7f01de9c3e3ba830da0a3a5: Status 404 returned error can't find the container with id b966c307443a1464069a8c6c9ccc0c610342db5ac7f01de9c3e3ba830da0a3a5 Jul 06 15:21:14.669637 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:14.669602 2570 generic.go:356] "Generic (PLEG): container finished" podID="48c083a8-ef73-46dc-8a62-d4748a1d4b4c" containerID="4a83f8ff45364ba748343330053337158cb2170d67707bc563f2b146ecdb0cca" exitCode=0 Jul 06 15:21:14.670039 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:14.669693 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"48c083a8-ef73-46dc-8a62-d4748a1d4b4c","Type":"ContainerDied","Data":"4a83f8ff45364ba748343330053337158cb2170d67707bc563f2b146ecdb0cca"} Jul 06 15:21:14.670039 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:14.669771 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"48c083a8-ef73-46dc-8a62-d4748a1d4b4c","Type":"ContainerStarted","Data":"b966c307443a1464069a8c6c9ccc0c610342db5ac7f01de9c3e3ba830da0a3a5"} Jul 06 15:21:14.801373 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:14.801345 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccb599aa-5195-41dc-9d9e-d61c6e8f5192" path="/var/lib/kubelet/pods/ccb599aa-5195-41dc-9d9e-d61c6e8f5192/volumes" Jul 06 15:21:15.678111 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:15.678068 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"48c083a8-ef73-46dc-8a62-d4748a1d4b4c","Type":"ContainerStarted","Data":"90f8b20f238dd14cdff61b12ec408bc8b03e83d15b8f1c055d9aa16e06c3593f"} Jul 06 15:21:15.678536 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:15.678141 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"48c083a8-ef73-46dc-8a62-d4748a1d4b4c","Type":"ContainerStarted","Data":"532bdab1a489586faa80a87a77bbaa5c81982992567e039a2b36331e27ea04f4"} Jul 06 15:21:15.678536 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:15.678167 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"48c083a8-ef73-46dc-8a62-d4748a1d4b4c","Type":"ContainerStarted","Data":"25ca102ccabbe6f0fb0fb47bee0484582fbdb4257f68ff9d91dd8de7c5a2fae4"} Jul 06 15:21:15.678536 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:15.678191 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"48c083a8-ef73-46dc-8a62-d4748a1d4b4c","Type":"ContainerStarted","Data":"5cc7d72a6ea7f8655e312ca05054dd91fd5b7af733fd6ec9ddbf329fbb5b5b77"} Jul 06 15:21:15.678536 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:15.678215 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"48c083a8-ef73-46dc-8a62-d4748a1d4b4c","Type":"ContainerStarted","Data":"a3008c8aa98d89cc65614cd8dafeaee43112efe74248e22e95478e5ff8230445"} Jul 06 15:21:15.678536 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:15.678239 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"48c083a8-ef73-46dc-8a62-d4748a1d4b4c","Type":"ContainerStarted","Data":"d9d713118c8d64a1a43118f2e72b89b512b215a81de8c1107c170e1048b70626"} Jul 06 15:21:15.724332 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:15.724276 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.705082247 podStartE2EDuration="2.724258721s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.176474ms" isStatefulPod=false podCreationTimestamp="2026-07-06 15:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:21:15.721730852 +0000 UTC m=+222.335479388" watchObservedRunningTime="2026-07-06 15:21:15.724258721 +0000 UTC m=+222.338007256" Jul 06 15:21:16.477558 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:16.477523 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:21:16.578245 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:16.578215 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:21:16.673006 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:16.672970 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:21:18.956774 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:18.956741 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:21:19.045372 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:19.045337 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:21:19.060215 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:19.060190 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:21:28.517439 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:28.517404 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:21:28.588558 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:28.588526 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:21:28.712092 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:28.712059 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:21:34.831270 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:34.831234 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:21:34.949685 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:21:34.949603 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:22:10.742326 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:22:10.742244 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:22:10.839877 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:22:10.839847 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:22:14.060234 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:22:14.060195 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:22:14.075464 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:22:14.075440 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:22:14.914795 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:22:14.914768 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:22:34.053203 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:22:34.053168 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:22:34.606994 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:22:34.606962 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-mlw9w_91e0c855-5b32-4ca5-bab6-8e3aae20c1e0/console-operator/2.log" Jul 06 15:22:34.610772 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:22:34.610745 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hxpw_3eca6180-d678-4729-ad6f-979a025af841/ovn-acl-logging/0.log" Jul 06 15:22:34.614407 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:22:34.614384 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-mlw9w_91e0c855-5b32-4ca5-bab6-8e3aae20c1e0/console-operator/2.log" Jul 06 15:22:34.618818 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:22:34.618781 2570 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 15:22:34.619404 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:22:34.619385 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hxpw_3eca6180-d678-4729-ad6f-979a025af841/ovn-acl-logging/0.log" Jul 06 15:22:35.036810 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:22:35.036733 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:22:37.936302 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:22:37.936264 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:22:38.035181 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:22:38.035147 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:22:39.768152 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:22:39.768114 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:22:39.915819 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:22:39.915785 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:22:40.126596 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:22:40.126551 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:22:40.260578 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:22:40.260544 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:22:41.795229 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:22:41.795193 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:22:42.152913 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:22:42.152880 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:22:44.660276 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:22:44.660244 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:22:44.773931 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:22:44.773893 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:22:46.311655 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:22:46.311624 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:22:46.409884 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:22:46.409849 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:22:50.349614 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:22:50.349582 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:22:52.955148 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:22:52.955109 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:22:53.107745 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:22:53.107714 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:22:58.503263 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:22:58.503236 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:22:58.604027 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:22:58.603987 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:00.332108 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:00.332072 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:16.587561 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:16.587512 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:16.680848 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:16.680811 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:17.145210 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:17.145176 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:17.484794 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:17.484708 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:17.686156 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:17.686069 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:17.729093 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:17.729061 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9957545c6-6drld"] Jul 06 15:23:17.732461 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:17.732442 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-6drld" Jul 06 15:23:17.738260 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:23:17.738185 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-manager-service-cert\" is forbidden: User \"system:node:ip-10-0-130-108.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-108.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret" Jul 06 15:23:17.738893 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:23:17.738851 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"opendatahub-operator-controller-manager-9957545c6-6drld\" is forbidden: User \"system:node:ip-10-0-130-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-108.ec2.internal' and this object" podUID="65571a24-12fe-4e9c-a18b-2fa6eab3aa6e" pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-6drld" Jul 06 15:23:17.739014 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:23:17.738936 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-manager-dockercfg-8n8jq\" is forbidden: User \"system:node:ip-10-0-130-108.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-108.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8n8jq\"" type="*v1.Secret" Jul 06 15:23:17.739154 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:23:17.739122 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-130-108.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-108.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 06 15:23:17.739281 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:23:17.739133 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-webhook-cert\" is forbidden: User \"system:node:ip-10-0-130-108.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-108.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" type="*v1.Secret" Jul 06 15:23:17.739281 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:23:17.739190 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-130-108.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-108.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 06 15:23:17.783186 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:17.783155 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9957545c6-6drld"] Jul 06 15:23:17.804002 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:17.803978 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/65571a24-12fe-4e9c-a18b-2fa6eab3aa6e-manifests\") pod \"opendatahub-operator-controller-manager-9957545c6-6drld\" (UID: \"65571a24-12fe-4e9c-a18b-2fa6eab3aa6e\") " pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-6drld" Jul 06 15:23:17.804152 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:17.804005 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/65571a24-12fe-4e9c-a18b-2fa6eab3aa6e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9957545c6-6drld\" (UID: \"65571a24-12fe-4e9c-a18b-2fa6eab3aa6e\") " pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-6drld" Jul 06 15:23:17.804152 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:17.804029 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65571a24-12fe-4e9c-a18b-2fa6eab3aa6e-tmp\") pod \"opendatahub-operator-controller-manager-9957545c6-6drld\" (UID: \"65571a24-12fe-4e9c-a18b-2fa6eab3aa6e\") " pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-6drld" Jul 06 15:23:17.804152 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:17.804120 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65571a24-12fe-4e9c-a18b-2fa6eab3aa6e-webhook-cert\") pod \"opendatahub-operator-controller-manager-9957545c6-6drld\" (UID: \"65571a24-12fe-4e9c-a18b-2fa6eab3aa6e\") " pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-6drld" Jul 06 15:23:17.804152 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:17.804147 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm8xt\" (UniqueName: \"kubernetes.io/projected/65571a24-12fe-4e9c-a18b-2fa6eab3aa6e-kube-api-access-fm8xt\") pod \"opendatahub-operator-controller-manager-9957545c6-6drld\" (UID: \"65571a24-12fe-4e9c-a18b-2fa6eab3aa6e\") " pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-6drld" Jul 06 15:23:17.905190 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:17.905153 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65571a24-12fe-4e9c-a18b-2fa6eab3aa6e-tmp\") pod \"opendatahub-operator-controller-manager-9957545c6-6drld\" (UID: \"65571a24-12fe-4e9c-a18b-2fa6eab3aa6e\") " pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-6drld" Jul 06 15:23:17.905372 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:17.905221 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65571a24-12fe-4e9c-a18b-2fa6eab3aa6e-webhook-cert\") pod \"opendatahub-operator-controller-manager-9957545c6-6drld\" (UID: \"65571a24-12fe-4e9c-a18b-2fa6eab3aa6e\") " pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-6drld" Jul 06 15:23:17.905372 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:17.905240 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm8xt\" (UniqueName: \"kubernetes.io/projected/65571a24-12fe-4e9c-a18b-2fa6eab3aa6e-kube-api-access-fm8xt\") pod \"opendatahub-operator-controller-manager-9957545c6-6drld\" (UID: \"65571a24-12fe-4e9c-a18b-2fa6eab3aa6e\") " pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-6drld" Jul 06 15:23:17.905372 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:17.905287 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/65571a24-12fe-4e9c-a18b-2fa6eab3aa6e-manifests\") pod \"opendatahub-operator-controller-manager-9957545c6-6drld\" (UID: \"65571a24-12fe-4e9c-a18b-2fa6eab3aa6e\") " pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-6drld" Jul 06 15:23:17.905372 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:17.905301 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/65571a24-12fe-4e9c-a18b-2fa6eab3aa6e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9957545c6-6drld\" (UID: \"65571a24-12fe-4e9c-a18b-2fa6eab3aa6e\") " pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-6drld" Jul 06 15:23:17.905726 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:17.905702 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/65571a24-12fe-4e9c-a18b-2fa6eab3aa6e-manifests\") pod \"opendatahub-operator-controller-manager-9957545c6-6drld\" (UID: \"65571a24-12fe-4e9c-a18b-2fa6eab3aa6e\") " pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-6drld" Jul 06 15:23:17.907308 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:17.907276 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65571a24-12fe-4e9c-a18b-2fa6eab3aa6e-tmp\") pod \"opendatahub-operator-controller-manager-9957545c6-6drld\" (UID: \"65571a24-12fe-4e9c-a18b-2fa6eab3aa6e\") " pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-6drld" Jul 06 15:23:17.944937 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:17.944910 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:18.060648 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:18.060553 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:18.154190 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:18.154158 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:18.257867 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:18.257832 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:18.357840 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:18.357794 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:18.457282 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:18.457250 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:18.557148 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:18.557114 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:18.717415 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:18.717336 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 15:23:18.775216 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:18.775190 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 15:23:18.800977 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:18.800955 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8n8jq\"" Jul 06 15:23:18.905534 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:23:18.905485 2570 secret.go:189] Couldn't get secret opendatahub/opendatahub-operator-controller-manager-service-cert: failed to sync secret cache: timed out waiting for the condition Jul 06 15:23:18.905702 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:23:18.905590 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65571a24-12fe-4e9c-a18b-2fa6eab3aa6e-apiservice-cert podName:65571a24-12fe-4e9c-a18b-2fa6eab3aa6e nodeName:}" failed. No retries permitted until 2026-07-06 15:23:19.4055729 +0000 UTC m=+346.019321413 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "apiservice-cert" (UniqueName: "kubernetes.io/secret/65571a24-12fe-4e9c-a18b-2fa6eab3aa6e-apiservice-cert") pod "opendatahub-operator-controller-manager-9957545c6-6drld" (UID: "65571a24-12fe-4e9c-a18b-2fa6eab3aa6e") : failed to sync secret cache: timed out waiting for the condition Jul 06 15:23:18.905702 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:23:18.905486 2570 secret.go:189] Couldn't get secret opendatahub/opendatahub-operator-controller-manager-service-cert: failed to sync secret cache: timed out waiting for the condition Jul 06 15:23:18.905702 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:23:18.905672 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65571a24-12fe-4e9c-a18b-2fa6eab3aa6e-webhook-cert podName:65571a24-12fe-4e9c-a18b-2fa6eab3aa6e nodeName:}" failed. No retries permitted until 2026-07-06 15:23:19.405653897 +0000 UTC m=+346.019402427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/65571a24-12fe-4e9c-a18b-2fa6eab3aa6e-webhook-cert") pod "opendatahub-operator-controller-manager-9957545c6-6drld" (UID: "65571a24-12fe-4e9c-a18b-2fa6eab3aa6e") : failed to sync secret cache: timed out waiting for the condition Jul 06 15:23:19.036292 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:19.036215 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 15:23:19.038949 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:19.038930 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 15:23:19.039800 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:19.039779 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm8xt\" (UniqueName: \"kubernetes.io/projected/65571a24-12fe-4e9c-a18b-2fa6eab3aa6e-kube-api-access-fm8xt\") pod \"opendatahub-operator-controller-manager-9957545c6-6drld\" (UID: \"65571a24-12fe-4e9c-a18b-2fa6eab3aa6e\") " pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-6drld" Jul 06 15:23:19.418248 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:19.418208 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/65571a24-12fe-4e9c-a18b-2fa6eab3aa6e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9957545c6-6drld\" (UID: \"65571a24-12fe-4e9c-a18b-2fa6eab3aa6e\") " pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-6drld" Jul 06 15:23:19.418422 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:19.418300 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65571a24-12fe-4e9c-a18b-2fa6eab3aa6e-webhook-cert\") pod \"opendatahub-operator-controller-manager-9957545c6-6drld\" (UID: \"65571a24-12fe-4e9c-a18b-2fa6eab3aa6e\") " pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-6drld" Jul 06 15:23:19.420850 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:19.420825 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/65571a24-12fe-4e9c-a18b-2fa6eab3aa6e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9957545c6-6drld\" (UID: \"65571a24-12fe-4e9c-a18b-2fa6eab3aa6e\") " pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-6drld" Jul 06 15:23:19.420918 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:19.420860 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65571a24-12fe-4e9c-a18b-2fa6eab3aa6e-webhook-cert\") pod \"opendatahub-operator-controller-manager-9957545c6-6drld\" (UID: \"65571a24-12fe-4e9c-a18b-2fa6eab3aa6e\") " pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-6drld" Jul 06 15:23:19.542543 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:19.542469 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-6drld" Jul 06 15:23:19.675745 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:19.675721 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9957545c6-6drld"] Jul 06 15:23:19.676891 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:23:19.676871 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65571a24_12fe_4e9c_a18b_2fa6eab3aa6e.slice/crio-f76c5e9308c610a2c135d949cec0563d9fbb6a3fdca69bb545186e7ddf086c0b WatchSource:0}: Error finding container f76c5e9308c610a2c135d949cec0563d9fbb6a3fdca69bb545186e7ddf086c0b: Status 404 returned error can't find the container with id f76c5e9308c610a2c135d949cec0563d9fbb6a3fdca69bb545186e7ddf086c0b Jul 06 15:23:19.678653 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:19.678635 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 15:23:20.140166 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:20.140133 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-6drld" event={"ID":"65571a24-12fe-4e9c-a18b-2fa6eab3aa6e","Type":"ContainerStarted","Data":"f76c5e9308c610a2c135d949cec0563d9fbb6a3fdca69bb545186e7ddf086c0b"} Jul 06 15:23:23.155922 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:23.155888 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-6drld" event={"ID":"65571a24-12fe-4e9c-a18b-2fa6eab3aa6e","Type":"ContainerStarted","Data":"bb515a1286d75de7d896cae05d50ab87a6e91eaac119fc6e75cd4d22faea58e7"} Jul 06 15:23:24.160649 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:24.160616 2570 generic.go:356] "Generic (PLEG): container finished" podID="65571a24-12fe-4e9c-a18b-2fa6eab3aa6e" containerID="bb515a1286d75de7d896cae05d50ab87a6e91eaac119fc6e75cd4d22faea58e7" exitCode=0 Jul 06 15:23:24.161021 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:24.160663 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-6drld" event={"ID":"65571a24-12fe-4e9c-a18b-2fa6eab3aa6e","Type":"ContainerDied","Data":"bb515a1286d75de7d896cae05d50ab87a6e91eaac119fc6e75cd4d22faea58e7"} Jul 06 15:23:24.594573 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:24.594537 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:25.168452 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:25.168375 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-6drld" event={"ID":"65571a24-12fe-4e9c-a18b-2fa6eab3aa6e","Type":"ContainerStarted","Data":"019250e2f1b8ff32ffd3233d24f217a94da5db4af136904b3da5ef28cc343dee"} Jul 06 15:23:25.168833 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:25.168485 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-6drld" Jul 06 15:23:25.818972 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:25.818934 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:36.174547 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:36.174515 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-6drld" Jul 06 15:23:36.203019 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:36.202962 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-6drld" podStartSLOduration=14.922650157 podStartE2EDuration="19.202938701s" totalImagesPullingTime="3.233994469s" totalInitContainerRuntime="1.046294075s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:23:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:23:25.20312547 +0000 UTC m=+351.816874006" watchObservedRunningTime="2026-07-06 15:23:36.202938701 +0000 UTC m=+362.816687237" Jul 06 15:23:37.901228 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:37.901192 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:38.108142 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:38.108100 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:38.549678 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:38.549644 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:38.707854 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:38.707818 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:38.872360 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:38.872319 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:39.135431 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:39.135343 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:39.310436 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:39.310396 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:39.422698 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:39.422617 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:40.098421 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:40.098383 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:40.297009 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:40.296978 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:40.413693 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:40.413598 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:40.505655 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:40.505615 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:41.804006 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:41.803952 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:42.032181 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:42.032146 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:46.081129 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:46.081091 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:46.188650 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:46.188613 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:47.077349 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.077315 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d"] Jul 06 15:23:47.087031 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.087005 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" Jul 06 15:23:47.089994 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.089975 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 15:23:47.090274 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.090257 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-dnp9j\"" Jul 06 15:23:47.129496 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.129467 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d"] Jul 06 15:23:47.150958 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.150930 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:47.160148 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.160121 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa0b326-347a-4d2e-be45-5ff1c965dcf7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dgsb7d\" (UID: \"6aa0b326-347a-4d2e-be45-5ff1c965dcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" Jul 06 15:23:47.160265 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.160155 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa0b326-347a-4d2e-be45-5ff1c965dcf7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dgsb7d\" (UID: \"6aa0b326-347a-4d2e-be45-5ff1c965dcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" Jul 06 15:23:47.160265 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.160187 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa0b326-347a-4d2e-be45-5ff1c965dcf7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dgsb7d\" (UID: \"6aa0b326-347a-4d2e-be45-5ff1c965dcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" Jul 06 15:23:47.160350 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.160268 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa0b326-347a-4d2e-be45-5ff1c965dcf7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dgsb7d\" (UID: \"6aa0b326-347a-4d2e-be45-5ff1c965dcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" Jul 06 15:23:47.160350 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.160313 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa0b326-347a-4d2e-be45-5ff1c965dcf7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dgsb7d\" (UID: \"6aa0b326-347a-4d2e-be45-5ff1c965dcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" Jul 06 15:23:47.160423 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.160372 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa0b326-347a-4d2e-be45-5ff1c965dcf7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dgsb7d\" (UID: \"6aa0b326-347a-4d2e-be45-5ff1c965dcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" Jul 06 15:23:47.160423 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.160391 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vs29\" (UniqueName: \"kubernetes.io/projected/6aa0b326-347a-4d2e-be45-5ff1c965dcf7-kube-api-access-2vs29\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dgsb7d\" (UID: \"6aa0b326-347a-4d2e-be45-5ff1c965dcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" Jul 06 15:23:47.160423 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.160413 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa0b326-347a-4d2e-be45-5ff1c965dcf7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dgsb7d\" (UID: \"6aa0b326-347a-4d2e-be45-5ff1c965dcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" Jul 06 15:23:47.160558 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.160428 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aa0b326-347a-4d2e-be45-5ff1c965dcf7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dgsb7d\" (UID: \"6aa0b326-347a-4d2e-be45-5ff1c965dcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" Jul 06 15:23:47.260853 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.260819 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa0b326-347a-4d2e-be45-5ff1c965dcf7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dgsb7d\" (UID: \"6aa0b326-347a-4d2e-be45-5ff1c965dcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" Jul 06 15:23:47.261024 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.260864 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa0b326-347a-4d2e-be45-5ff1c965dcf7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dgsb7d\" (UID: \"6aa0b326-347a-4d2e-be45-5ff1c965dcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" Jul 06 15:23:47.261024 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.260889 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vs29\" (UniqueName: \"kubernetes.io/projected/6aa0b326-347a-4d2e-be45-5ff1c965dcf7-kube-api-access-2vs29\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dgsb7d\" (UID: \"6aa0b326-347a-4d2e-be45-5ff1c965dcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" Jul 06 15:23:47.261024 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.260915 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa0b326-347a-4d2e-be45-5ff1c965dcf7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dgsb7d\" (UID: \"6aa0b326-347a-4d2e-be45-5ff1c965dcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" Jul 06 15:23:47.261024 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.260932 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aa0b326-347a-4d2e-be45-5ff1c965dcf7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dgsb7d\" (UID: \"6aa0b326-347a-4d2e-be45-5ff1c965dcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" Jul 06 15:23:47.261024 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.260974 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa0b326-347a-4d2e-be45-5ff1c965dcf7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dgsb7d\" (UID: \"6aa0b326-347a-4d2e-be45-5ff1c965dcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" Jul 06 15:23:47.261024 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.260998 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa0b326-347a-4d2e-be45-5ff1c965dcf7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dgsb7d\" (UID: \"6aa0b326-347a-4d2e-be45-5ff1c965dcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" Jul 06 15:23:47.261359 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.261119 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa0b326-347a-4d2e-be45-5ff1c965dcf7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dgsb7d\" (UID: \"6aa0b326-347a-4d2e-be45-5ff1c965dcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" Jul 06 15:23:47.261359 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.261190 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa0b326-347a-4d2e-be45-5ff1c965dcf7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dgsb7d\" (UID: \"6aa0b326-347a-4d2e-be45-5ff1c965dcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" Jul 06 15:23:47.261359 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.261305 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa0b326-347a-4d2e-be45-5ff1c965dcf7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dgsb7d\" (UID: \"6aa0b326-347a-4d2e-be45-5ff1c965dcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" Jul 06 15:23:47.261648 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.261619 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa0b326-347a-4d2e-be45-5ff1c965dcf7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dgsb7d\" (UID: \"6aa0b326-347a-4d2e-be45-5ff1c965dcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" Jul 06 15:23:47.261826 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.261687 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa0b326-347a-4d2e-be45-5ff1c965dcf7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dgsb7d\" (UID: \"6aa0b326-347a-4d2e-be45-5ff1c965dcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" Jul 06 15:23:47.261935 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.261832 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa0b326-347a-4d2e-be45-5ff1c965dcf7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dgsb7d\" (UID: \"6aa0b326-347a-4d2e-be45-5ff1c965dcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" Jul 06 15:23:47.261995 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.261958 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa0b326-347a-4d2e-be45-5ff1c965dcf7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dgsb7d\" (UID: \"6aa0b326-347a-4d2e-be45-5ff1c965dcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" Jul 06 15:23:47.263483 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.263461 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa0b326-347a-4d2e-be45-5ff1c965dcf7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dgsb7d\" (UID: \"6aa0b326-347a-4d2e-be45-5ff1c965dcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" Jul 06 15:23:47.263601 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.263562 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa0b326-347a-4d2e-be45-5ff1c965dcf7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dgsb7d\" (UID: \"6aa0b326-347a-4d2e-be45-5ff1c965dcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" Jul 06 15:23:47.277044 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.277019 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vs29\" (UniqueName: \"kubernetes.io/projected/6aa0b326-347a-4d2e-be45-5ff1c965dcf7-kube-api-access-2vs29\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dgsb7d\" (UID: \"6aa0b326-347a-4d2e-be45-5ff1c965dcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" Jul 06 15:23:47.280019 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.279998 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aa0b326-347a-4d2e-be45-5ff1c965dcf7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-695cd9447dgsb7d\" (UID: \"6aa0b326-347a-4d2e-be45-5ff1c965dcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" Jul 06 15:23:47.282552 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.282529 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:47.388793 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.388709 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:47.396612 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.396583 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" Jul 06 15:23:47.541242 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:47.541215 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d"] Jul 06 15:23:47.543019 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:23:47.542996 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa0b326_347a_4d2e_be45_5ff1c965dcf7.slice/crio-46bacbf4ca74b1c91e9a250c6dfe08e59edc101ec55eaf6a82374c9e96ae348a WatchSource:0}: Error finding container 46bacbf4ca74b1c91e9a250c6dfe08e59edc101ec55eaf6a82374c9e96ae348a: Status 404 returned error can't find the container with id 46bacbf4ca74b1c91e9a250c6dfe08e59edc101ec55eaf6a82374c9e96ae348a Jul 06 15:23:48.266710 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:48.266664 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" event={"ID":"6aa0b326-347a-4d2e-be45-5ff1c965dcf7","Type":"ContainerStarted","Data":"46bacbf4ca74b1c91e9a250c6dfe08e59edc101ec55eaf6a82374c9e96ae348a"} Jul 06 15:23:48.890128 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:48.890075 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-59fc8bcc45-67btx"] Jul 06 15:23:48.892616 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:48.892593 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-59fc8bcc45-67btx" Jul 06 15:23:48.899381 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:48.899354 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 06 15:23:48.899525 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:48.899403 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 06 15:23:48.899525 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:48.899492 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-pmxsn\"" Jul 06 15:23:48.899659 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:48.899597 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 15:23:48.899716 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:48.899692 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 06 15:23:48.899766 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:48.899735 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 15:23:48.917272 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:48.917245 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-59fc8bcc45-67btx"] Jul 06 15:23:48.978910 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:48.978877 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4f192268-d9b0-426f-af32-e188ce5c25c1-cert\") pod \"lws-controller-manager-59fc8bcc45-67btx\" (UID: \"4f192268-d9b0-426f-af32-e188ce5c25c1\") " pod="openshift-lws-operator/lws-controller-manager-59fc8bcc45-67btx" Jul 06 15:23:48.979096 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:48.978925 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl6rv\" (UniqueName: \"kubernetes.io/projected/4f192268-d9b0-426f-af32-e188ce5c25c1-kube-api-access-xl6rv\") pod \"lws-controller-manager-59fc8bcc45-67btx\" (UID: \"4f192268-d9b0-426f-af32-e188ce5c25c1\") " pod="openshift-lws-operator/lws-controller-manager-59fc8bcc45-67btx" Jul 06 15:23:48.979096 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:48.978996 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4f192268-d9b0-426f-af32-e188ce5c25c1-manager-config\") pod \"lws-controller-manager-59fc8bcc45-67btx\" (UID: \"4f192268-d9b0-426f-af32-e188ce5c25c1\") " pod="openshift-lws-operator/lws-controller-manager-59fc8bcc45-67btx" Jul 06 15:23:48.979096 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:48.979029 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f192268-d9b0-426f-af32-e188ce5c25c1-metrics-cert\") pod \"lws-controller-manager-59fc8bcc45-67btx\" (UID: \"4f192268-d9b0-426f-af32-e188ce5c25c1\") " pod="openshift-lws-operator/lws-controller-manager-59fc8bcc45-67btx" Jul 06 15:23:49.079904 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:49.079873 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4f192268-d9b0-426f-af32-e188ce5c25c1-cert\") pod \"lws-controller-manager-59fc8bcc45-67btx\" (UID: \"4f192268-d9b0-426f-af32-e188ce5c25c1\") " pod="openshift-lws-operator/lws-controller-manager-59fc8bcc45-67btx" Jul 06 15:23:49.080078 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:49.079921 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl6rv\" (UniqueName: \"kubernetes.io/projected/4f192268-d9b0-426f-af32-e188ce5c25c1-kube-api-access-xl6rv\") pod \"lws-controller-manager-59fc8bcc45-67btx\" (UID: \"4f192268-d9b0-426f-af32-e188ce5c25c1\") " pod="openshift-lws-operator/lws-controller-manager-59fc8bcc45-67btx" Jul 06 15:23:49.080078 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:49.079969 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4f192268-d9b0-426f-af32-e188ce5c25c1-manager-config\") pod \"lws-controller-manager-59fc8bcc45-67btx\" (UID: \"4f192268-d9b0-426f-af32-e188ce5c25c1\") " pod="openshift-lws-operator/lws-controller-manager-59fc8bcc45-67btx" Jul 06 15:23:49.080078 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:49.079991 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f192268-d9b0-426f-af32-e188ce5c25c1-metrics-cert\") pod \"lws-controller-manager-59fc8bcc45-67btx\" (UID: \"4f192268-d9b0-426f-af32-e188ce5c25c1\") " pod="openshift-lws-operator/lws-controller-manager-59fc8bcc45-67btx" Jul 06 15:23:49.080791 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:49.080769 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4f192268-d9b0-426f-af32-e188ce5c25c1-manager-config\") pod \"lws-controller-manager-59fc8bcc45-67btx\" (UID: \"4f192268-d9b0-426f-af32-e188ce5c25c1\") " pod="openshift-lws-operator/lws-controller-manager-59fc8bcc45-67btx" Jul 06 15:23:49.083078 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:49.083051 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4f192268-d9b0-426f-af32-e188ce5c25c1-cert\") pod \"lws-controller-manager-59fc8bcc45-67btx\" (UID: \"4f192268-d9b0-426f-af32-e188ce5c25c1\") " pod="openshift-lws-operator/lws-controller-manager-59fc8bcc45-67btx" Jul 06 15:23:49.083189 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:49.083107 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f192268-d9b0-426f-af32-e188ce5c25c1-metrics-cert\") pod \"lws-controller-manager-59fc8bcc45-67btx\" (UID: \"4f192268-d9b0-426f-af32-e188ce5c25c1\") " pod="openshift-lws-operator/lws-controller-manager-59fc8bcc45-67btx" Jul 06 15:23:49.096184 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:49.096161 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl6rv\" (UniqueName: \"kubernetes.io/projected/4f192268-d9b0-426f-af32-e188ce5c25c1-kube-api-access-xl6rv\") pod \"lws-controller-manager-59fc8bcc45-67btx\" (UID: \"4f192268-d9b0-426f-af32-e188ce5c25c1\") " pod="openshift-lws-operator/lws-controller-manager-59fc8bcc45-67btx" Jul 06 15:23:49.204545 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:49.204444 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-59fc8bcc45-67btx" Jul 06 15:23:49.950675 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:49.950623 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:23:49.950993 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:49.950729 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:23:49.950993 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:49.950781 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:23:50.049557 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:50.049533 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-59fc8bcc45-67btx"] Jul 06 15:23:50.050755 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:23:50.050733 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f192268_d9b0_426f_af32_e188ce5c25c1.slice/crio-6ab364420c0680405cb4a1409a7ca00290704b42230fad1aaa28005d27b021c0 WatchSource:0}: Error finding container 6ab364420c0680405cb4a1409a7ca00290704b42230fad1aaa28005d27b021c0: Status 404 returned error can't find the container with id 6ab364420c0680405cb4a1409a7ca00290704b42230fad1aaa28005d27b021c0 Jul 06 15:23:50.276314 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:50.276219 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" event={"ID":"6aa0b326-347a-4d2e-be45-5ff1c965dcf7","Type":"ContainerStarted","Data":"6006bc5551d849de92eb43f54e5cd65aef55724b6235afcc38906f625df8891d"} Jul 06 15:23:50.277349 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:50.277320 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-59fc8bcc45-67btx" event={"ID":"4f192268-d9b0-426f-af32-e188ce5c25c1","Type":"ContainerStarted","Data":"6ab364420c0680405cb4a1409a7ca00290704b42230fad1aaa28005d27b021c0"} Jul 06 15:23:50.307072 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:50.307027 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" podStartSLOduration=0.901851094 podStartE2EDuration="3.307011862s" totalImagesPullingTime="2.405160768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:23:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:23:50.304459268 +0000 UTC m=+376.918207804" watchObservedRunningTime="2026-07-06 15:23:50.307011862 +0000 UTC m=+376.920760397" Jul 06 15:23:50.397555 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:50.397519 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" Jul 06 15:23:51.350999 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:51.350951 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:51.402157 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:51.402129 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" Jul 06 15:23:51.408733 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:51.408707 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-v26k2"] Jul 06 15:23:51.411243 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:51.411223 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-v26k2" Jul 06 15:23:51.415927 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:51.415849 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 06 15:23:51.416116 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:51.416060 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-rvqm6\"" Jul 06 15:23:51.453225 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:51.453191 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-v26k2"] Jul 06 15:23:51.500202 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:51.500162 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7ct8\" (UniqueName: \"kubernetes.io/projected/88aec541-064a-45e1-957e-8e8ef732b044-kube-api-access-d7ct8\") pod \"kserve-controller-manager-d6f4b65b8-v26k2\" (UID: \"88aec541-064a-45e1-957e-8e8ef732b044\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-v26k2" Jul 06 15:23:51.500375 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:51.500266 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88aec541-064a-45e1-957e-8e8ef732b044-cert\") pod \"kserve-controller-manager-d6f4b65b8-v26k2\" (UID: \"88aec541-064a-45e1-957e-8e8ef732b044\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-v26k2" Jul 06 15:23:51.570247 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:51.570217 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:51.601539 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:51.601431 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88aec541-064a-45e1-957e-8e8ef732b044-cert\") pod \"kserve-controller-manager-d6f4b65b8-v26k2\" (UID: \"88aec541-064a-45e1-957e-8e8ef732b044\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-v26k2" Jul 06 15:23:51.601703 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:51.601544 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7ct8\" (UniqueName: \"kubernetes.io/projected/88aec541-064a-45e1-957e-8e8ef732b044-kube-api-access-d7ct8\") pod \"kserve-controller-manager-d6f4b65b8-v26k2\" (UID: \"88aec541-064a-45e1-957e-8e8ef732b044\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-v26k2" Jul 06 15:23:51.601703 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:23:51.601617 2570 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 06 15:23:51.601821 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:23:51.601710 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88aec541-064a-45e1-957e-8e8ef732b044-cert podName:88aec541-064a-45e1-957e-8e8ef732b044 nodeName:}" failed. No retries permitted until 2026-07-06 15:23:52.101687306 +0000 UTC m=+378.715435821 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/88aec541-064a-45e1-957e-8e8ef732b044-cert") pod "kserve-controller-manager-d6f4b65b8-v26k2" (UID: "88aec541-064a-45e1-957e-8e8ef732b044") : secret "kserve-webhook-server-cert" not found Jul 06 15:23:51.634439 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:51.634409 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7ct8\" (UniqueName: \"kubernetes.io/projected/88aec541-064a-45e1-957e-8e8ef732b044-kube-api-access-d7ct8\") pod \"kserve-controller-manager-d6f4b65b8-v26k2\" (UID: \"88aec541-064a-45e1-957e-8e8ef732b044\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-v26k2" Jul 06 15:23:51.833659 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:51.833621 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:51.939371 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:51.939279 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:52.106607 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:52.106571 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88aec541-064a-45e1-957e-8e8ef732b044-cert\") pod \"kserve-controller-manager-d6f4b65b8-v26k2\" (UID: \"88aec541-064a-45e1-957e-8e8ef732b044\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-v26k2" Jul 06 15:23:52.109248 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:52.109223 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88aec541-064a-45e1-957e-8e8ef732b044-cert\") pod \"kserve-controller-manager-d6f4b65b8-v26k2\" (UID: \"88aec541-064a-45e1-957e-8e8ef732b044\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-v26k2" Jul 06 15:23:52.287095 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:52.287019 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" Jul 06 15:23:52.287931 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:52.287910 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-695cd9447dgsb7d" Jul 06 15:23:52.325308 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:52.325262 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-v26k2" Jul 06 15:23:52.495195 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:52.495158 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:23:52.981812 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:52.981779 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-v26k2"] Jul 06 15:23:52.986278 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:23:52.986248 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88aec541_064a_45e1_957e_8e8ef732b044.slice/crio-dc93bb0465e46e173f6b05e89c6383f24f9a8cb2c6f58743f0e6458b1a83f244 WatchSource:0}: Error finding container dc93bb0465e46e173f6b05e89c6383f24f9a8cb2c6f58743f0e6458b1a83f244: Status 404 returned error can't find the container with id dc93bb0465e46e173f6b05e89c6383f24f9a8cb2c6f58743f0e6458b1a83f244 Jul 06 15:23:53.292816 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:53.292718 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-59fc8bcc45-67btx" event={"ID":"4f192268-d9b0-426f-af32-e188ce5c25c1","Type":"ContainerStarted","Data":"0e29d93134b9c83f7b3021ee6723ad9910968581323b872bd02d98dde44981a8"} Jul 06 15:23:53.292985 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:53.292827 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-59fc8bcc45-67btx" Jul 06 15:23:53.293912 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:53.293890 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-v26k2" event={"ID":"88aec541-064a-45e1-957e-8e8ef732b044","Type":"ContainerStarted","Data":"dc93bb0465e46e173f6b05e89c6383f24f9a8cb2c6f58743f0e6458b1a83f244"} Jul 06 15:23:53.340030 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:53.339976 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-59fc8bcc45-67btx" podStartSLOduration=2.523158153 podStartE2EDuration="5.339959911s" totalImagesPullingTime="2.816801758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:23:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:23:53.338114955 +0000 UTC m=+379.951863479" watchObservedRunningTime="2026-07-06 15:23:53.339959911 +0000 UTC m=+379.953708445" Jul 06 15:23:57.313945 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:57.313912 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-v26k2" event={"ID":"88aec541-064a-45e1-957e-8e8ef732b044","Type":"ContainerStarted","Data":"1b9e42166494a68368d2f9905b038b2af01542857910c591739efccc149035ca"} Jul 06 15:23:57.314381 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:57.314026 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-v26k2" Jul 06 15:23:57.407410 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:23:57.407352 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-v26k2" podStartSLOduration=2.826791041 podStartE2EDuration="6.407330315s" totalImagesPullingTime="3.580539274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:23:57.388718692 +0000 UTC m=+384.002467228" watchObservedRunningTime="2026-07-06 15:23:57.407330315 +0000 UTC m=+384.021078854" Jul 06 15:24:04.300299 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:24:04.300272 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-59fc8bcc45-67btx" Jul 06 15:24:28.323253 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:24:28.323223 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-v26k2" Jul 06 15:24:35.135408 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:24:35.135365 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:24:35.208190 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:24:35.208160 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:24:36.880654 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:24:36.880620 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:24:36.947027 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:24:36.946998 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:24:39.298016 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:24:39.297973 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:24:39.482516 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:24:39.482475 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:24:39.581577 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:24:39.581534 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:24:40.218156 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:24:40.218125 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:24:40.418045 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:24:40.418014 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:25:13.512295 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:13.512208 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:25:13.582578 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:13.582537 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:25:16.489373 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:16.489330 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:25:19.563077 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:19.563037 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:25:21.512298 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:21.512262 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:25:21.867551 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:21.867483 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:25:22.624950 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:22.624910 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:25:24.483416 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:24.483384 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:25:24.853971 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:24.853929 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:25:24.953076 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:24.953048 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:25:25.052922 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:25.052889 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:25:26.515541 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:26.515487 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:25:26.751129 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:26.751100 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:25:29.207573 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:29.207535 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:25:30.732339 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:30.732302 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:25:30.833889 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:30.833844 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:25:31.110381 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:31.110348 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:25:31.207455 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:31.207412 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:25:32.725738 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:32.725697 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:25:33.124516 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:33.124472 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:25:40.188775 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:40.188741 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:25:40.319490 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:40.319454 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:25:42.624622 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:42.624592 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vx2ww"] Jul 06 15:25:42.628145 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:42.628119 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vx2ww" Jul 06 15:25:42.632085 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:42.632064 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 15:25:42.632207 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:42.632064 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-bhhp6\"" Jul 06 15:25:42.633109 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:42.633092 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 15:25:42.645468 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:42.645443 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h5vd\" (UniqueName: \"kubernetes.io/projected/765378c0-492a-45a3-9856-ce85a713feb8-kube-api-access-7h5vd\") pod \"limitador-operator-controller-manager-859d7b78bd-vx2ww\" (UID: \"765378c0-492a-45a3-9856-ce85a713feb8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vx2ww" Jul 06 15:25:42.654560 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:42.654531 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vx2ww"] Jul 06 15:25:42.657643 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:42.657619 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:25:42.746029 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:42.745997 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h5vd\" (UniqueName: \"kubernetes.io/projected/765378c0-492a-45a3-9856-ce85a713feb8-kube-api-access-7h5vd\") pod \"limitador-operator-controller-manager-859d7b78bd-vx2ww\" (UID: \"765378c0-492a-45a3-9856-ce85a713feb8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vx2ww" Jul 06 15:25:42.761898 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:42.761869 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:25:42.765175 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:42.765149 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h5vd\" (UniqueName: \"kubernetes.io/projected/765378c0-492a-45a3-9856-ce85a713feb8-kube-api-access-7h5vd\") pod \"limitador-operator-controller-manager-859d7b78bd-vx2ww\" (UID: \"765378c0-492a-45a3-9856-ce85a713feb8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vx2ww" Jul 06 15:25:42.938516 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:42.938414 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vx2ww" Jul 06 15:25:43.089412 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:43.089384 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vx2ww"] Jul 06 15:25:43.090852 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:25:43.090825 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod765378c0_492a_45a3_9856_ce85a713feb8.slice/crio-f12290341d561c55034a283d230993320972cd6e2bb55c1ec246c6fde83d32e1 WatchSource:0}: Error finding container f12290341d561c55034a283d230993320972cd6e2bb55c1ec246c6fde83d32e1: Status 404 returned error can't find the container with id f12290341d561c55034a283d230993320972cd6e2bb55c1ec246c6fde83d32e1 Jul 06 15:25:43.777673 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:43.777630 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vx2ww" event={"ID":"765378c0-492a-45a3-9856-ce85a713feb8","Type":"ContainerStarted","Data":"f12290341d561c55034a283d230993320972cd6e2bb55c1ec246c6fde83d32e1"} Jul 06 15:25:45.787436 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:45.787401 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vx2ww" event={"ID":"765378c0-492a-45a3-9856-ce85a713feb8","Type":"ContainerStarted","Data":"eebbbe548323de220c6d0684d33553819cc0d9830ee2986a5a409b2c92cb439f"} Jul 06 15:25:45.787872 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:45.787476 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vx2ww" Jul 06 15:25:45.813274 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:45.813215 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vx2ww" podStartSLOduration=1.584387583 podStartE2EDuration="3.813194032s" totalImagesPullingTime="2.228806449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:25:45.811708075 +0000 UTC m=+492.425456614" watchObservedRunningTime="2026-07-06 15:25:45.813194032 +0000 UTC m=+492.426942597" Jul 06 15:25:47.308719 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:47.308686 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:25:47.412773 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:47.412735 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:25:51.138855 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:51.138814 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:25:51.242326 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:51.242290 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:25:52.635389 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:52.635350 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:25:52.736411 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:52.736383 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:25:56.798321 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:56.798295 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vx2ww" Jul 06 15:25:59.968611 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:25:59.968576 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:00.068182 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:00.068149 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:02.400243 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:02.400209 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:02.481172 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:02.481139 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:07.136191 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:07.136143 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:07.257036 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:07.256999 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:11.331524 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:11.331471 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:11.409424 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:11.409390 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:20.347376 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:20.347331 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:20.448045 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:20.448008 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:22.821221 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:22.821166 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:22.894842 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:22.894803 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:25.385660 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.385574 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z"] Jul 06 15:26:25.389243 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.389220 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:26:25.393063 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.393040 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-f644w\"" Jul 06 15:26:25.407834 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.407804 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z"] Jul 06 15:26:25.413042 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.413009 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:25.527107 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.527066 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e54ba255-5f61-4cbb-a1f0-df4bb0d58876-istio-data\") pod \"maas-default-gateway-openshift-default-5df9b4f89-7h54z\" (UID: \"e54ba255-5f61-4cbb-a1f0-df4bb0d58876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:26:25.527107 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.527110 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e54ba255-5f61-4cbb-a1f0-df4bb0d58876-istio-token\") pod \"maas-default-gateway-openshift-default-5df9b4f89-7h54z\" (UID: \"e54ba255-5f61-4cbb-a1f0-df4bb0d58876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:26:25.527354 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.527168 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e54ba255-5f61-4cbb-a1f0-df4bb0d58876-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df9b4f89-7h54z\" (UID: \"e54ba255-5f61-4cbb-a1f0-df4bb0d58876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:26:25.527354 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.527207 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e54ba255-5f61-4cbb-a1f0-df4bb0d58876-workload-socket\") pod \"maas-default-gateway-openshift-default-5df9b4f89-7h54z\" (UID: \"e54ba255-5f61-4cbb-a1f0-df4bb0d58876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:26:25.527354 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.527227 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e54ba255-5f61-4cbb-a1f0-df4bb0d58876-workload-certs\") pod \"maas-default-gateway-openshift-default-5df9b4f89-7h54z\" (UID: \"e54ba255-5f61-4cbb-a1f0-df4bb0d58876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:26:25.527354 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.527322 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e54ba255-5f61-4cbb-a1f0-df4bb0d58876-credential-socket\") pod \"maas-default-gateway-openshift-default-5df9b4f89-7h54z\" (UID: \"e54ba255-5f61-4cbb-a1f0-df4bb0d58876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:26:25.527580 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.527389 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn5zt\" (UniqueName: \"kubernetes.io/projected/e54ba255-5f61-4cbb-a1f0-df4bb0d58876-kube-api-access-sn5zt\") pod \"maas-default-gateway-openshift-default-5df9b4f89-7h54z\" (UID: \"e54ba255-5f61-4cbb-a1f0-df4bb0d58876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:26:25.527580 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.527430 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e54ba255-5f61-4cbb-a1f0-df4bb0d58876-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df9b4f89-7h54z\" (UID: \"e54ba255-5f61-4cbb-a1f0-df4bb0d58876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:26:25.527580 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.527468 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e54ba255-5f61-4cbb-a1f0-df4bb0d58876-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df9b4f89-7h54z\" (UID: \"e54ba255-5f61-4cbb-a1f0-df4bb0d58876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:26:25.628805 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.628763 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn5zt\" (UniqueName: \"kubernetes.io/projected/e54ba255-5f61-4cbb-a1f0-df4bb0d58876-kube-api-access-sn5zt\") pod \"maas-default-gateway-openshift-default-5df9b4f89-7h54z\" (UID: \"e54ba255-5f61-4cbb-a1f0-df4bb0d58876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:26:25.628805 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.628812 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e54ba255-5f61-4cbb-a1f0-df4bb0d58876-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df9b4f89-7h54z\" (UID: \"e54ba255-5f61-4cbb-a1f0-df4bb0d58876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:26:25.629043 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.628831 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e54ba255-5f61-4cbb-a1f0-df4bb0d58876-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df9b4f89-7h54z\" (UID: \"e54ba255-5f61-4cbb-a1f0-df4bb0d58876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:26:25.629043 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.628889 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e54ba255-5f61-4cbb-a1f0-df4bb0d58876-istio-data\") pod \"maas-default-gateway-openshift-default-5df9b4f89-7h54z\" (UID: \"e54ba255-5f61-4cbb-a1f0-df4bb0d58876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:26:25.629043 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.628918 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e54ba255-5f61-4cbb-a1f0-df4bb0d58876-istio-token\") pod \"maas-default-gateway-openshift-default-5df9b4f89-7h54z\" (UID: \"e54ba255-5f61-4cbb-a1f0-df4bb0d58876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:26:25.629043 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.628944 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e54ba255-5f61-4cbb-a1f0-df4bb0d58876-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df9b4f89-7h54z\" (UID: \"e54ba255-5f61-4cbb-a1f0-df4bb0d58876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:26:25.629043 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.628974 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e54ba255-5f61-4cbb-a1f0-df4bb0d58876-workload-socket\") pod \"maas-default-gateway-openshift-default-5df9b4f89-7h54z\" (UID: \"e54ba255-5f61-4cbb-a1f0-df4bb0d58876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:26:25.629043 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.628998 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e54ba255-5f61-4cbb-a1f0-df4bb0d58876-workload-certs\") pod \"maas-default-gateway-openshift-default-5df9b4f89-7h54z\" (UID: \"e54ba255-5f61-4cbb-a1f0-df4bb0d58876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:26:25.629330 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.629056 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e54ba255-5f61-4cbb-a1f0-df4bb0d58876-credential-socket\") pod \"maas-default-gateway-openshift-default-5df9b4f89-7h54z\" (UID: \"e54ba255-5f61-4cbb-a1f0-df4bb0d58876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:26:25.629462 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.629434 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e54ba255-5f61-4cbb-a1f0-df4bb0d58876-istio-data\") pod \"maas-default-gateway-openshift-default-5df9b4f89-7h54z\" (UID: \"e54ba255-5f61-4cbb-a1f0-df4bb0d58876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:26:25.629613 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.629477 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e54ba255-5f61-4cbb-a1f0-df4bb0d58876-workload-socket\") pod \"maas-default-gateway-openshift-default-5df9b4f89-7h54z\" (UID: \"e54ba255-5f61-4cbb-a1f0-df4bb0d58876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:26:25.629613 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.629577 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e54ba255-5f61-4cbb-a1f0-df4bb0d58876-workload-certs\") pod \"maas-default-gateway-openshift-default-5df9b4f89-7h54z\" (UID: \"e54ba255-5f61-4cbb-a1f0-df4bb0d58876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:26:25.629742 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.629618 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e54ba255-5f61-4cbb-a1f0-df4bb0d58876-credential-socket\") pod \"maas-default-gateway-openshift-default-5df9b4f89-7h54z\" (UID: \"e54ba255-5f61-4cbb-a1f0-df4bb0d58876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:26:25.629742 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.629702 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e54ba255-5f61-4cbb-a1f0-df4bb0d58876-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df9b4f89-7h54z\" (UID: \"e54ba255-5f61-4cbb-a1f0-df4bb0d58876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:26:25.631115 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.631095 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e54ba255-5f61-4cbb-a1f0-df4bb0d58876-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df9b4f89-7h54z\" (UID: \"e54ba255-5f61-4cbb-a1f0-df4bb0d58876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:26:25.631363 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.631343 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e54ba255-5f61-4cbb-a1f0-df4bb0d58876-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df9b4f89-7h54z\" (UID: \"e54ba255-5f61-4cbb-a1f0-df4bb0d58876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:26:25.639079 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.639019 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn5zt\" (UniqueName: \"kubernetes.io/projected/e54ba255-5f61-4cbb-a1f0-df4bb0d58876-kube-api-access-sn5zt\") pod \"maas-default-gateway-openshift-default-5df9b4f89-7h54z\" (UID: \"e54ba255-5f61-4cbb-a1f0-df4bb0d58876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:26:25.639475 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.639455 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e54ba255-5f61-4cbb-a1f0-df4bb0d58876-istio-token\") pod \"maas-default-gateway-openshift-default-5df9b4f89-7h54z\" (UID: \"e54ba255-5f61-4cbb-a1f0-df4bb0d58876\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:26:25.699532 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.699466 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:26:25.765790 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.765759 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:25.846363 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.846289 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z"] Jul 06 15:26:25.848612 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:26:25.848584 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode54ba255_5f61_4cbb_a1f0_df4bb0d58876.slice/crio-19991752060c64f8047997de8972ff3f7923ceb810a1c5fd486716a3c3e71505 WatchSource:0}: Error finding container 19991752060c64f8047997de8972ff3f7923ceb810a1c5fd486716a3c3e71505: Status 404 returned error can't find the container with id 19991752060c64f8047997de8972ff3f7923ceb810a1c5fd486716a3c3e71505 Jul 06 15:26:25.850678 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.850638 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:26:25.850778 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.850719 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:26:25.850778 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.850765 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:26:25.957117 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.957084 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" event={"ID":"e54ba255-5f61-4cbb-a1f0-df4bb0d58876","Type":"ContainerStarted","Data":"5cb77851a82eecf75447f2d2013e423c97557735b6b1558a2179b392372fd112"} Jul 06 15:26:25.957306 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.957136 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" event={"ID":"e54ba255-5f61-4cbb-a1f0-df4bb0d58876","Type":"ContainerStarted","Data":"19991752060c64f8047997de8972ff3f7923ceb810a1c5fd486716a3c3e71505"} Jul 06 15:26:25.984975 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:25.984919 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" podStartSLOduration=0.984886788 podStartE2EDuration="984.886788ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:26:25.983836646 +0000 UTC m=+532.597585196" watchObservedRunningTime="2026-07-06 15:26:25.984886788 +0000 UTC m=+532.598635323" Jul 06 15:26:26.700076 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:26.700039 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:26:26.704841 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:26.704816 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:26:26.962114 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:26.962026 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:26:26.963160 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:26.963138 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:26:27.111789 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:27.111747 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:27.213423 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:27.213327 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:27.317513 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:27.317472 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:27.600520 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:27.600472 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:27.706251 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:27.706210 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:28.093959 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:28.093921 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:28.194689 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:28.194655 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:28.296456 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:28.296423 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-r88g9"] Jul 06 15:26:28.299249 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:28.299230 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-r88g9" Jul 06 15:26:28.302372 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:28.302349 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-trg2f\"" Jul 06 15:26:28.310866 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:28.310843 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-r88g9"] Jul 06 15:26:28.353060 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:28.352970 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9c8n\" (UniqueName: \"kubernetes.io/projected/5e0f9b61-28d7-46d8-90e6-cb8d04560c8c-kube-api-access-l9c8n\") pod \"authorino-75d556bd77-r88g9\" (UID: \"5e0f9b61-28d7-46d8-90e6-cb8d04560c8c\") " pod="kuadrant-system/authorino-75d556bd77-r88g9" Jul 06 15:26:28.453859 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:28.453825 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9c8n\" (UniqueName: \"kubernetes.io/projected/5e0f9b61-28d7-46d8-90e6-cb8d04560c8c-kube-api-access-l9c8n\") pod \"authorino-75d556bd77-r88g9\" (UID: \"5e0f9b61-28d7-46d8-90e6-cb8d04560c8c\") " pod="kuadrant-system/authorino-75d556bd77-r88g9" Jul 06 15:26:28.463085 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:28.463061 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9c8n\" (UniqueName: \"kubernetes.io/projected/5e0f9b61-28d7-46d8-90e6-cb8d04560c8c-kube-api-access-l9c8n\") pod \"authorino-75d556bd77-r88g9\" (UID: \"5e0f9b61-28d7-46d8-90e6-cb8d04560c8c\") " pod="kuadrant-system/authorino-75d556bd77-r88g9" Jul 06 15:26:28.621256 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:28.621168 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-r88g9" Jul 06 15:26:28.749882 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:28.749856 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-r88g9"] Jul 06 15:26:28.752066 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:26:28.752037 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e0f9b61_28d7_46d8_90e6_cb8d04560c8c.slice/crio-451cef7c437401b257e2f4331ea487190a13ebebbf4e4bde0035db42987cc9e6 WatchSource:0}: Error finding container 451cef7c437401b257e2f4331ea487190a13ebebbf4e4bde0035db42987cc9e6: Status 404 returned error can't find the container with id 451cef7c437401b257e2f4331ea487190a13ebebbf4e4bde0035db42987cc9e6 Jul 06 15:26:28.971150 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:28.971063 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-r88g9" event={"ID":"5e0f9b61-28d7-46d8-90e6-cb8d04560c8c","Type":"ContainerStarted","Data":"451cef7c437401b257e2f4331ea487190a13ebebbf4e4bde0035db42987cc9e6"} Jul 06 15:26:32.999049 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:32.999012 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-r88g9" event={"ID":"5e0f9b61-28d7-46d8-90e6-cb8d04560c8c","Type":"ContainerStarted","Data":"9b5e4918a463c41702dda6b8837cc190922e26ae7c17c8b8fc5dad1b2270a239"} Jul 06 15:26:33.019412 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:33.019356 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-r88g9" podStartSLOduration=1.483516596 podStartE2EDuration="5.0193391s" totalImagesPullingTime="3.535822504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:26:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:26:33.017244233 +0000 UTC m=+539.630992771" watchObservedRunningTime="2026-07-06 15:26:33.0193391 +0000 UTC m=+539.633087636" Jul 06 15:26:47.650333 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:47.650288 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:48.735837 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:26:48.735801 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:27:34.635167 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:34.635138 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-mlw9w_91e0c855-5b32-4ca5-bab6-8e3aae20c1e0/console-operator/2.log" Jul 06 15:27:34.638937 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:34.638919 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hxpw_3eca6180-d678-4729-ad6f-979a025af841/ovn-acl-logging/0.log" Jul 06 15:27:34.640632 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:34.640616 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-mlw9w_91e0c855-5b32-4ca5-bab6-8e3aae20c1e0/console-operator/2.log" Jul 06 15:27:34.644804 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:34.644786 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hxpw_3eca6180-d678-4729-ad6f-979a025af841/ovn-acl-logging/0.log" Jul 06 15:27:41.710977 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:41.710941 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:27:41.875157 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:41.875124 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:27:44.463923 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:44.463893 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-r88g9"] Jul 06 15:27:44.464341 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:44.464085 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-r88g9" podUID="5e0f9b61-28d7-46d8-90e6-cb8d04560c8c" containerName="authorino" containerID="cri-o://9b5e4918a463c41702dda6b8837cc190922e26ae7c17c8b8fc5dad1b2270a239" gracePeriod=30 Jul 06 15:27:44.705640 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:44.705617 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-r88g9" Jul 06 15:27:44.816554 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:44.816477 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9c8n\" (UniqueName: \"kubernetes.io/projected/5e0f9b61-28d7-46d8-90e6-cb8d04560c8c-kube-api-access-l9c8n\") pod \"5e0f9b61-28d7-46d8-90e6-cb8d04560c8c\" (UID: \"5e0f9b61-28d7-46d8-90e6-cb8d04560c8c\") " Jul 06 15:27:44.818685 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:44.818656 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e0f9b61-28d7-46d8-90e6-cb8d04560c8c-kube-api-access-l9c8n" (OuterVolumeSpecName: "kube-api-access-l9c8n") pod "5e0f9b61-28d7-46d8-90e6-cb8d04560c8c" (UID: "5e0f9b61-28d7-46d8-90e6-cb8d04560c8c"). InnerVolumeSpecName "kube-api-access-l9c8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:27:44.917859 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:44.917832 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9c8n\" (UniqueName: \"kubernetes.io/projected/5e0f9b61-28d7-46d8-90e6-cb8d04560c8c-kube-api-access-l9c8n\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:27:45.323584 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:45.323544 2570 generic.go:356] "Generic (PLEG): container finished" podID="5e0f9b61-28d7-46d8-90e6-cb8d04560c8c" containerID="9b5e4918a463c41702dda6b8837cc190922e26ae7c17c8b8fc5dad1b2270a239" exitCode=0 Jul 06 15:27:45.323584 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:45.323587 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-r88g9" Jul 06 15:27:45.323838 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:45.323630 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-r88g9" event={"ID":"5e0f9b61-28d7-46d8-90e6-cb8d04560c8c","Type":"ContainerDied","Data":"9b5e4918a463c41702dda6b8837cc190922e26ae7c17c8b8fc5dad1b2270a239"} Jul 06 15:27:45.323838 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:45.323673 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-r88g9" event={"ID":"5e0f9b61-28d7-46d8-90e6-cb8d04560c8c","Type":"ContainerDied","Data":"451cef7c437401b257e2f4331ea487190a13ebebbf4e4bde0035db42987cc9e6"} Jul 06 15:27:45.323838 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:45.323692 2570 scope.go:117] "RemoveContainer" containerID="9b5e4918a463c41702dda6b8837cc190922e26ae7c17c8b8fc5dad1b2270a239" Jul 06 15:27:45.331934 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:45.331917 2570 scope.go:117] "RemoveContainer" containerID="9b5e4918a463c41702dda6b8837cc190922e26ae7c17c8b8fc5dad1b2270a239" Jul 06 15:27:45.332189 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:27:45.332162 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b5e4918a463c41702dda6b8837cc190922e26ae7c17c8b8fc5dad1b2270a239\": container with ID starting with 9b5e4918a463c41702dda6b8837cc190922e26ae7c17c8b8fc5dad1b2270a239 not found: ID does not exist" containerID="9b5e4918a463c41702dda6b8837cc190922e26ae7c17c8b8fc5dad1b2270a239" Jul 06 15:27:45.332254 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:45.332197 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b5e4918a463c41702dda6b8837cc190922e26ae7c17c8b8fc5dad1b2270a239"} err="failed to get container status \"9b5e4918a463c41702dda6b8837cc190922e26ae7c17c8b8fc5dad1b2270a239\": rpc error: code = NotFound desc = could not find container \"9b5e4918a463c41702dda6b8837cc190922e26ae7c17c8b8fc5dad1b2270a239\": container with ID starting with 9b5e4918a463c41702dda6b8837cc190922e26ae7c17c8b8fc5dad1b2270a239 not found: ID does not exist" Jul 06 15:27:45.350246 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:45.350217 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-r88g9"] Jul 06 15:27:45.359053 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:45.359030 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-r88g9"] Jul 06 15:27:46.798514 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:46.798467 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e0f9b61-28d7-46d8-90e6-cb8d04560c8c" path="/var/lib/kubelet/pods/5e0f9b61-28d7-46d8-90e6-cb8d04560c8c/volumes" Jul 06 15:27:48.812914 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:48.812872 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:27:50.079208 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:50.079179 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:27:50.117093 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:50.117063 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7ff8768844-m7456"] Jul 06 15:27:50.117420 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:50.117407 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e0f9b61-28d7-46d8-90e6-cb8d04560c8c" containerName="authorino" Jul 06 15:27:50.117462 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:50.117422 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e0f9b61-28d7-46d8-90e6-cb8d04560c8c" containerName="authorino" Jul 06 15:27:50.117512 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:50.117491 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e0f9b61-28d7-46d8-90e6-cb8d04560c8c" containerName="authorino" Jul 06 15:27:50.120159 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:50.120140 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ff8768844-m7456" Jul 06 15:27:50.134476 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:50.134456 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 15:27:50.134629 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:50.134459 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 15:27:50.134629 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:50.134490 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7fmgx\"" Jul 06 15:27:50.149599 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:50.149574 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7ff8768844-m7456"] Jul 06 15:27:50.166283 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:50.166254 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgv9r\" (UniqueName: \"kubernetes.io/projected/14a95504-d1ac-48e7-9e76-3744865c824b-kube-api-access-tgv9r\") pod \"maas-controller-7ff8768844-m7456\" (UID: \"14a95504-d1ac-48e7-9e76-3744865c824b\") " pod="opendatahub/maas-controller-7ff8768844-m7456" Jul 06 15:27:50.166437 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:50.166320 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14a95504-d1ac-48e7-9e76-3744865c824b-webhook-cert\") pod \"maas-controller-7ff8768844-m7456\" (UID: \"14a95504-d1ac-48e7-9e76-3744865c824b\") " pod="opendatahub/maas-controller-7ff8768844-m7456" Jul 06 15:27:50.232791 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:50.232755 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:27:50.266961 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:50.266933 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14a95504-d1ac-48e7-9e76-3744865c824b-webhook-cert\") pod \"maas-controller-7ff8768844-m7456\" (UID: \"14a95504-d1ac-48e7-9e76-3744865c824b\") " pod="opendatahub/maas-controller-7ff8768844-m7456" Jul 06 15:27:50.267084 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:50.267020 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgv9r\" (UniqueName: \"kubernetes.io/projected/14a95504-d1ac-48e7-9e76-3744865c824b-kube-api-access-tgv9r\") pod \"maas-controller-7ff8768844-m7456\" (UID: \"14a95504-d1ac-48e7-9e76-3744865c824b\") " pod="opendatahub/maas-controller-7ff8768844-m7456" Jul 06 15:27:50.269397 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:50.269375 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14a95504-d1ac-48e7-9e76-3744865c824b-webhook-cert\") pod \"maas-controller-7ff8768844-m7456\" (UID: \"14a95504-d1ac-48e7-9e76-3744865c824b\") " pod="opendatahub/maas-controller-7ff8768844-m7456" Jul 06 15:27:50.277711 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:50.277683 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgv9r\" (UniqueName: \"kubernetes.io/projected/14a95504-d1ac-48e7-9e76-3744865c824b-kube-api-access-tgv9r\") pod \"maas-controller-7ff8768844-m7456\" (UID: \"14a95504-d1ac-48e7-9e76-3744865c824b\") " pod="opendatahub/maas-controller-7ff8768844-m7456" Jul 06 15:27:50.333693 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:50.333605 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:27:50.344019 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:50.343990 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-78d5649946-v6hvr"] Jul 06 15:27:50.346618 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:50.346599 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78d5649946-v6hvr" Jul 06 15:27:50.362452 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:50.362424 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78d5649946-v6hvr"] Jul 06 15:27:50.429824 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:50.429787 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ff8768844-m7456" Jul 06 15:27:50.468974 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:50.468944 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/51132d72-1bf2-4b8b-bfda-30023f590820-webhook-cert\") pod \"maas-controller-78d5649946-v6hvr\" (UID: \"51132d72-1bf2-4b8b-bfda-30023f590820\") " pod="opendatahub/maas-controller-78d5649946-v6hvr" Jul 06 15:27:50.468974 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:50.468981 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcss8\" (UniqueName: \"kubernetes.io/projected/51132d72-1bf2-4b8b-bfda-30023f590820-kube-api-access-fcss8\") pod \"maas-controller-78d5649946-v6hvr\" (UID: \"51132d72-1bf2-4b8b-bfda-30023f590820\") " pod="opendatahub/maas-controller-78d5649946-v6hvr" Jul 06 15:27:50.562525 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:50.562482 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7ff8768844-m7456"] Jul 06 15:27:50.564995 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:27:50.564968 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a95504_d1ac_48e7_9e76_3744865c824b.slice/crio-68d1cf6691a716392aaadaaaaaab6339bd91eccacac479cd491366968837446b WatchSource:0}: Error finding container 68d1cf6691a716392aaadaaaaaab6339bd91eccacac479cd491366968837446b: Status 404 returned error can't find the container with id 68d1cf6691a716392aaadaaaaaab6339bd91eccacac479cd491366968837446b Jul 06 15:27:50.570020 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:50.569993 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/51132d72-1bf2-4b8b-bfda-30023f590820-webhook-cert\") pod \"maas-controller-78d5649946-v6hvr\" (UID: \"51132d72-1bf2-4b8b-bfda-30023f590820\") " pod="opendatahub/maas-controller-78d5649946-v6hvr" Jul 06 15:27:50.570154 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:50.570035 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcss8\" (UniqueName: \"kubernetes.io/projected/51132d72-1bf2-4b8b-bfda-30023f590820-kube-api-access-fcss8\") pod \"maas-controller-78d5649946-v6hvr\" (UID: \"51132d72-1bf2-4b8b-bfda-30023f590820\") " pod="opendatahub/maas-controller-78d5649946-v6hvr" Jul 06 15:27:50.572721 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:50.572695 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/51132d72-1bf2-4b8b-bfda-30023f590820-webhook-cert\") pod \"maas-controller-78d5649946-v6hvr\" (UID: \"51132d72-1bf2-4b8b-bfda-30023f590820\") " pod="opendatahub/maas-controller-78d5649946-v6hvr" Jul 06 15:27:50.579226 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:50.579170 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcss8\" (UniqueName: \"kubernetes.io/projected/51132d72-1bf2-4b8b-bfda-30023f590820-kube-api-access-fcss8\") pod \"maas-controller-78d5649946-v6hvr\" (UID: \"51132d72-1bf2-4b8b-bfda-30023f590820\") " pod="opendatahub/maas-controller-78d5649946-v6hvr" Jul 06 15:27:50.656677 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:50.656587 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78d5649946-v6hvr" Jul 06 15:27:50.790103 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:50.790070 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78d5649946-v6hvr"] Jul 06 15:27:50.792036 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:27:50.792012 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51132d72_1bf2_4b8b_bfda_30023f590820.slice/crio-18bc99bb01c72b4eb9481b15b2418be3aa97b9bd07c6dab234f4ef5ce5e6f9ef WatchSource:0}: Error finding container 18bc99bb01c72b4eb9481b15b2418be3aa97b9bd07c6dab234f4ef5ce5e6f9ef: Status 404 returned error can't find the container with id 18bc99bb01c72b4eb9481b15b2418be3aa97b9bd07c6dab234f4ef5ce5e6f9ef Jul 06 15:27:51.356573 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:51.356533 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78d5649946-v6hvr" event={"ID":"51132d72-1bf2-4b8b-bfda-30023f590820","Type":"ContainerStarted","Data":"18bc99bb01c72b4eb9481b15b2418be3aa97b9bd07c6dab234f4ef5ce5e6f9ef"} Jul 06 15:27:51.358692 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:51.358627 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ff8768844-m7456" event={"ID":"14a95504-d1ac-48e7-9e76-3744865c824b","Type":"ContainerStarted","Data":"68d1cf6691a716392aaadaaaaaab6339bd91eccacac479cd491366968837446b"} Jul 06 15:27:53.374917 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:53.374882 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78d5649946-v6hvr" event={"ID":"51132d72-1bf2-4b8b-bfda-30023f590820","Type":"ContainerStarted","Data":"4dc8a3d95e440fbcdc8590fa7fd7c7bc1d725dce2b13616a2e33b16444a1678d"} Jul 06 15:27:53.375353 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:53.374969 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-78d5649946-v6hvr" Jul 06 15:27:53.376447 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:53.376424 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ff8768844-m7456" event={"ID":"14a95504-d1ac-48e7-9e76-3744865c824b","Type":"ContainerStarted","Data":"d04f480b01b534a21022d9b61a9f86398428c065feb7fab54102e92bf51b9e10"} Jul 06 15:27:53.376567 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:53.376548 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7ff8768844-m7456" Jul 06 15:27:53.437883 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:53.437832 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-78d5649946-v6hvr" podStartSLOduration=0.996369252 podStartE2EDuration="3.437812076s" totalImagesPullingTime="2.441442824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:27:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:27:53.435069128 +0000 UTC m=+620.048817664" watchObservedRunningTime="2026-07-06 15:27:53.437812076 +0000 UTC m=+620.051560613" Jul 06 15:27:53.495602 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:53.495484 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7ff8768844-m7456" podStartSLOduration=0.836239117 podStartE2EDuration="3.495466698s" totalImagesPullingTime="2.659227581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:27:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:27:53.494679316 +0000 UTC m=+620.108427853" watchObservedRunningTime="2026-07-06 15:27:53.495466698 +0000 UTC m=+620.109215235" Jul 06 15:27:53.524545 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:53.524495 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:27:53.632494 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:27:53.632461 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:04.386969 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:04.386899 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7ff8768844-m7456" Jul 06 15:28:04.387325 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:04.386975 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-78d5649946-v6hvr" Jul 06 15:28:04.466187 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:04.466152 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7ff8768844-m7456"] Jul 06 15:28:04.466402 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:04.466366 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-7ff8768844-m7456" podUID="14a95504-d1ac-48e7-9e76-3744865c824b" containerName="manager" containerID="cri-o://d04f480b01b534a21022d9b61a9f86398428c065feb7fab54102e92bf51b9e10" gracePeriod=30 Jul 06 15:28:04.707733 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:04.707704 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ff8768844-m7456" Jul 06 15:28:04.804290 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:04.804262 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14a95504-d1ac-48e7-9e76-3744865c824b-webhook-cert\") pod \"14a95504-d1ac-48e7-9e76-3744865c824b\" (UID: \"14a95504-d1ac-48e7-9e76-3744865c824b\") " Jul 06 15:28:04.804435 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:04.804415 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgv9r\" (UniqueName: \"kubernetes.io/projected/14a95504-d1ac-48e7-9e76-3744865c824b-kube-api-access-tgv9r\") pod \"14a95504-d1ac-48e7-9e76-3744865c824b\" (UID: \"14a95504-d1ac-48e7-9e76-3744865c824b\") " Jul 06 15:28:04.806561 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:04.806526 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14a95504-d1ac-48e7-9e76-3744865c824b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "14a95504-d1ac-48e7-9e76-3744865c824b" (UID: "14a95504-d1ac-48e7-9e76-3744865c824b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:28:04.806708 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:04.806560 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a95504-d1ac-48e7-9e76-3744865c824b-kube-api-access-tgv9r" (OuterVolumeSpecName: "kube-api-access-tgv9r") pod "14a95504-d1ac-48e7-9e76-3744865c824b" (UID: "14a95504-d1ac-48e7-9e76-3744865c824b"). InnerVolumeSpecName "kube-api-access-tgv9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:28:04.905363 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:04.905334 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgv9r\" (UniqueName: \"kubernetes.io/projected/14a95504-d1ac-48e7-9e76-3744865c824b-kube-api-access-tgv9r\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:28:04.905363 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:04.905363 2570 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14a95504-d1ac-48e7-9e76-3744865c824b-webhook-cert\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:28:05.431524 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:05.431473 2570 generic.go:356] "Generic (PLEG): container finished" podID="14a95504-d1ac-48e7-9e76-3744865c824b" containerID="d04f480b01b534a21022d9b61a9f86398428c065feb7fab54102e92bf51b9e10" exitCode=0 Jul 06 15:28:05.431939 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:05.431536 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ff8768844-m7456" Jul 06 15:28:05.431939 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:05.431566 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ff8768844-m7456" event={"ID":"14a95504-d1ac-48e7-9e76-3744865c824b","Type":"ContainerDied","Data":"d04f480b01b534a21022d9b61a9f86398428c065feb7fab54102e92bf51b9e10"} Jul 06 15:28:05.431939 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:05.431626 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ff8768844-m7456" event={"ID":"14a95504-d1ac-48e7-9e76-3744865c824b","Type":"ContainerDied","Data":"68d1cf6691a716392aaadaaaaaab6339bd91eccacac479cd491366968837446b"} Jul 06 15:28:05.431939 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:05.431655 2570 scope.go:117] "RemoveContainer" containerID="d04f480b01b534a21022d9b61a9f86398428c065feb7fab54102e92bf51b9e10" Jul 06 15:28:05.440298 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:05.440279 2570 scope.go:117] "RemoveContainer" containerID="d04f480b01b534a21022d9b61a9f86398428c065feb7fab54102e92bf51b9e10" Jul 06 15:28:05.440565 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:28:05.440546 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d04f480b01b534a21022d9b61a9f86398428c065feb7fab54102e92bf51b9e10\": container with ID starting with d04f480b01b534a21022d9b61a9f86398428c065feb7fab54102e92bf51b9e10 not found: ID does not exist" containerID="d04f480b01b534a21022d9b61a9f86398428c065feb7fab54102e92bf51b9e10" Jul 06 15:28:05.440628 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:05.440575 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d04f480b01b534a21022d9b61a9f86398428c065feb7fab54102e92bf51b9e10"} err="failed to get container status \"d04f480b01b534a21022d9b61a9f86398428c065feb7fab54102e92bf51b9e10\": rpc error: code = NotFound desc = could not find container \"d04f480b01b534a21022d9b61a9f86398428c065feb7fab54102e92bf51b9e10\": container with ID starting with d04f480b01b534a21022d9b61a9f86398428c065feb7fab54102e92bf51b9e10 not found: ID does not exist" Jul 06 15:28:05.483192 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:05.483161 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7ff8768844-m7456"] Jul 06 15:28:05.485679 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:05.485657 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7ff8768844-m7456"] Jul 06 15:28:06.798618 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:06.798585 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14a95504-d1ac-48e7-9e76-3744865c824b" path="/var/lib/kubelet/pods/14a95504-d1ac-48e7-9e76-3744865c824b/volumes" Jul 06 15:28:26.022307 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:26.022271 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:26.495607 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:26.495573 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-78ddc7f974-xfcw7"] Jul 06 15:28:26.495939 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:26.495926 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14a95504-d1ac-48e7-9e76-3744865c824b" containerName="manager" Jul 06 15:28:26.495982 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:26.495940 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="14a95504-d1ac-48e7-9e76-3744865c824b" containerName="manager" Jul 06 15:28:26.496019 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:26.495993 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="14a95504-d1ac-48e7-9e76-3744865c824b" containerName="manager" Jul 06 15:28:26.502679 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:26.502653 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-78ddc7f974-xfcw7" Jul 06 15:28:26.506184 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:26.506161 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-gjmr6\"" Jul 06 15:28:26.506314 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:26.506242 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 06 15:28:26.517624 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:26.517597 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-78ddc7f974-xfcw7"] Jul 06 15:28:26.591335 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:26.591300 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12f2c889-1197-4510-ae8b-42ba30b4da87-openshift-service-ca\") pod \"maas-api-78ddc7f974-xfcw7\" (UID: \"12f2c889-1197-4510-ae8b-42ba30b4da87\") " pod="opendatahub/maas-api-78ddc7f974-xfcw7" Jul 06 15:28:26.591548 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:26.591353 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-222s5\" (UniqueName: \"kubernetes.io/projected/12f2c889-1197-4510-ae8b-42ba30b4da87-kube-api-access-222s5\") pod \"maas-api-78ddc7f974-xfcw7\" (UID: \"12f2c889-1197-4510-ae8b-42ba30b4da87\") " pod="opendatahub/maas-api-78ddc7f974-xfcw7" Jul 06 15:28:26.591548 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:26.591433 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12f2c889-1197-4510-ae8b-42ba30b4da87-maas-api-tls\") pod \"maas-api-78ddc7f974-xfcw7\" (UID: \"12f2c889-1197-4510-ae8b-42ba30b4da87\") " pod="opendatahub/maas-api-78ddc7f974-xfcw7" Jul 06 15:28:26.692100 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:26.692069 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12f2c889-1197-4510-ae8b-42ba30b4da87-maas-api-tls\") pod \"maas-api-78ddc7f974-xfcw7\" (UID: \"12f2c889-1197-4510-ae8b-42ba30b4da87\") " pod="opendatahub/maas-api-78ddc7f974-xfcw7" Jul 06 15:28:26.692269 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:26.692136 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12f2c889-1197-4510-ae8b-42ba30b4da87-openshift-service-ca\") pod \"maas-api-78ddc7f974-xfcw7\" (UID: \"12f2c889-1197-4510-ae8b-42ba30b4da87\") " pod="opendatahub/maas-api-78ddc7f974-xfcw7" Jul 06 15:28:26.692269 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:26.692174 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-222s5\" (UniqueName: \"kubernetes.io/projected/12f2c889-1197-4510-ae8b-42ba30b4da87-kube-api-access-222s5\") pod \"maas-api-78ddc7f974-xfcw7\" (UID: \"12f2c889-1197-4510-ae8b-42ba30b4da87\") " pod="opendatahub/maas-api-78ddc7f974-xfcw7" Jul 06 15:28:26.692812 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:26.692791 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12f2c889-1197-4510-ae8b-42ba30b4da87-openshift-service-ca\") pod \"maas-api-78ddc7f974-xfcw7\" (UID: \"12f2c889-1197-4510-ae8b-42ba30b4da87\") " pod="opendatahub/maas-api-78ddc7f974-xfcw7" Jul 06 15:28:26.694702 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:26.694676 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12f2c889-1197-4510-ae8b-42ba30b4da87-maas-api-tls\") pod \"maas-api-78ddc7f974-xfcw7\" (UID: \"12f2c889-1197-4510-ae8b-42ba30b4da87\") " pod="opendatahub/maas-api-78ddc7f974-xfcw7" Jul 06 15:28:26.701392 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:26.701371 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-222s5\" (UniqueName: \"kubernetes.io/projected/12f2c889-1197-4510-ae8b-42ba30b4da87-kube-api-access-222s5\") pod \"maas-api-78ddc7f974-xfcw7\" (UID: \"12f2c889-1197-4510-ae8b-42ba30b4da87\") " pod="opendatahub/maas-api-78ddc7f974-xfcw7" Jul 06 15:28:26.813701 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:26.813617 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-78ddc7f974-xfcw7" Jul 06 15:28:26.977951 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:26.977910 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-78ddc7f974-xfcw7"] Jul 06 15:28:26.980446 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:28:26.980405 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12f2c889_1197_4510_ae8b_42ba30b4da87.slice/crio-bdd915867578914f41c13567ddf4f8d35a8a928878d9f11567e3e5da0cb55af8 WatchSource:0}: Error finding container bdd915867578914f41c13567ddf4f8d35a8a928878d9f11567e3e5da0cb55af8: Status 404 returned error can't find the container with id bdd915867578914f41c13567ddf4f8d35a8a928878d9f11567e3e5da0cb55af8 Jul 06 15:28:26.981770 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:26.981750 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 15:28:27.155183 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:27.155151 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-kpczm"] Jul 06 15:28:27.159812 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:27.159792 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-kpczm" Jul 06 15:28:27.162748 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:27.162725 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 06 15:28:27.162876 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:27.162725 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-swqh6\"" Jul 06 15:28:27.172821 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:27.172798 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-kpczm"] Jul 06 15:28:27.196468 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:27.196433 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkclf\" (UniqueName: \"kubernetes.io/projected/e67caf7d-9368-4d1e-b12b-074d1a9e6bfd-kube-api-access-vkclf\") pod \"payload-pre-processing-7964457696-kpczm\" (UID: \"e67caf7d-9368-4d1e-b12b-074d1a9e6bfd\") " pod="openshift-ingress/payload-pre-processing-7964457696-kpczm" Jul 06 15:28:27.196660 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:27.196485 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e67caf7d-9368-4d1e-b12b-074d1a9e6bfd-plugins-config-volume\") pod \"payload-pre-processing-7964457696-kpczm\" (UID: \"e67caf7d-9368-4d1e-b12b-074d1a9e6bfd\") " pod="openshift-ingress/payload-pre-processing-7964457696-kpczm" Jul 06 15:28:27.297720 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:27.297686 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkclf\" (UniqueName: \"kubernetes.io/projected/e67caf7d-9368-4d1e-b12b-074d1a9e6bfd-kube-api-access-vkclf\") pod \"payload-pre-processing-7964457696-kpczm\" (UID: \"e67caf7d-9368-4d1e-b12b-074d1a9e6bfd\") " pod="openshift-ingress/payload-pre-processing-7964457696-kpczm" Jul 06 15:28:27.297888 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:27.297736 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e67caf7d-9368-4d1e-b12b-074d1a9e6bfd-plugins-config-volume\") pod \"payload-pre-processing-7964457696-kpczm\" (UID: \"e67caf7d-9368-4d1e-b12b-074d1a9e6bfd\") " pod="openshift-ingress/payload-pre-processing-7964457696-kpczm" Jul 06 15:28:27.298407 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:27.298388 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e67caf7d-9368-4d1e-b12b-074d1a9e6bfd-plugins-config-volume\") pod \"payload-pre-processing-7964457696-kpczm\" (UID: \"e67caf7d-9368-4d1e-b12b-074d1a9e6bfd\") " pod="openshift-ingress/payload-pre-processing-7964457696-kpczm" Jul 06 15:28:27.307981 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:27.307948 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkclf\" (UniqueName: \"kubernetes.io/projected/e67caf7d-9368-4d1e-b12b-074d1a9e6bfd-kube-api-access-vkclf\") pod \"payload-pre-processing-7964457696-kpczm\" (UID: \"e67caf7d-9368-4d1e-b12b-074d1a9e6bfd\") " pod="openshift-ingress/payload-pre-processing-7964457696-kpczm" Jul 06 15:28:27.470537 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:27.470423 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-kpczm" Jul 06 15:28:27.532149 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:27.532118 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-78ddc7f974-xfcw7" event={"ID":"12f2c889-1197-4510-ae8b-42ba30b4da87","Type":"ContainerStarted","Data":"bdd915867578914f41c13567ddf4f8d35a8a928878d9f11567e3e5da0cb55af8"} Jul 06 15:28:27.611743 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:27.611716 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-kpczm"] Jul 06 15:28:27.613080 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:28:27.613037 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode67caf7d_9368_4d1e_b12b_074d1a9e6bfd.slice/crio-fee04a2c8ea833f00cd71fd16e9d465cb0a4b553d7425970c2f52c70df2cbee0 WatchSource:0}: Error finding container fee04a2c8ea833f00cd71fd16e9d465cb0a4b553d7425970c2f52c70df2cbee0: Status 404 returned error can't find the container with id fee04a2c8ea833f00cd71fd16e9d465cb0a4b553d7425970c2f52c70df2cbee0 Jul 06 15:28:27.626683 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:27.626585 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:28.565818 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:28.565778 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-kpczm" event={"ID":"e67caf7d-9368-4d1e-b12b-074d1a9e6bfd","Type":"ContainerStarted","Data":"fee04a2c8ea833f00cd71fd16e9d465cb0a4b553d7425970c2f52c70df2cbee0"} Jul 06 15:28:29.573874 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:29.573772 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-78ddc7f974-xfcw7" event={"ID":"12f2c889-1197-4510-ae8b-42ba30b4da87","Type":"ContainerStarted","Data":"cb98b5f6bb76cac3d76dcfd968d9ce3fa9f7f06e866a3c6203676ab735471e3f"} Jul 06 15:28:29.574323 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:29.573930 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-78ddc7f974-xfcw7" Jul 06 15:28:29.575320 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:29.575293 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-kpczm" event={"ID":"e67caf7d-9368-4d1e-b12b-074d1a9e6bfd","Type":"ContainerStarted","Data":"1451b183664e7e10c61dabb2835f8e01e28d7ce7b51516b482412fd2450c668e"} Jul 06 15:28:29.575457 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:29.575425 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-kpczm" Jul 06 15:28:29.598395 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:29.598342 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-78ddc7f974-xfcw7" podStartSLOduration=1.8521286890000002 podStartE2EDuration="3.598325266s" totalImagesPullingTime="1.746196577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:28:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:28:29.597309657 +0000 UTC m=+656.211058194" watchObservedRunningTime="2026-07-06 15:28:29.598325266 +0000 UTC m=+656.212073802" Jul 06 15:28:29.621318 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:29.621269 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-kpczm" podStartSLOduration=1.033136993 podStartE2EDuration="2.62125237s" totalImagesPullingTime="1.588115377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:28:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:28:29.620466261 +0000 UTC m=+656.234214796" watchObservedRunningTime="2026-07-06 15:28:29.62125237 +0000 UTC m=+656.235000906" Jul 06 15:28:34.600635 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:34.600601 2570 generic.go:356] "Generic (PLEG): container finished" podID="e54ba255-5f61-4cbb-a1f0-df4bb0d58876" containerID="5cb77851a82eecf75447f2d2013e423c97557735b6b1558a2179b392372fd112" exitCode=0 Jul 06 15:28:34.601034 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:34.600673 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" event={"ID":"e54ba255-5f61-4cbb-a1f0-df4bb0d58876","Type":"ContainerDied","Data":"5cb77851a82eecf75447f2d2013e423c97557735b6b1558a2179b392372fd112"} Jul 06 15:28:34.601076 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:34.601035 2570 scope.go:117] "RemoveContainer" containerID="5cb77851a82eecf75447f2d2013e423c97557735b6b1558a2179b392372fd112" Jul 06 15:28:34.602024 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:34.601994 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:28:34.602108 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:34.602079 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:28:34.602160 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:34.602141 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:28:34.900511 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:34.900475 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:35.000159 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:35.000124 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:35.585900 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:35.585869 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-78ddc7f974-xfcw7" Jul 06 15:28:35.608199 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:35.608167 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" event={"ID":"e54ba255-5f61-4cbb-a1f0-df4bb0d58876","Type":"ContainerStarted","Data":"8974adda711f38824fa53179e73788e7a3cd0d0cbc5b416216bd132ded5dcd0a"} Jul 06 15:28:35.700703 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:35.700666 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:28:35.700969 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:35.700941 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5df9b4f89-7h54z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body= Jul 06 15:28:35.701056 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:35.701000 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" podUID="e54ba255-5f61-4cbb-a1f0-df4bb0d58876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" Jul 06 15:28:36.620175 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:36.620138 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:36.700476 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:36.700439 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5df9b4f89-7h54z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body= Jul 06 15:28:36.700678 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:36.700490 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" podUID="e54ba255-5f61-4cbb-a1f0-df4bb0d58876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" Jul 06 15:28:36.719126 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:36.719092 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:37.700239 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:37.700200 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5df9b4f89-7h54z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body= Jul 06 15:28:37.700710 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:37.700253 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" podUID="e54ba255-5f61-4cbb-a1f0-df4bb0d58876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" Jul 06 15:28:38.017272 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:38.017192 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:38.700677 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:38.700639 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5df9b4f89-7h54z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body= Jul 06 15:28:38.701137 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:38.700701 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" podUID="e54ba255-5f61-4cbb-a1f0-df4bb0d58876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" Jul 06 15:28:39.700424 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:39.700383 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5df9b4f89-7h54z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body= Jul 06 15:28:39.700647 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:39.700443 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" podUID="e54ba255-5f61-4cbb-a1f0-df4bb0d58876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" Jul 06 15:28:40.582104 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:40.582065 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-kpczm" Jul 06 15:28:40.704714 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:40.704686 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:28:40.705079 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:40.705051 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:28:40.705821 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:40.705797 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df9b4f89-7h54z" Jul 06 15:28:46.578159 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:46.578124 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:52.160409 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:52.160377 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt"] Jul 06 15:28:52.165005 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:52.164982 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt" Jul 06 15:28:52.168215 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:52.168195 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 15:28:52.169690 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:52.169669 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 15:28:52.169796 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:52.169714 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8k4vd\"" Jul 06 15:28:52.169843 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:52.169803 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 06 15:28:52.173732 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:52.173713 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt"] Jul 06 15:28:52.207126 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:52.207093 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:52.329693 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:52.329666 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef5f40da-0145-420a-8ddc-9144d6898cb7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt\" (UID: \"ef5f40da-0145-420a-8ddc-9144d6898cb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt" Jul 06 15:28:52.329872 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:52.329763 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef5f40da-0145-420a-8ddc-9144d6898cb7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt\" (UID: \"ef5f40da-0145-420a-8ddc-9144d6898cb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt" Jul 06 15:28:52.329872 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:52.329831 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef5f40da-0145-420a-8ddc-9144d6898cb7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt\" (UID: \"ef5f40da-0145-420a-8ddc-9144d6898cb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt" Jul 06 15:28:52.329872 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:52.329855 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef5f40da-0145-420a-8ddc-9144d6898cb7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt\" (UID: \"ef5f40da-0145-420a-8ddc-9144d6898cb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt" Jul 06 15:28:52.329998 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:52.329927 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef5f40da-0145-420a-8ddc-9144d6898cb7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt\" (UID: \"ef5f40da-0145-420a-8ddc-9144d6898cb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt" Jul 06 15:28:52.329998 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:52.329963 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pstkl\" (UniqueName: \"kubernetes.io/projected/ef5f40da-0145-420a-8ddc-9144d6898cb7-kube-api-access-pstkl\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt\" (UID: \"ef5f40da-0145-420a-8ddc-9144d6898cb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt" Jul 06 15:28:52.430716 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:52.430623 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef5f40da-0145-420a-8ddc-9144d6898cb7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt\" (UID: \"ef5f40da-0145-420a-8ddc-9144d6898cb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt" Jul 06 15:28:52.430716 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:52.430685 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef5f40da-0145-420a-8ddc-9144d6898cb7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt\" (UID: \"ef5f40da-0145-420a-8ddc-9144d6898cb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt" Jul 06 15:28:52.430716 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:52.430711 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef5f40da-0145-420a-8ddc-9144d6898cb7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt\" (UID: \"ef5f40da-0145-420a-8ddc-9144d6898cb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt" Jul 06 15:28:52.431008 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:52.430832 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef5f40da-0145-420a-8ddc-9144d6898cb7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt\" (UID: \"ef5f40da-0145-420a-8ddc-9144d6898cb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt" Jul 06 15:28:52.431008 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:52.430870 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pstkl\" (UniqueName: \"kubernetes.io/projected/ef5f40da-0145-420a-8ddc-9144d6898cb7-kube-api-access-pstkl\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt\" (UID: \"ef5f40da-0145-420a-8ddc-9144d6898cb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt" Jul 06 15:28:52.431008 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:52.430912 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef5f40da-0145-420a-8ddc-9144d6898cb7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt\" (UID: \"ef5f40da-0145-420a-8ddc-9144d6898cb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt" Jul 06 15:28:52.431192 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:52.431032 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef5f40da-0145-420a-8ddc-9144d6898cb7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt\" (UID: \"ef5f40da-0145-420a-8ddc-9144d6898cb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt" Jul 06 15:28:52.431192 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:52.431134 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef5f40da-0145-420a-8ddc-9144d6898cb7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt\" (UID: \"ef5f40da-0145-420a-8ddc-9144d6898cb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt" Jul 06 15:28:52.431192 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:52.431165 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef5f40da-0145-420a-8ddc-9144d6898cb7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt\" (UID: \"ef5f40da-0145-420a-8ddc-9144d6898cb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt" Jul 06 15:28:52.433106 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:52.433075 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef5f40da-0145-420a-8ddc-9144d6898cb7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt\" (UID: \"ef5f40da-0145-420a-8ddc-9144d6898cb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt" Jul 06 15:28:52.433289 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:52.433275 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef5f40da-0145-420a-8ddc-9144d6898cb7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt\" (UID: \"ef5f40da-0145-420a-8ddc-9144d6898cb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt" Jul 06 15:28:52.440451 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:52.440426 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pstkl\" (UniqueName: \"kubernetes.io/projected/ef5f40da-0145-420a-8ddc-9144d6898cb7-kube-api-access-pstkl\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt\" (UID: \"ef5f40da-0145-420a-8ddc-9144d6898cb7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt" Jul 06 15:28:52.476343 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:52.476309 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt" Jul 06 15:28:52.622270 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:52.622243 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt"] Jul 06 15:28:52.624075 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:28:52.624043 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef5f40da_0145_420a_8ddc_9144d6898cb7.slice/crio-6ee72c536732e4069fd98e801fe5ab5f20ce953b58090eeb951eb0b39c601712 WatchSource:0}: Error finding container 6ee72c536732e4069fd98e801fe5ab5f20ce953b58090eeb951eb0b39c601712: Status 404 returned error can't find the container with id 6ee72c536732e4069fd98e801fe5ab5f20ce953b58090eeb951eb0b39c601712 Jul 06 15:28:52.685325 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:52.685242 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt" event={"ID":"ef5f40da-0145-420a-8ddc-9144d6898cb7","Type":"ContainerStarted","Data":"6ee72c536732e4069fd98e801fe5ab5f20ce953b58090eeb951eb0b39c601712"} Jul 06 15:28:54.696707 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:54.696672 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt" event={"ID":"ef5f40da-0145-420a-8ddc-9144d6898cb7","Type":"ContainerStarted","Data":"06bd96bfb92eba55b898530ec3dda2525fa5cfa2938e6500d3f50a6687da3852"} Jul 06 15:28:54.718020 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:54.717962 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt" podStartSLOduration=1.51693489 podStartE2EDuration="2.717942114s" totalImagesPullingTime="1.201007224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:28:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:28:54.717640592 +0000 UTC m=+681.331389129" watchObservedRunningTime="2026-07-06 15:28:54.717942114 +0000 UTC m=+681.331690650" Jul 06 15:28:58.146315 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:28:58.146282 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:29:01.904086 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:01.904055 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:29:02.476775 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:02.476734 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt" Jul 06 15:29:02.476971 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:02.476829 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt" Jul 06 15:29:02.488950 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:02.488923 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt" Jul 06 15:29:02.744989 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:02.744902 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt" Jul 06 15:29:12.335294 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:12.335262 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-d6zzt"] Jul 06 15:29:12.338873 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:12.338836 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d6zzt" Jul 06 15:29:12.341745 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:12.341723 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 06 15:29:12.349782 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:12.349753 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-d6zzt"] Jul 06 15:29:12.388658 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:12.388606 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:29:12.408602 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:12.408573 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t52m2\" (UniqueName: \"kubernetes.io/projected/22e4c371-1e9b-433e-b0d3-658cb2894eae-kube-api-access-t52m2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d6zzt\" (UID: \"22e4c371-1e9b-433e-b0d3-658cb2894eae\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d6zzt" Jul 06 15:29:12.408785 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:12.408613 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22e4c371-1e9b-433e-b0d3-658cb2894eae-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d6zzt\" (UID: \"22e4c371-1e9b-433e-b0d3-658cb2894eae\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d6zzt" Jul 06 15:29:12.408785 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:12.408640 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22e4c371-1e9b-433e-b0d3-658cb2894eae-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d6zzt\" (UID: \"22e4c371-1e9b-433e-b0d3-658cb2894eae\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d6zzt" Jul 06 15:29:12.408785 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:12.408703 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22e4c371-1e9b-433e-b0d3-658cb2894eae-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d6zzt\" (UID: \"22e4c371-1e9b-433e-b0d3-658cb2894eae\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d6zzt" Jul 06 15:29:12.408949 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:12.408806 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22e4c371-1e9b-433e-b0d3-658cb2894eae-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d6zzt\" (UID: \"22e4c371-1e9b-433e-b0d3-658cb2894eae\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d6zzt" Jul 06 15:29:12.408949 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:12.408836 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22e4c371-1e9b-433e-b0d3-658cb2894eae-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d6zzt\" (UID: \"22e4c371-1e9b-433e-b0d3-658cb2894eae\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d6zzt" Jul 06 15:29:12.510211 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:12.510173 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t52m2\" (UniqueName: \"kubernetes.io/projected/22e4c371-1e9b-433e-b0d3-658cb2894eae-kube-api-access-t52m2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d6zzt\" (UID: \"22e4c371-1e9b-433e-b0d3-658cb2894eae\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d6zzt" Jul 06 15:29:12.510211 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:12.510215 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22e4c371-1e9b-433e-b0d3-658cb2894eae-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d6zzt\" (UID: \"22e4c371-1e9b-433e-b0d3-658cb2894eae\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d6zzt" Jul 06 15:29:12.510467 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:12.510241 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22e4c371-1e9b-433e-b0d3-658cb2894eae-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d6zzt\" (UID: \"22e4c371-1e9b-433e-b0d3-658cb2894eae\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d6zzt" Jul 06 15:29:12.510467 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:12.510264 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22e4c371-1e9b-433e-b0d3-658cb2894eae-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d6zzt\" (UID: \"22e4c371-1e9b-433e-b0d3-658cb2894eae\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d6zzt" Jul 06 15:29:12.510467 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:12.510346 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22e4c371-1e9b-433e-b0d3-658cb2894eae-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d6zzt\" (UID: \"22e4c371-1e9b-433e-b0d3-658cb2894eae\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d6zzt" Jul 06 15:29:12.510467 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:12.510374 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22e4c371-1e9b-433e-b0d3-658cb2894eae-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d6zzt\" (UID: \"22e4c371-1e9b-433e-b0d3-658cb2894eae\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d6zzt" Jul 06 15:29:12.510772 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:12.510738 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22e4c371-1e9b-433e-b0d3-658cb2894eae-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d6zzt\" (UID: \"22e4c371-1e9b-433e-b0d3-658cb2894eae\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d6zzt" Jul 06 15:29:12.510831 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:12.510790 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22e4c371-1e9b-433e-b0d3-658cb2894eae-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d6zzt\" (UID: \"22e4c371-1e9b-433e-b0d3-658cb2894eae\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d6zzt" Jul 06 15:29:12.510898 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:12.510879 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22e4c371-1e9b-433e-b0d3-658cb2894eae-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d6zzt\" (UID: \"22e4c371-1e9b-433e-b0d3-658cb2894eae\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d6zzt" Jul 06 15:29:12.512705 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:12.512684 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22e4c371-1e9b-433e-b0d3-658cb2894eae-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d6zzt\" (UID: \"22e4c371-1e9b-433e-b0d3-658cb2894eae\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d6zzt" Jul 06 15:29:12.512850 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:12.512836 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22e4c371-1e9b-433e-b0d3-658cb2894eae-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d6zzt\" (UID: \"22e4c371-1e9b-433e-b0d3-658cb2894eae\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d6zzt" Jul 06 15:29:12.519468 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:12.519447 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t52m2\" (UniqueName: \"kubernetes.io/projected/22e4c371-1e9b-433e-b0d3-658cb2894eae-kube-api-access-t52m2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d6zzt\" (UID: \"22e4c371-1e9b-433e-b0d3-658cb2894eae\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d6zzt" Jul 06 15:29:12.651204 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:12.651106 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d6zzt" Jul 06 15:29:12.787060 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:12.787037 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-d6zzt"] Jul 06 15:29:12.789141 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:29:12.789107 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e4c371_1e9b_433e_b0d3_658cb2894eae.slice/crio-0dd7e4bc4ad1f6ececb18356453d160d31b5bfe0796d416ef35a590cae865bee WatchSource:0}: Error finding container 0dd7e4bc4ad1f6ececb18356453d160d31b5bfe0796d416ef35a590cae865bee: Status 404 returned error can't find the container with id 0dd7e4bc4ad1f6ececb18356453d160d31b5bfe0796d416ef35a590cae865bee Jul 06 15:29:13.783792 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:13.783762 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d6zzt" event={"ID":"22e4c371-1e9b-433e-b0d3-658cb2894eae","Type":"ContainerStarted","Data":"f4933af099790b8c76a864f014425d2ac00583d54122696e342a4e9070c04aea"} Jul 06 15:29:13.784179 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:13.783812 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d6zzt" event={"ID":"22e4c371-1e9b-433e-b0d3-658cb2894eae","Type":"ContainerStarted","Data":"0dd7e4bc4ad1f6ececb18356453d160d31b5bfe0796d416ef35a590cae865bee"} Jul 06 15:29:13.811302 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:13.808326 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d6zzt" podStartSLOduration=1.561961396 podStartE2EDuration="1.808304168s" totalImagesPullingTime="246.342772ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:29:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:29:13.804684931 +0000 UTC m=+700.418433477" watchObservedRunningTime="2026-07-06 15:29:13.808304168 +0000 UTC m=+700.422052705" Jul 06 15:29:21.338262 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:21.338222 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw"] Jul 06 15:29:21.341939 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:21.341919 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw" Jul 06 15:29:21.344790 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:21.344770 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 06 15:29:21.355196 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:21.355173 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw"] Jul 06 15:29:21.376706 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:21.376649 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:29:21.390708 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:21.390677 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a239c966-2d9d-4709-8f4e-43eb160cad14-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw\" (UID: \"a239c966-2d9d-4709-8f4e-43eb160cad14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw" Jul 06 15:29:21.390856 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:21.390710 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a239c966-2d9d-4709-8f4e-43eb160cad14-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw\" (UID: \"a239c966-2d9d-4709-8f4e-43eb160cad14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw" Jul 06 15:29:21.390856 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:21.390756 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a239c966-2d9d-4709-8f4e-43eb160cad14-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw\" (UID: \"a239c966-2d9d-4709-8f4e-43eb160cad14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw" Jul 06 15:29:21.390856 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:21.390833 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqh8q\" (UniqueName: \"kubernetes.io/projected/a239c966-2d9d-4709-8f4e-43eb160cad14-kube-api-access-tqh8q\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw\" (UID: \"a239c966-2d9d-4709-8f4e-43eb160cad14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw" Jul 06 15:29:21.391006 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:21.390945 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a239c966-2d9d-4709-8f4e-43eb160cad14-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw\" (UID: \"a239c966-2d9d-4709-8f4e-43eb160cad14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw" Jul 06 15:29:21.391006 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:21.390983 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a239c966-2d9d-4709-8f4e-43eb160cad14-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw\" (UID: \"a239c966-2d9d-4709-8f4e-43eb160cad14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw" Jul 06 15:29:21.491748 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:21.491709 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a239c966-2d9d-4709-8f4e-43eb160cad14-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw\" (UID: \"a239c966-2d9d-4709-8f4e-43eb160cad14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw" Jul 06 15:29:21.491748 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:21.491754 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a239c966-2d9d-4709-8f4e-43eb160cad14-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw\" (UID: \"a239c966-2d9d-4709-8f4e-43eb160cad14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw" Jul 06 15:29:21.492007 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:21.491792 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a239c966-2d9d-4709-8f4e-43eb160cad14-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw\" (UID: \"a239c966-2d9d-4709-8f4e-43eb160cad14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw" Jul 06 15:29:21.492007 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:21.491816 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a239c966-2d9d-4709-8f4e-43eb160cad14-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw\" (UID: \"a239c966-2d9d-4709-8f4e-43eb160cad14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw" Jul 06 15:29:21.492007 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:21.491846 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a239c966-2d9d-4709-8f4e-43eb160cad14-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw\" (UID: \"a239c966-2d9d-4709-8f4e-43eb160cad14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw" Jul 06 15:29:21.492007 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:21.491877 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqh8q\" (UniqueName: \"kubernetes.io/projected/a239c966-2d9d-4709-8f4e-43eb160cad14-kube-api-access-tqh8q\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw\" (UID: \"a239c966-2d9d-4709-8f4e-43eb160cad14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw" Jul 06 15:29:21.492218 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:21.492193 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a239c966-2d9d-4709-8f4e-43eb160cad14-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw\" (UID: \"a239c966-2d9d-4709-8f4e-43eb160cad14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw" Jul 06 15:29:21.492267 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:21.492252 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a239c966-2d9d-4709-8f4e-43eb160cad14-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw\" (UID: \"a239c966-2d9d-4709-8f4e-43eb160cad14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw" Jul 06 15:29:21.492321 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:21.492299 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a239c966-2d9d-4709-8f4e-43eb160cad14-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw\" (UID: \"a239c966-2d9d-4709-8f4e-43eb160cad14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw" Jul 06 15:29:21.494460 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:21.494433 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a239c966-2d9d-4709-8f4e-43eb160cad14-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw\" (UID: \"a239c966-2d9d-4709-8f4e-43eb160cad14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw" Jul 06 15:29:21.494789 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:21.494769 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a239c966-2d9d-4709-8f4e-43eb160cad14-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw\" (UID: \"a239c966-2d9d-4709-8f4e-43eb160cad14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw" Jul 06 15:29:21.500791 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:21.500763 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqh8q\" (UniqueName: \"kubernetes.io/projected/a239c966-2d9d-4709-8f4e-43eb160cad14-kube-api-access-tqh8q\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw\" (UID: \"a239c966-2d9d-4709-8f4e-43eb160cad14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw" Jul 06 15:29:21.652093 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:21.651991 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw" Jul 06 15:29:21.787020 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:21.786994 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw"] Jul 06 15:29:21.788874 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:29:21.788846 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda239c966_2d9d_4709_8f4e_43eb160cad14.slice/crio-95bb4ea4a97b28fb4212c9d45389a1ba20595a8206207c0f9a5f4b5e929bc5b9 WatchSource:0}: Error finding container 95bb4ea4a97b28fb4212c9d45389a1ba20595a8206207c0f9a5f4b5e929bc5b9: Status 404 returned error can't find the container with id 95bb4ea4a97b28fb4212c9d45389a1ba20595a8206207c0f9a5f4b5e929bc5b9 Jul 06 15:29:21.821145 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:21.821104 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw" event={"ID":"a239c966-2d9d-4709-8f4e-43eb160cad14","Type":"ContainerStarted","Data":"95bb4ea4a97b28fb4212c9d45389a1ba20595a8206207c0f9a5f4b5e929bc5b9"} Jul 06 15:29:22.651329 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:22.651292 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d6zzt" Jul 06 15:29:22.651766 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:22.651361 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d6zzt" Jul 06 15:29:22.663956 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:22.663924 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d6zzt" Jul 06 15:29:22.826573 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:22.826533 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw" event={"ID":"a239c966-2d9d-4709-8f4e-43eb160cad14","Type":"ContainerStarted","Data":"e4c5105b3a04c2fd7b6f7f325b349c263fc0c83a2792d5252527a9162103fbd0"} Jul 06 15:29:22.838054 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:22.838033 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d6zzt" Jul 06 15:29:22.866377 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:22.866322 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw" podStartSLOduration=1.649944684 podStartE2EDuration="1.866304264s" totalImagesPullingTime="216.35958ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:29:22.864669893 +0000 UTC m=+709.478418430" watchObservedRunningTime="2026-07-06 15:29:22.866304264 +0000 UTC m=+709.480052799" Jul 06 15:29:31.652558 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:31.652425 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw" Jul 06 15:29:31.652558 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:31.652541 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw" Jul 06 15:29:31.664811 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:31.664784 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw" Jul 06 15:29:31.879965 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:31.879936 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw" Jul 06 15:29:36.615942 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:36.615906 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-rzzwm"] Jul 06 15:29:36.619165 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:36.619148 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rzzwm" Jul 06 15:29:36.624051 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:36.624027 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 06 15:29:36.634586 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:36.634565 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-rzzwm"] Jul 06 15:29:36.690480 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:36.690435 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:29:36.737954 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:36.737924 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7338594-417e-4104-afd8-4e8e6ced455c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rzzwm\" (UID: \"c7338594-417e-4104-afd8-4e8e6ced455c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rzzwm" Jul 06 15:29:36.737954 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:36.737959 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7338594-417e-4104-afd8-4e8e6ced455c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rzzwm\" (UID: \"c7338594-417e-4104-afd8-4e8e6ced455c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rzzwm" Jul 06 15:29:36.738191 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:36.737988 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9shts\" (UniqueName: \"kubernetes.io/projected/c7338594-417e-4104-afd8-4e8e6ced455c-kube-api-access-9shts\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rzzwm\" (UID: \"c7338594-417e-4104-afd8-4e8e6ced455c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rzzwm" Jul 06 15:29:36.738191 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:36.738078 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7338594-417e-4104-afd8-4e8e6ced455c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rzzwm\" (UID: \"c7338594-417e-4104-afd8-4e8e6ced455c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rzzwm" Jul 06 15:29:36.738191 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:36.738147 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7338594-417e-4104-afd8-4e8e6ced455c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rzzwm\" (UID: \"c7338594-417e-4104-afd8-4e8e6ced455c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rzzwm" Jul 06 15:29:36.738314 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:36.738197 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7338594-417e-4104-afd8-4e8e6ced455c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rzzwm\" (UID: \"c7338594-417e-4104-afd8-4e8e6ced455c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rzzwm" Jul 06 15:29:36.839744 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:36.839706 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7338594-417e-4104-afd8-4e8e6ced455c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rzzwm\" (UID: \"c7338594-417e-4104-afd8-4e8e6ced455c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rzzwm" Jul 06 15:29:36.839744 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:36.839749 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7338594-417e-4104-afd8-4e8e6ced455c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rzzwm\" (UID: \"c7338594-417e-4104-afd8-4e8e6ced455c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rzzwm" Jul 06 15:29:36.840046 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:36.839791 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9shts\" (UniqueName: \"kubernetes.io/projected/c7338594-417e-4104-afd8-4e8e6ced455c-kube-api-access-9shts\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rzzwm\" (UID: \"c7338594-417e-4104-afd8-4e8e6ced455c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rzzwm" Jul 06 15:29:36.840046 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:36.839828 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7338594-417e-4104-afd8-4e8e6ced455c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rzzwm\" (UID: \"c7338594-417e-4104-afd8-4e8e6ced455c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rzzwm" Jul 06 15:29:36.840046 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:36.839844 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7338594-417e-4104-afd8-4e8e6ced455c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rzzwm\" (UID: \"c7338594-417e-4104-afd8-4e8e6ced455c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rzzwm" Jul 06 15:29:36.840046 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:36.839891 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7338594-417e-4104-afd8-4e8e6ced455c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rzzwm\" (UID: \"c7338594-417e-4104-afd8-4e8e6ced455c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rzzwm" Jul 06 15:29:36.840274 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:36.840248 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7338594-417e-4104-afd8-4e8e6ced455c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rzzwm\" (UID: \"c7338594-417e-4104-afd8-4e8e6ced455c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rzzwm" Jul 06 15:29:36.840334 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:36.840258 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7338594-417e-4104-afd8-4e8e6ced455c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rzzwm\" (UID: \"c7338594-417e-4104-afd8-4e8e6ced455c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rzzwm" Jul 06 15:29:36.840334 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:36.840316 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7338594-417e-4104-afd8-4e8e6ced455c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rzzwm\" (UID: \"c7338594-417e-4104-afd8-4e8e6ced455c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rzzwm" Jul 06 15:29:36.842119 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:36.842097 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7338594-417e-4104-afd8-4e8e6ced455c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rzzwm\" (UID: \"c7338594-417e-4104-afd8-4e8e6ced455c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rzzwm" Jul 06 15:29:36.842299 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:36.842282 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7338594-417e-4104-afd8-4e8e6ced455c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rzzwm\" (UID: \"c7338594-417e-4104-afd8-4e8e6ced455c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rzzwm" Jul 06 15:29:36.850250 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:36.850226 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9shts\" (UniqueName: \"kubernetes.io/projected/c7338594-417e-4104-afd8-4e8e6ced455c-kube-api-access-9shts\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rzzwm\" (UID: \"c7338594-417e-4104-afd8-4e8e6ced455c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rzzwm" Jul 06 15:29:36.929792 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:36.929699 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rzzwm" Jul 06 15:29:37.078592 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:37.078561 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-rzzwm"] Jul 06 15:29:37.079047 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:29:37.079023 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7338594_417e_4104_afd8_4e8e6ced455c.slice/crio-910fd4ffd76e809610cd4c08083f227ca4625f8b23bd29650913acde6ff50f08 WatchSource:0}: Error finding container 910fd4ffd76e809610cd4c08083f227ca4625f8b23bd29650913acde6ff50f08: Status 404 returned error can't find the container with id 910fd4ffd76e809610cd4c08083f227ca4625f8b23bd29650913acde6ff50f08 Jul 06 15:29:37.897335 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:37.897291 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rzzwm" event={"ID":"c7338594-417e-4104-afd8-4e8e6ced455c","Type":"ContainerStarted","Data":"3699349aeebdadfd7ebd594a6f7fd2cb680c6a15a306241dc9a3beb2c9228d96"} Jul 06 15:29:37.897851 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:37.897349 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rzzwm" event={"ID":"c7338594-417e-4104-afd8-4e8e6ced455c","Type":"ContainerStarted","Data":"910fd4ffd76e809610cd4c08083f227ca4625f8b23bd29650913acde6ff50f08"} Jul 06 15:29:37.918129 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:37.918066 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rzzwm" podStartSLOduration=1.6285809279999999 podStartE2EDuration="1.918049074s" totalImagesPullingTime="289.468146ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:29:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:29:37.917575903 +0000 UTC m=+724.531324440" watchObservedRunningTime="2026-07-06 15:29:37.918049074 +0000 UTC m=+724.531797627" Jul 06 15:29:46.929912 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:46.929874 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rzzwm" Jul 06 15:29:46.930322 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:46.929947 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rzzwm" Jul 06 15:29:46.943547 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:46.943510 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rzzwm" Jul 06 15:29:46.954967 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:46.954936 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rzzwm" Jul 06 15:29:52.623449 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:52.623415 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:29:52.724593 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:29:52.724558 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:30:12.562189 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:30:12.562153 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:30:12.634739 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:30:12.634705 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:30:38.391176 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:30:38.391138 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:30:38.449243 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:30:38.449210 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:31:29.915248 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:31:29.915208 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:31:30.036169 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:31:30.036122 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:31:31.228401 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:31:31.228365 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:31:31.350434 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:31:31.350394 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:31:36.244615 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:31:36.244569 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:31:41.334331 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:31:41.334298 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:32:34.665900 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:34.665869 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-mlw9w_91e0c855-5b32-4ca5-bab6-8e3aae20c1e0/console-operator/2.log" Jul 06 15:32:34.668213 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:34.668192 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-mlw9w_91e0c855-5b32-4ca5-bab6-8e3aae20c1e0/console-operator/2.log" Jul 06 15:32:34.670021 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:34.670000 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hxpw_3eca6180-d678-4729-ad6f-979a025af841/ovn-acl-logging/0.log" Jul 06 15:32:34.672359 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:34.672341 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hxpw_3eca6180-d678-4729-ad6f-979a025af841/ovn-acl-logging/0.log" Jul 06 15:32:42.718012 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:42.717975 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:32:42.831422 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:42.831387 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78d5649946-v6hvr"] Jul 06 15:32:42.831640 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:42.831603 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-78d5649946-v6hvr" podUID="51132d72-1bf2-4b8b-bfda-30023f590820" containerName="manager" containerID="cri-o://4dc8a3d95e440fbcdc8590fa7fd7c7bc1d725dce2b13616a2e33b16444a1678d" gracePeriod=30 Jul 06 15:32:43.084279 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:43.084253 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78d5649946-v6hvr" Jul 06 15:32:43.149407 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:43.149372 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcss8\" (UniqueName: \"kubernetes.io/projected/51132d72-1bf2-4b8b-bfda-30023f590820-kube-api-access-fcss8\") pod \"51132d72-1bf2-4b8b-bfda-30023f590820\" (UID: \"51132d72-1bf2-4b8b-bfda-30023f590820\") " Jul 06 15:32:43.149595 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:43.149483 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/51132d72-1bf2-4b8b-bfda-30023f590820-webhook-cert\") pod \"51132d72-1bf2-4b8b-bfda-30023f590820\" (UID: \"51132d72-1bf2-4b8b-bfda-30023f590820\") " Jul 06 15:32:43.151913 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:43.151869 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51132d72-1bf2-4b8b-bfda-30023f590820-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "51132d72-1bf2-4b8b-bfda-30023f590820" (UID: "51132d72-1bf2-4b8b-bfda-30023f590820"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:32:43.152200 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:43.152177 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51132d72-1bf2-4b8b-bfda-30023f590820-kube-api-access-fcss8" (OuterVolumeSpecName: "kube-api-access-fcss8") pod "51132d72-1bf2-4b8b-bfda-30023f590820" (UID: "51132d72-1bf2-4b8b-bfda-30023f590820"). InnerVolumeSpecName "kube-api-access-fcss8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:32:43.250269 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:43.250190 2570 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/51132d72-1bf2-4b8b-bfda-30023f590820-webhook-cert\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:32:43.250269 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:43.250217 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcss8\" (UniqueName: \"kubernetes.io/projected/51132d72-1bf2-4b8b-bfda-30023f590820-kube-api-access-fcss8\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:32:43.784019 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:43.783979 2570 generic.go:356] "Generic (PLEG): container finished" podID="51132d72-1bf2-4b8b-bfda-30023f590820" containerID="4dc8a3d95e440fbcdc8590fa7fd7c7bc1d725dce2b13616a2e33b16444a1678d" exitCode=0 Jul 06 15:32:43.784451 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:43.784033 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78d5649946-v6hvr" Jul 06 15:32:43.784451 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:43.784060 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78d5649946-v6hvr" event={"ID":"51132d72-1bf2-4b8b-bfda-30023f590820","Type":"ContainerDied","Data":"4dc8a3d95e440fbcdc8590fa7fd7c7bc1d725dce2b13616a2e33b16444a1678d"} Jul 06 15:32:43.784451 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:43.784146 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78d5649946-v6hvr" event={"ID":"51132d72-1bf2-4b8b-bfda-30023f590820","Type":"ContainerDied","Data":"18bc99bb01c72b4eb9481b15b2418be3aa97b9bd07c6dab234f4ef5ce5e6f9ef"} Jul 06 15:32:43.784451 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:43.784174 2570 scope.go:117] "RemoveContainer" containerID="4dc8a3d95e440fbcdc8590fa7fd7c7bc1d725dce2b13616a2e33b16444a1678d" Jul 06 15:32:43.793564 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:43.793546 2570 scope.go:117] "RemoveContainer" containerID="4dc8a3d95e440fbcdc8590fa7fd7c7bc1d725dce2b13616a2e33b16444a1678d" Jul 06 15:32:43.793835 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:32:43.793816 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dc8a3d95e440fbcdc8590fa7fd7c7bc1d725dce2b13616a2e33b16444a1678d\": container with ID starting with 4dc8a3d95e440fbcdc8590fa7fd7c7bc1d725dce2b13616a2e33b16444a1678d not found: ID does not exist" containerID="4dc8a3d95e440fbcdc8590fa7fd7c7bc1d725dce2b13616a2e33b16444a1678d" Jul 06 15:32:43.793927 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:43.793845 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dc8a3d95e440fbcdc8590fa7fd7c7bc1d725dce2b13616a2e33b16444a1678d"} err="failed to get container status \"4dc8a3d95e440fbcdc8590fa7fd7c7bc1d725dce2b13616a2e33b16444a1678d\": rpc error: code = NotFound desc = could not find container \"4dc8a3d95e440fbcdc8590fa7fd7c7bc1d725dce2b13616a2e33b16444a1678d\": container with ID starting with 4dc8a3d95e440fbcdc8590fa7fd7c7bc1d725dce2b13616a2e33b16444a1678d not found: ID does not exist" Jul 06 15:32:43.811873 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:43.811847 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78d5649946-v6hvr"] Jul 06 15:32:43.814254 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:43.814233 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-78d5649946-v6hvr"] Jul 06 15:32:44.553383 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:44.553346 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:32:44.665469 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:44.665423 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-78d5649946-8r2fm"] Jul 06 15:32:44.665884 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:44.665868 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51132d72-1bf2-4b8b-bfda-30023f590820" containerName="manager" Jul 06 15:32:44.665884 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:44.665885 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="51132d72-1bf2-4b8b-bfda-30023f590820" containerName="manager" Jul 06 15:32:44.666017 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:44.665964 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="51132d72-1bf2-4b8b-bfda-30023f590820" containerName="manager" Jul 06 15:32:44.670618 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:44.670599 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78d5649946-8r2fm" Jul 06 15:32:44.674578 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:44.674555 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 15:32:44.674696 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:44.674555 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7fmgx\"" Jul 06 15:32:44.674696 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:44.674591 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 15:32:44.680720 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:44.680694 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78d5649946-8r2fm"] Jul 06 15:32:44.766808 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:44.766771 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twh6c\" (UniqueName: \"kubernetes.io/projected/89e49b52-85a2-4a39-bf50-13c848cde6f2-kube-api-access-twh6c\") pod \"maas-controller-78d5649946-8r2fm\" (UID: \"89e49b52-85a2-4a39-bf50-13c848cde6f2\") " pod="opendatahub/maas-controller-78d5649946-8r2fm" Jul 06 15:32:44.766986 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:44.766826 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/89e49b52-85a2-4a39-bf50-13c848cde6f2-webhook-cert\") pod \"maas-controller-78d5649946-8r2fm\" (UID: \"89e49b52-85a2-4a39-bf50-13c848cde6f2\") " pod="opendatahub/maas-controller-78d5649946-8r2fm" Jul 06 15:32:44.798981 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:44.798952 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51132d72-1bf2-4b8b-bfda-30023f590820" path="/var/lib/kubelet/pods/51132d72-1bf2-4b8b-bfda-30023f590820/volumes" Jul 06 15:32:44.868088 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:44.868056 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twh6c\" (UniqueName: \"kubernetes.io/projected/89e49b52-85a2-4a39-bf50-13c848cde6f2-kube-api-access-twh6c\") pod \"maas-controller-78d5649946-8r2fm\" (UID: \"89e49b52-85a2-4a39-bf50-13c848cde6f2\") " pod="opendatahub/maas-controller-78d5649946-8r2fm" Jul 06 15:32:44.868276 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:44.868111 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/89e49b52-85a2-4a39-bf50-13c848cde6f2-webhook-cert\") pod \"maas-controller-78d5649946-8r2fm\" (UID: \"89e49b52-85a2-4a39-bf50-13c848cde6f2\") " pod="opendatahub/maas-controller-78d5649946-8r2fm" Jul 06 15:32:44.870762 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:44.870739 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/89e49b52-85a2-4a39-bf50-13c848cde6f2-webhook-cert\") pod \"maas-controller-78d5649946-8r2fm\" (UID: \"89e49b52-85a2-4a39-bf50-13c848cde6f2\") " pod="opendatahub/maas-controller-78d5649946-8r2fm" Jul 06 15:32:44.878823 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:44.878801 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twh6c\" (UniqueName: \"kubernetes.io/projected/89e49b52-85a2-4a39-bf50-13c848cde6f2-kube-api-access-twh6c\") pod \"maas-controller-78d5649946-8r2fm\" (UID: \"89e49b52-85a2-4a39-bf50-13c848cde6f2\") " pod="opendatahub/maas-controller-78d5649946-8r2fm" Jul 06 15:32:44.982011 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:44.981976 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78d5649946-8r2fm" Jul 06 15:32:45.128182 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:45.128045 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78d5649946-8r2fm"] Jul 06 15:32:45.130996 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:32:45.130968 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e49b52_85a2_4a39_bf50_13c848cde6f2.slice/crio-8a8fd7a07f6450e676ecbf9651ec16e6c674d25f3a4164971ed3cc7e5069bfd3 WatchSource:0}: Error finding container 8a8fd7a07f6450e676ecbf9651ec16e6c674d25f3a4164971ed3cc7e5069bfd3: Status 404 returned error can't find the container with id 8a8fd7a07f6450e676ecbf9651ec16e6c674d25f3a4164971ed3cc7e5069bfd3 Jul 06 15:32:45.795221 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:45.795185 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78d5649946-8r2fm" event={"ID":"89e49b52-85a2-4a39-bf50-13c848cde6f2","Type":"ContainerStarted","Data":"6ea76d0a6e3a4cee8da840d3dad3b46692645847f7567db4399c59b1b19203c3"} Jul 06 15:32:45.795221 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:45.795237 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78d5649946-8r2fm" event={"ID":"89e49b52-85a2-4a39-bf50-13c848cde6f2","Type":"ContainerStarted","Data":"8a8fd7a07f6450e676ecbf9651ec16e6c674d25f3a4164971ed3cc7e5069bfd3"} Jul 06 15:32:45.795449 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:45.795264 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-78d5649946-8r2fm" Jul 06 15:32:45.823304 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:45.823254 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-78d5649946-8r2fm" podStartSLOduration=1.3419327110000001 podStartE2EDuration="1.823239071s" totalImagesPullingTime="481.30636ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:32:45.820879224 +0000 UTC m=+912.434627759" watchObservedRunningTime="2026-07-06 15:32:45.823239071 +0000 UTC m=+912.436987655" Jul 06 15:32:56.806469 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:32:56.806440 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-78d5649946-8r2fm" Jul 06 15:33:25.855875 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:33:25.855842 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:25.983751 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:33:25.983720 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:26.178325 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:33:26.178232 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:35.939016 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:33:35.938983 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:38.147858 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:33:38.147826 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:43.104912 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:33:43.104875 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:43.199457 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:33:43.199423 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:48.328066 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:33:48.328029 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:36:00.071787 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:36:00.071756 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:37:34.704378 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:37:34.704345 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-mlw9w_91e0c855-5b32-4ca5-bab6-8e3aae20c1e0/console-operator/2.log" Jul 06 15:37:34.704975 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:37:34.704377 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-mlw9w_91e0c855-5b32-4ca5-bab6-8e3aae20c1e0/console-operator/2.log" Jul 06 15:37:34.708274 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:37:34.708252 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hxpw_3eca6180-d678-4729-ad6f-979a025af841/ovn-acl-logging/0.log" Jul 06 15:37:34.708461 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:37:34.708444 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hxpw_3eca6180-d678-4729-ad6f-979a025af841/ovn-acl-logging/0.log" Jul 06 15:37:48.758634 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:37:48.758592 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:37:48.846353 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:37:48.846322 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:37:54.179802 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:37:54.179768 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:40:30.088336 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:40:30.088302 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:40:32.154462 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:40:32.154421 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:42:13.409731 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:42:13.409696 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:42:34.735264 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:42:34.735235 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-mlw9w_91e0c855-5b32-4ca5-bab6-8e3aae20c1e0/console-operator/2.log" Jul 06 15:42:34.736253 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:42:34.736231 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-mlw9w_91e0c855-5b32-4ca5-bab6-8e3aae20c1e0/console-operator/2.log" Jul 06 15:42:34.739016 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:42:34.738998 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hxpw_3eca6180-d678-4729-ad6f-979a025af841/ovn-acl-logging/0.log" Jul 06 15:42:34.740089 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:42:34.740074 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hxpw_3eca6180-d678-4729-ad6f-979a025af841/ovn-acl-logging/0.log" Jul 06 15:43:57.760856 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:43:57.760815 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:17.394058 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:17.394025 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:33.660688 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:33.660599 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:33.839900 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:33.839871 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:34.202576 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:34.202548 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:40.174177 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.174142 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:40.362450 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.362420 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:40.454858 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.454765 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z"] Jul 06 15:44:40.463952 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.463924 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:40.466982 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.466959 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f1d34632-openshift-default-dockercfg-sbtj9\"" Jul 06 15:44:40.485974 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.485939 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z"] Jul 06 15:44:40.503087 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.503060 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:40.603922 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.603891 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:40.637205 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.637172 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a608b15-6e09-457f-8ff0-078a50343773-credential-socket\") pod \"e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:40.637339 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.637213 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a608b15-6e09-457f-8ff0-078a50343773-workload-socket\") pod \"e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:40.637339 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.637306 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fzmg\" (UniqueName: \"kubernetes.io/projected/1a608b15-6e09-457f-8ff0-078a50343773-kube-api-access-4fzmg\") pod \"e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:40.637414 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.637349 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a608b15-6e09-457f-8ff0-078a50343773-workload-certs\") pod \"e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:40.637414 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.637399 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a608b15-6e09-457f-8ff0-078a50343773-istio-podinfo\") pod \"e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:40.637485 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.637415 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a608b15-6e09-457f-8ff0-078a50343773-istio-token\") pod \"e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:40.637485 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.637442 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a608b15-6e09-457f-8ff0-078a50343773-istio-data\") pod \"e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:40.637631 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.637590 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a608b15-6e09-457f-8ff0-078a50343773-istio-envoy\") pod \"e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:40.637686 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.637626 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a608b15-6e09-457f-8ff0-078a50343773-istiod-ca-cert\") pod \"e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:40.738632 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.738543 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a608b15-6e09-457f-8ff0-078a50343773-istio-envoy\") pod \"e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:40.738632 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.738579 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a608b15-6e09-457f-8ff0-078a50343773-istiod-ca-cert\") pod \"e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:40.738632 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.738614 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a608b15-6e09-457f-8ff0-078a50343773-credential-socket\") pod \"e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:40.738887 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.738648 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a608b15-6e09-457f-8ff0-078a50343773-workload-socket\") pod \"e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:40.738887 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.738702 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fzmg\" (UniqueName: \"kubernetes.io/projected/1a608b15-6e09-457f-8ff0-078a50343773-kube-api-access-4fzmg\") pod \"e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:40.738887 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.738738 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a608b15-6e09-457f-8ff0-078a50343773-workload-certs\") pod \"e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:40.739033 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.738902 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a608b15-6e09-457f-8ff0-078a50343773-istio-podinfo\") pod \"e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:40.739033 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.738969 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a608b15-6e09-457f-8ff0-078a50343773-istio-token\") pod \"e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:40.739033 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.739027 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a608b15-6e09-457f-8ff0-078a50343773-istio-data\") pod \"e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:40.739174 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.739088 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a608b15-6e09-457f-8ff0-078a50343773-credential-socket\") pod \"e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:40.739277 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.739249 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a608b15-6e09-457f-8ff0-078a50343773-workload-socket\") pod \"e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:40.739357 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.739331 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a608b15-6e09-457f-8ff0-078a50343773-workload-certs\") pod \"e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:40.739430 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.739411 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a608b15-6e09-457f-8ff0-078a50343773-istio-data\") pod \"e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:40.739478 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.739442 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a608b15-6e09-457f-8ff0-078a50343773-istiod-ca-cert\") pod \"e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:40.740900 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.740881 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a608b15-6e09-457f-8ff0-078a50343773-istio-envoy\") pod \"e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:40.741398 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.741381 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a608b15-6e09-457f-8ff0-078a50343773-istio-podinfo\") pod \"e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:40.748075 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.748054 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a608b15-6e09-457f-8ff0-078a50343773-istio-token\") pod \"e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:40.748417 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.748397 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fzmg\" (UniqueName: \"kubernetes.io/projected/1a608b15-6e09-457f-8ff0-078a50343773-kube-api-access-4fzmg\") pod \"e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:40.775002 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.774971 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:40.849858 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.849756 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:40.924681 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.924654 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z"] Jul 06 15:44:40.926475 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:44:40.926450 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a608b15_6e09_457f_8ff0_078a50343773.slice/crio-a53c33b21504211f50018efb7d49ce5ebfe7b9dd39f4bd634e9c334d52b1a766 WatchSource:0}: Error finding container a53c33b21504211f50018efb7d49ce5ebfe7b9dd39f4bd634e9c334d52b1a766: Status 404 returned error can't find the container with id a53c33b21504211f50018efb7d49ce5ebfe7b9dd39f4bd634e9c334d52b1a766 Jul 06 15:44:40.928564 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.928532 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:44:40.928684 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.928607 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:44:40.928684 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.928649 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:44:40.971804 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:40.971777 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:41.151068 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:41.151034 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" event={"ID":"1a608b15-6e09-457f-8ff0-078a50343773","Type":"ContainerStarted","Data":"33018b6c5567f837ad3b8871c0816f147fa7bcd46a0a0ab23fc4f0cd1fe01d72"} Jul 06 15:44:41.151238 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:41.151081 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" event={"ID":"1a608b15-6e09-457f-8ff0-078a50343773","Type":"ContainerStarted","Data":"a53c33b21504211f50018efb7d49ce5ebfe7b9dd39f4bd634e9c334d52b1a766"} Jul 06 15:44:41.671060 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:41.671030 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:41.775690 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:41.775653 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:41.780654 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:41.780632 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:41.814076 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:41.814039 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:41.834813 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:41.834764 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" podStartSLOduration=1.8347473939999999 podStartE2EDuration="1.834747394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:44:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:44:41.183393953 +0000 UTC m=+1627.797142488" watchObservedRunningTime="2026-07-06 15:44:41.834747394 +0000 UTC m=+1628.448495932" Jul 06 15:44:42.045289 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:42.045203 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-f1d34632-6686c566b-ks6wh"] Jul 06 15:44:42.048915 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:42.048896 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-f1d34632-6686c566b-ks6wh" Jul 06 15:44:42.053159 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:42.053138 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-f1d34632-serving-cert\"" Jul 06 15:44:42.082145 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:42.082115 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-f1d34632-6686c566b-ks6wh"] Jul 06 15:44:42.153095 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:42.153068 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr5gb\" (UniqueName: \"kubernetes.io/projected/77474074-4335-4696-b217-46f5f7f2494a-kube-api-access-sr5gb\") pod \"maas-api-e2e-ait-f1d34632-6686c566b-ks6wh\" (UID: \"77474074-4335-4696-b217-46f5f7f2494a\") " pod="opendatahub/maas-api-e2e-ait-f1d34632-6686c566b-ks6wh" Jul 06 15:44:42.153262 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:42.153121 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77474074-4335-4696-b217-46f5f7f2494a-openshift-service-ca\") pod \"maas-api-e2e-ait-f1d34632-6686c566b-ks6wh\" (UID: \"77474074-4335-4696-b217-46f5f7f2494a\") " pod="opendatahub/maas-api-e2e-ait-f1d34632-6686c566b-ks6wh" Jul 06 15:44:42.153262 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:42.153196 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77474074-4335-4696-b217-46f5f7f2494a-maas-api-tls\") pod \"maas-api-e2e-ait-f1d34632-6686c566b-ks6wh\" (UID: \"77474074-4335-4696-b217-46f5f7f2494a\") " pod="opendatahub/maas-api-e2e-ait-f1d34632-6686c566b-ks6wh" Jul 06 15:44:42.155866 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:42.155843 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:42.156782 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:42.156764 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:42.157705 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:42.157679 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:42.253846 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:42.253799 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr5gb\" (UniqueName: \"kubernetes.io/projected/77474074-4335-4696-b217-46f5f7f2494a-kube-api-access-sr5gb\") pod \"maas-api-e2e-ait-f1d34632-6686c566b-ks6wh\" (UID: \"77474074-4335-4696-b217-46f5f7f2494a\") " pod="opendatahub/maas-api-e2e-ait-f1d34632-6686c566b-ks6wh" Jul 06 15:44:42.254039 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:42.253897 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77474074-4335-4696-b217-46f5f7f2494a-openshift-service-ca\") pod \"maas-api-e2e-ait-f1d34632-6686c566b-ks6wh\" (UID: \"77474074-4335-4696-b217-46f5f7f2494a\") " pod="opendatahub/maas-api-e2e-ait-f1d34632-6686c566b-ks6wh" Jul 06 15:44:42.254039 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:42.253972 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77474074-4335-4696-b217-46f5f7f2494a-maas-api-tls\") pod \"maas-api-e2e-ait-f1d34632-6686c566b-ks6wh\" (UID: \"77474074-4335-4696-b217-46f5f7f2494a\") " pod="opendatahub/maas-api-e2e-ait-f1d34632-6686c566b-ks6wh" Jul 06 15:44:42.254180 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:44:42.254119 2570 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-f1d34632-serving-cert: secret "maas-api-e2e-ait-f1d34632-serving-cert" not found Jul 06 15:44:42.254242 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:44:42.254200 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77474074-4335-4696-b217-46f5f7f2494a-maas-api-tls podName:77474074-4335-4696-b217-46f5f7f2494a nodeName:}" failed. No retries permitted until 2026-07-06 15:44:42.754179072 +0000 UTC m=+1629.367927604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/77474074-4335-4696-b217-46f5f7f2494a-maas-api-tls") pod "maas-api-e2e-ait-f1d34632-6686c566b-ks6wh" (UID: "77474074-4335-4696-b217-46f5f7f2494a") : secret "maas-api-e2e-ait-f1d34632-serving-cert" not found Jul 06 15:44:42.254853 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:42.254824 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77474074-4335-4696-b217-46f5f7f2494a-openshift-service-ca\") pod \"maas-api-e2e-ait-f1d34632-6686c566b-ks6wh\" (UID: \"77474074-4335-4696-b217-46f5f7f2494a\") " pod="opendatahub/maas-api-e2e-ait-f1d34632-6686c566b-ks6wh" Jul 06 15:44:42.280044 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:42.280015 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr5gb\" (UniqueName: \"kubernetes.io/projected/77474074-4335-4696-b217-46f5f7f2494a-kube-api-access-sr5gb\") pod \"maas-api-e2e-ait-f1d34632-6686c566b-ks6wh\" (UID: \"77474074-4335-4696-b217-46f5f7f2494a\") " pod="opendatahub/maas-api-e2e-ait-f1d34632-6686c566b-ks6wh" Jul 06 15:44:42.418890 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:42.418863 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:42.759029 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:42.758937 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77474074-4335-4696-b217-46f5f7f2494a-maas-api-tls\") pod \"maas-api-e2e-ait-f1d34632-6686c566b-ks6wh\" (UID: \"77474074-4335-4696-b217-46f5f7f2494a\") " pod="opendatahub/maas-api-e2e-ait-f1d34632-6686c566b-ks6wh" Jul 06 15:44:42.761430 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:42.761409 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77474074-4335-4696-b217-46f5f7f2494a-maas-api-tls\") pod \"maas-api-e2e-ait-f1d34632-6686c566b-ks6wh\" (UID: \"77474074-4335-4696-b217-46f5f7f2494a\") " pod="opendatahub/maas-api-e2e-ait-f1d34632-6686c566b-ks6wh" Jul 06 15:44:42.769698 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:42.769674 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:42.869750 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:42.869721 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:42.959336 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:42.959305 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-f1d34632-6686c566b-ks6wh" Jul 06 15:44:43.107641 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:43.107610 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-f1d34632-6686c566b-ks6wh"] Jul 06 15:44:43.111117 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:44:43.111086 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77474074_4335_4696_b217_46f5f7f2494a.slice/crio-be71e0d948ef8e71a578276af970b314148412d19961b414f94a39645694652d WatchSource:0}: Error finding container be71e0d948ef8e71a578276af970b314148412d19961b414f94a39645694652d: Status 404 returned error can't find the container with id be71e0d948ef8e71a578276af970b314148412d19961b414f94a39645694652d Jul 06 15:44:43.112625 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:43.112606 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 15:44:43.161463 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:43.161431 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-f1d34632-6686c566b-ks6wh" event={"ID":"77474074-4335-4696-b217-46f5f7f2494a","Type":"ContainerStarted","Data":"be71e0d948ef8e71a578276af970b314148412d19961b414f94a39645694652d"} Jul 06 15:44:44.167238 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:44.167201 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-f1d34632-6686c566b-ks6wh" event={"ID":"77474074-4335-4696-b217-46f5f7f2494a","Type":"ContainerStarted","Data":"140e527195164d5504549f76c4032fe646e3650a8663c2eb74bfef43bfe04d53"} Jul 06 15:44:44.167648 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:44.167476 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-f1d34632-6686c566b-ks6wh" Jul 06 15:44:44.187466 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:44.187343 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-f1d34632-6686c566b-ks6wh" podStartSLOduration=2.7210338160000003 podStartE2EDuration="3.187323002s" totalImagesPullingTime="466.289186ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:44:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:44:44.187076972 +0000 UTC m=+1630.800825508" watchObservedRunningTime="2026-07-06 15:44:44.187323002 +0000 UTC m=+1630.801071539" Jul 06 15:44:44.262527 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:44.262461 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:46.844983 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:46.844942 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:46.966054 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:46.966016 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z"] Jul 06 15:44:46.966295 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:46.966250 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" podUID="1a608b15-6e09-457f-8ff0-078a50343773" containerName="istio-proxy" containerID="cri-o://33018b6c5567f837ad3b8871c0816f147fa7bcd46a0a0ab23fc4f0cd1fe01d72" gracePeriod=30 Jul 06 15:44:47.040514 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:47.040441 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:47.238588 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:47.238456 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:47.393666 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:47.393632 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-f1d34632-6686c566b-ks6wh"] Jul 06 15:44:47.393831 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:47.393807 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-f1d34632-6686c566b-ks6wh" podUID="77474074-4335-4696-b217-46f5f7f2494a" containerName="maas-api" containerID="cri-o://140e527195164d5504549f76c4032fe646e3650a8663c2eb74bfef43bfe04d53" gracePeriod=30 Jul 06 15:44:47.464464 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:47.464434 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:47.641720 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:47.641699 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-f1d34632-6686c566b-ks6wh" Jul 06 15:44:47.706913 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:47.706879 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sr5gb\" (UniqueName: \"kubernetes.io/projected/77474074-4335-4696-b217-46f5f7f2494a-kube-api-access-sr5gb\") pod \"77474074-4335-4696-b217-46f5f7f2494a\" (UID: \"77474074-4335-4696-b217-46f5f7f2494a\") " Jul 06 15:44:47.707069 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:47.706939 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77474074-4335-4696-b217-46f5f7f2494a-maas-api-tls\") pod \"77474074-4335-4696-b217-46f5f7f2494a\" (UID: \"77474074-4335-4696-b217-46f5f7f2494a\") " Jul 06 15:44:47.707069 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:47.707040 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77474074-4335-4696-b217-46f5f7f2494a-openshift-service-ca\") pod \"77474074-4335-4696-b217-46f5f7f2494a\" (UID: \"77474074-4335-4696-b217-46f5f7f2494a\") " Jul 06 15:44:47.707417 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:47.707383 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77474074-4335-4696-b217-46f5f7f2494a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "77474074-4335-4696-b217-46f5f7f2494a" (UID: "77474074-4335-4696-b217-46f5f7f2494a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:44:47.708924 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:47.708898 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77474074-4335-4696-b217-46f5f7f2494a-kube-api-access-sr5gb" (OuterVolumeSpecName: "kube-api-access-sr5gb") pod "77474074-4335-4696-b217-46f5f7f2494a" (UID: "77474074-4335-4696-b217-46f5f7f2494a"). InnerVolumeSpecName "kube-api-access-sr5gb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:44:47.709042 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:47.709018 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77474074-4335-4696-b217-46f5f7f2494a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "77474074-4335-4696-b217-46f5f7f2494a" (UID: "77474074-4335-4696-b217-46f5f7f2494a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:44:47.808388 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:47.808288 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77474074-4335-4696-b217-46f5f7f2494a-openshift-service-ca\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:44:47.808388 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:47.808331 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr5gb\" (UniqueName: \"kubernetes.io/projected/77474074-4335-4696-b217-46f5f7f2494a-kube-api-access-sr5gb\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:44:47.808388 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:47.808348 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77474074-4335-4696-b217-46f5f7f2494a-maas-api-tls\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:44:48.188901 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:48.188869 2570 generic.go:356] "Generic (PLEG): container finished" podID="77474074-4335-4696-b217-46f5f7f2494a" containerID="140e527195164d5504549f76c4032fe646e3650a8663c2eb74bfef43bfe04d53" exitCode=0 Jul 06 15:44:48.189348 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:48.188912 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-f1d34632-6686c566b-ks6wh" Jul 06 15:44:48.189348 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:48.188955 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-f1d34632-6686c566b-ks6wh" event={"ID":"77474074-4335-4696-b217-46f5f7f2494a","Type":"ContainerDied","Data":"140e527195164d5504549f76c4032fe646e3650a8663c2eb74bfef43bfe04d53"} Jul 06 15:44:48.189348 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:48.189002 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-f1d34632-6686c566b-ks6wh" event={"ID":"77474074-4335-4696-b217-46f5f7f2494a","Type":"ContainerDied","Data":"be71e0d948ef8e71a578276af970b314148412d19961b414f94a39645694652d"} Jul 06 15:44:48.189348 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:48.189026 2570 scope.go:117] "RemoveContainer" containerID="140e527195164d5504549f76c4032fe646e3650a8663c2eb74bfef43bfe04d53" Jul 06 15:44:48.198494 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:48.198479 2570 scope.go:117] "RemoveContainer" containerID="140e527195164d5504549f76c4032fe646e3650a8663c2eb74bfef43bfe04d53" Jul 06 15:44:48.198751 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:44:48.198730 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"140e527195164d5504549f76c4032fe646e3650a8663c2eb74bfef43bfe04d53\": container with ID starting with 140e527195164d5504549f76c4032fe646e3650a8663c2eb74bfef43bfe04d53 not found: ID does not exist" containerID="140e527195164d5504549f76c4032fe646e3650a8663c2eb74bfef43bfe04d53" Jul 06 15:44:48.198816 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:48.198760 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"140e527195164d5504549f76c4032fe646e3650a8663c2eb74bfef43bfe04d53"} err="failed to get container status \"140e527195164d5504549f76c4032fe646e3650a8663c2eb74bfef43bfe04d53\": rpc error: code = NotFound desc = could not find container \"140e527195164d5504549f76c4032fe646e3650a8663c2eb74bfef43bfe04d53\": container with ID starting with 140e527195164d5504549f76c4032fe646e3650a8663c2eb74bfef43bfe04d53 not found: ID does not exist" Jul 06 15:44:48.219590 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:48.219568 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-f1d34632-6686c566b-ks6wh"] Jul 06 15:44:48.223160 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:48.223140 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-f1d34632-6686c566b-ks6wh"] Jul 06 15:44:48.801377 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:48.801346 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77474074-4335-4696-b217-46f5f7f2494a" path="/var/lib/kubelet/pods/77474074-4335-4696-b217-46f5f7f2494a/volumes" Jul 06 15:44:50.284628 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:50.284595 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:52.212956 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:52.212929 2570 generic.go:356] "Generic (PLEG): container finished" podID="1a608b15-6e09-457f-8ff0-078a50343773" containerID="33018b6c5567f837ad3b8871c0816f147fa7bcd46a0a0ab23fc4f0cd1fe01d72" exitCode=0 Jul 06 15:44:52.213276 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:52.213002 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" event={"ID":"1a608b15-6e09-457f-8ff0-078a50343773","Type":"ContainerDied","Data":"33018b6c5567f837ad3b8871c0816f147fa7bcd46a0a0ab23fc4f0cd1fe01d72"} Jul 06 15:44:52.213276 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:52.213054 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" event={"ID":"1a608b15-6e09-457f-8ff0-078a50343773","Type":"ContainerDied","Data":"a53c33b21504211f50018efb7d49ce5ebfe7b9dd39f4bd634e9c334d52b1a766"} Jul 06 15:44:52.213276 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:52.213072 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a53c33b21504211f50018efb7d49ce5ebfe7b9dd39f4bd634e9c334d52b1a766" Jul 06 15:44:52.227012 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:52.226959 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:52.252689 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:52.252660 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fzmg\" (UniqueName: \"kubernetes.io/projected/1a608b15-6e09-457f-8ff0-078a50343773-kube-api-access-4fzmg\") pod \"1a608b15-6e09-457f-8ff0-078a50343773\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " Jul 06 15:44:52.252849 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:52.252697 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a608b15-6e09-457f-8ff0-078a50343773-istio-envoy\") pod \"1a608b15-6e09-457f-8ff0-078a50343773\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " Jul 06 15:44:52.252849 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:52.252725 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a608b15-6e09-457f-8ff0-078a50343773-workload-socket\") pod \"1a608b15-6e09-457f-8ff0-078a50343773\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " Jul 06 15:44:52.252849 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:52.252756 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a608b15-6e09-457f-8ff0-078a50343773-istio-data\") pod \"1a608b15-6e09-457f-8ff0-078a50343773\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " Jul 06 15:44:52.252849 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:52.252786 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a608b15-6e09-457f-8ff0-078a50343773-credential-socket\") pod \"1a608b15-6e09-457f-8ff0-078a50343773\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " Jul 06 15:44:52.252849 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:52.252816 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a608b15-6e09-457f-8ff0-078a50343773-istiod-ca-cert\") pod \"1a608b15-6e09-457f-8ff0-078a50343773\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " Jul 06 15:44:52.253098 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:52.252880 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a608b15-6e09-457f-8ff0-078a50343773-workload-certs\") pod \"1a608b15-6e09-457f-8ff0-078a50343773\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " Jul 06 15:44:52.253098 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:52.253051 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a608b15-6e09-457f-8ff0-078a50343773-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1a608b15-6e09-457f-8ff0-078a50343773" (UID: "1a608b15-6e09-457f-8ff0-078a50343773"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:44:52.253204 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:52.253130 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a608b15-6e09-457f-8ff0-078a50343773-istio-data" (OuterVolumeSpecName: "istio-data") pod "1a608b15-6e09-457f-8ff0-078a50343773" (UID: "1a608b15-6e09-457f-8ff0-078a50343773"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:44:52.253204 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:52.253143 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a608b15-6e09-457f-8ff0-078a50343773-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1a608b15-6e09-457f-8ff0-078a50343773" (UID: "1a608b15-6e09-457f-8ff0-078a50343773"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:44:52.253204 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:52.253152 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a608b15-6e09-457f-8ff0-078a50343773-istio-podinfo\") pod \"1a608b15-6e09-457f-8ff0-078a50343773\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " Jul 06 15:44:52.253204 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:52.253204 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a608b15-6e09-457f-8ff0-078a50343773-istio-token\") pod \"1a608b15-6e09-457f-8ff0-078a50343773\" (UID: \"1a608b15-6e09-457f-8ff0-078a50343773\") " Jul 06 15:44:52.253414 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:52.253205 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a608b15-6e09-457f-8ff0-078a50343773-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1a608b15-6e09-457f-8ff0-078a50343773" (UID: "1a608b15-6e09-457f-8ff0-078a50343773"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:44:52.253614 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:52.253585 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a608b15-6e09-457f-8ff0-078a50343773-workload-socket\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:44:52.253614 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:52.253615 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a608b15-6e09-457f-8ff0-078a50343773-istio-data\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:44:52.253814 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:52.253630 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a608b15-6e09-457f-8ff0-078a50343773-istiod-ca-cert\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:44:52.253814 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:52.253644 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a608b15-6e09-457f-8ff0-078a50343773-workload-certs\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:44:52.253814 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:52.253647 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a608b15-6e09-457f-8ff0-078a50343773-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1a608b15-6e09-457f-8ff0-078a50343773" (UID: "1a608b15-6e09-457f-8ff0-078a50343773"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:44:52.255144 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:52.255114 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a608b15-6e09-457f-8ff0-078a50343773-istio-token" (OuterVolumeSpecName: "istio-token") pod "1a608b15-6e09-457f-8ff0-078a50343773" (UID: "1a608b15-6e09-457f-8ff0-078a50343773"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:44:52.255257 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:52.255160 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a608b15-6e09-457f-8ff0-078a50343773-kube-api-access-4fzmg" (OuterVolumeSpecName: "kube-api-access-4fzmg") pod "1a608b15-6e09-457f-8ff0-078a50343773" (UID: "1a608b15-6e09-457f-8ff0-078a50343773"). InnerVolumeSpecName "kube-api-access-4fzmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:44:52.255257 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:52.255169 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a608b15-6e09-457f-8ff0-078a50343773-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1a608b15-6e09-457f-8ff0-078a50343773" (UID: "1a608b15-6e09-457f-8ff0-078a50343773"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:44:52.255257 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:52.255199 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a608b15-6e09-457f-8ff0-078a50343773-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1a608b15-6e09-457f-8ff0-078a50343773" (UID: "1a608b15-6e09-457f-8ff0-078a50343773"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 15:44:52.355106 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:52.355073 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a608b15-6e09-457f-8ff0-078a50343773-istio-envoy\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:44:52.355106 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:52.355105 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a608b15-6e09-457f-8ff0-078a50343773-credential-socket\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:44:52.355345 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:52.355120 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a608b15-6e09-457f-8ff0-078a50343773-istio-podinfo\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:44:52.355345 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:52.355132 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a608b15-6e09-457f-8ff0-078a50343773-istio-token\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:44:52.355345 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:52.355144 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fzmg\" (UniqueName: \"kubernetes.io/projected/1a608b15-6e09-457f-8ff0-078a50343773-kube-api-access-4fzmg\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:44:53.217674 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.217642 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z" Jul 06 15:44:53.239833 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.239798 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z"] Jul 06 15:44:53.244294 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.244267 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f1d34632-openshift-default-69f884678f-7fh9z"] Jul 06 15:44:53.590113 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.590078 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:53.713003 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.712971 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:53.819901 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.819858 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2"] Jul 06 15:44:53.820486 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.820466 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77474074-4335-4696-b217-46f5f7f2494a" containerName="maas-api" Jul 06 15:44:53.820486 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.820487 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="77474074-4335-4696-b217-46f5f7f2494a" containerName="maas-api" Jul 06 15:44:53.820683 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.820521 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a608b15-6e09-457f-8ff0-078a50343773" containerName="istio-proxy" Jul 06 15:44:53.820683 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.820531 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a608b15-6e09-457f-8ff0-078a50343773" containerName="istio-proxy" Jul 06 15:44:53.820683 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.820615 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a608b15-6e09-457f-8ff0-078a50343773" containerName="istio-proxy" Jul 06 15:44:53.820683 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.820625 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="77474074-4335-4696-b217-46f5f7f2494a" containerName="maas-api" Jul 06 15:44:53.824314 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.824289 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:44:53.829361 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.829338 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-438fd2f6-openshift-default-dockercfg-vr89m\"" Jul 06 15:44:53.848763 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.848696 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2"] Jul 06 15:44:53.869492 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.869471 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-istio-envoy\") pod \"e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:44:53.869622 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.869514 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-istiod-ca-cert\") pod \"e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:44:53.869622 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.869541 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-workload-socket\") pod \"e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:44:53.869622 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.869601 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-istio-podinfo\") pod \"e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:44:53.869724 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.869645 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-workload-certs\") pod \"e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:44:53.869724 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.869664 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-istio-data\") pod \"e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:44:53.869788 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.869753 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw5sf\" (UniqueName: \"kubernetes.io/projected/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-kube-api-access-mw5sf\") pod \"e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:44:53.869876 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.869857 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-credential-socket\") pod \"e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:44:53.869911 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.869890 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-istio-token\") pod \"e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:44:53.957489 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.957458 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:53.970624 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.970598 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-workload-certs\") pod \"e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:44:53.970772 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.970629 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-istio-data\") pod \"e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:44:53.970772 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.970670 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw5sf\" (UniqueName: \"kubernetes.io/projected/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-kube-api-access-mw5sf\") pod \"e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:44:53.970772 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.970718 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-credential-socket\") pod \"e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:44:53.970772 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.970735 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-istio-token\") pod \"e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:44:53.970772 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.970760 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-istio-envoy\") pod \"e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:44:53.971010 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.970836 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-istiod-ca-cert\") pod \"e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:44:53.971010 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.970870 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-workload-socket\") pod \"e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:44:53.971010 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.970902 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-istio-podinfo\") pod \"e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:44:53.971160 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.971019 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-workload-certs\") pod \"e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:44:53.971160 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.971051 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-istio-data\") pod \"e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:44:53.971264 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.971157 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-credential-socket\") pod \"e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:44:53.971264 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.971234 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-workload-socket\") pod \"e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:44:53.971489 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.971470 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-istiod-ca-cert\") pod \"e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:44:53.972989 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.972972 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-istio-envoy\") pod \"e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:44:53.973208 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.973190 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-istio-podinfo\") pod \"e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:44:53.982903 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.982882 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-istio-token\") pod \"e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:44:53.995051 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:53.995025 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw5sf\" (UniqueName: \"kubernetes.io/projected/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-kube-api-access-mw5sf\") pod \"e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:44:54.087906 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:54.087858 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:54.138943 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:54.138859 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:44:54.303393 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:54.303366 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2"] Jul 06 15:44:54.304627 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:44:54.304603 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd2585da_3b1d_4eb5_8c39_d6b8c9a30bcb.slice/crio-6dbe7def0abb5f13d4b47afe5c884758fb625d6671cc12b9401618e2392f1fe6 WatchSource:0}: Error finding container 6dbe7def0abb5f13d4b47afe5c884758fb625d6671cc12b9401618e2392f1fe6: Status 404 returned error can't find the container with id 6dbe7def0abb5f13d4b47afe5c884758fb625d6671cc12b9401618e2392f1fe6 Jul 06 15:44:54.306841 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:54.306806 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:44:54.306926 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:54.306872 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:44:54.306926 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:54.306902 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:44:54.799325 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:54.799286 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a608b15-6e09-457f-8ff0-078a50343773" path="/var/lib/kubelet/pods/1a608b15-6e09-457f-8ff0-078a50343773/volumes" Jul 06 15:44:55.134690 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:55.134658 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:55.228053 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:55.228019 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" event={"ID":"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb","Type":"ContainerStarted","Data":"e347a9fa440270ca685e54fde17b567956dc9283fdfde5509746e80db544aaa0"} Jul 06 15:44:55.228262 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:55.228073 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" event={"ID":"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb","Type":"ContainerStarted","Data":"6dbe7def0abb5f13d4b47afe5c884758fb625d6671cc12b9401618e2392f1fe6"} Jul 06 15:44:55.256561 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:55.256531 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx"] Jul 06 15:44:55.260242 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:55.260219 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx" Jul 06 15:44:55.264682 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:55.264653 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-438fd2f6-serving-cert\"" Jul 06 15:44:55.283610 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:55.283580 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fph6x\" (UniqueName: \"kubernetes.io/projected/4b371265-bcb8-4ae1-a89a-2eb4b551b141-kube-api-access-fph6x\") pod \"maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx\" (UID: \"4b371265-bcb8-4ae1-a89a-2eb4b551b141\") " pod="opendatahub/maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx" Jul 06 15:44:55.283764 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:55.283647 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b371265-bcb8-4ae1-a89a-2eb4b551b141-maas-api-tls\") pod \"maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx\" (UID: \"4b371265-bcb8-4ae1-a89a-2eb4b551b141\") " pod="opendatahub/maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx" Jul 06 15:44:55.283877 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:55.283857 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b371265-bcb8-4ae1-a89a-2eb4b551b141-openshift-service-ca\") pod \"maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx\" (UID: \"4b371265-bcb8-4ae1-a89a-2eb4b551b141\") " pod="opendatahub/maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx" Jul 06 15:44:55.287408 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:55.287379 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx"] Jul 06 15:44:55.294520 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:55.294462 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" podStartSLOduration=2.2944487159999998 podStartE2EDuration="2.294448716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:44:55.294125776 +0000 UTC m=+1641.907874311" watchObservedRunningTime="2026-07-06 15:44:55.294448716 +0000 UTC m=+1641.908197252" Jul 06 15:44:55.319554 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:55.319473 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:55.385007 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:55.384923 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b371265-bcb8-4ae1-a89a-2eb4b551b141-openshift-service-ca\") pod \"maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx\" (UID: \"4b371265-bcb8-4ae1-a89a-2eb4b551b141\") " pod="opendatahub/maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx" Jul 06 15:44:55.385007 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:55.384975 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fph6x\" (UniqueName: \"kubernetes.io/projected/4b371265-bcb8-4ae1-a89a-2eb4b551b141-kube-api-access-fph6x\") pod \"maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx\" (UID: \"4b371265-bcb8-4ae1-a89a-2eb4b551b141\") " pod="opendatahub/maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx" Jul 06 15:44:55.385202 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:55.385023 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b371265-bcb8-4ae1-a89a-2eb4b551b141-maas-api-tls\") pod \"maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx\" (UID: \"4b371265-bcb8-4ae1-a89a-2eb4b551b141\") " pod="opendatahub/maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx" Jul 06 15:44:55.385202 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:44:55.385184 2570 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-438fd2f6-serving-cert: secret "maas-api-e2e-ait-438fd2f6-serving-cert" not found Jul 06 15:44:55.385270 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:44:55.385247 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b371265-bcb8-4ae1-a89a-2eb4b551b141-maas-api-tls podName:4b371265-bcb8-4ae1-a89a-2eb4b551b141 nodeName:}" failed. No retries permitted until 2026-07-06 15:44:55.885226079 +0000 UTC m=+1642.498974594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4b371265-bcb8-4ae1-a89a-2eb4b551b141-maas-api-tls") pod "maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx" (UID: "4b371265-bcb8-4ae1-a89a-2eb4b551b141") : secret "maas-api-e2e-ait-438fd2f6-serving-cert" not found Jul 06 15:44:55.385679 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:55.385660 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b371265-bcb8-4ae1-a89a-2eb4b551b141-openshift-service-ca\") pod \"maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx\" (UID: \"4b371265-bcb8-4ae1-a89a-2eb4b551b141\") " pod="opendatahub/maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx" Jul 06 15:44:55.432054 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:55.432023 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fph6x\" (UniqueName: \"kubernetes.io/projected/4b371265-bcb8-4ae1-a89a-2eb4b551b141-kube-api-access-fph6x\") pod \"maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx\" (UID: \"4b371265-bcb8-4ae1-a89a-2eb4b551b141\") " pod="opendatahub/maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx" Jul 06 15:44:55.520935 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:55.520903 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:55.664315 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:55.664236 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:55.890425 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:55.890390 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b371265-bcb8-4ae1-a89a-2eb4b551b141-maas-api-tls\") pod \"maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx\" (UID: \"4b371265-bcb8-4ae1-a89a-2eb4b551b141\") " pod="opendatahub/maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx" Jul 06 15:44:55.892771 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:55.892744 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b371265-bcb8-4ae1-a89a-2eb4b551b141-maas-api-tls\") pod \"maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx\" (UID: \"4b371265-bcb8-4ae1-a89a-2eb4b551b141\") " pod="opendatahub/maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx" Jul 06 15:44:56.139762 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:56.139725 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:44:56.144749 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:56.144727 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:44:56.170613 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:56.170582 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx" Jul 06 15:44:56.233396 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:56.233370 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:44:56.234465 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:56.234444 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:44:56.476919 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:44:56.476887 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b371265_bcb8_4ae1_a89a_2eb4b551b141.slice/crio-0596b97d9b6e18ad1cf24dc22384680173823f3214e1290f9e31ce418aff942c WatchSource:0}: Error finding container 0596b97d9b6e18ad1cf24dc22384680173823f3214e1290f9e31ce418aff942c: Status 404 returned error can't find the container with id 0596b97d9b6e18ad1cf24dc22384680173823f3214e1290f9e31ce418aff942c Jul 06 15:44:56.477715 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:56.477280 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx"] Jul 06 15:44:57.238962 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:57.238918 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx" event={"ID":"4b371265-bcb8-4ae1-a89a-2eb4b551b141","Type":"ContainerStarted","Data":"03863d3df3b1fd8551e386c85254fc4503fd2d44c2d33f9f873e9b0cb0639fa9"} Jul 06 15:44:57.239233 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:57.239220 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx" Jul 06 15:44:57.239291 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:57.239248 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx" event={"ID":"4b371265-bcb8-4ae1-a89a-2eb4b551b141","Type":"ContainerStarted","Data":"0596b97d9b6e18ad1cf24dc22384680173823f3214e1290f9e31ce418aff942c"} Jul 06 15:44:57.336619 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:57.336563 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx" podStartSLOduration=1.995482531 podStartE2EDuration="2.336547435s" totalImagesPullingTime="341.064904ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:44:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:44:57.33343352 +0000 UTC m=+1643.947182056" watchObservedRunningTime="2026-07-06 15:44:57.336547435 +0000 UTC m=+1643.950295969" Jul 06 15:44:57.640741 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:57.640672 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:58.005465 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:44:58.005382 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:00.472771 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:00.472731 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:01.266632 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:01.266600 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx"] Jul 06 15:45:01.266842 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:01.266792 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx" podUID="4b371265-bcb8-4ae1-a89a-2eb4b551b141" containerName="maas-api" containerID="cri-o://03863d3df3b1fd8551e386c85254fc4503fd2d44c2d33f9f873e9b0cb0639fa9" gracePeriod=30 Jul 06 15:45:01.271837 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:01.271817 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx" Jul 06 15:45:01.369143 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:01.369100 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:01.522740 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:01.522682 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx" Jul 06 15:45:01.647911 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:01.647878 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fph6x\" (UniqueName: \"kubernetes.io/projected/4b371265-bcb8-4ae1-a89a-2eb4b551b141-kube-api-access-fph6x\") pod \"4b371265-bcb8-4ae1-a89a-2eb4b551b141\" (UID: \"4b371265-bcb8-4ae1-a89a-2eb4b551b141\") " Jul 06 15:45:01.648088 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:01.647921 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b371265-bcb8-4ae1-a89a-2eb4b551b141-openshift-service-ca\") pod \"4b371265-bcb8-4ae1-a89a-2eb4b551b141\" (UID: \"4b371265-bcb8-4ae1-a89a-2eb4b551b141\") " Jul 06 15:45:01.648088 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:01.647941 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b371265-bcb8-4ae1-a89a-2eb4b551b141-maas-api-tls\") pod \"4b371265-bcb8-4ae1-a89a-2eb4b551b141\" (UID: \"4b371265-bcb8-4ae1-a89a-2eb4b551b141\") " Jul 06 15:45:01.648253 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:01.648226 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b371265-bcb8-4ae1-a89a-2eb4b551b141-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4b371265-bcb8-4ae1-a89a-2eb4b551b141" (UID: "4b371265-bcb8-4ae1-a89a-2eb4b551b141"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:45:01.650150 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:01.650123 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b371265-bcb8-4ae1-a89a-2eb4b551b141-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4b371265-bcb8-4ae1-a89a-2eb4b551b141" (UID: "4b371265-bcb8-4ae1-a89a-2eb4b551b141"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:45:01.650150 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:01.650140 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b371265-bcb8-4ae1-a89a-2eb4b551b141-kube-api-access-fph6x" (OuterVolumeSpecName: "kube-api-access-fph6x") pod "4b371265-bcb8-4ae1-a89a-2eb4b551b141" (UID: "4b371265-bcb8-4ae1-a89a-2eb4b551b141"). InnerVolumeSpecName "kube-api-access-fph6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:45:01.749583 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:01.749543 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fph6x\" (UniqueName: \"kubernetes.io/projected/4b371265-bcb8-4ae1-a89a-2eb4b551b141-kube-api-access-fph6x\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:45:01.749583 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:01.749574 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b371265-bcb8-4ae1-a89a-2eb4b551b141-openshift-service-ca\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:45:01.749583 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:01.749584 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b371265-bcb8-4ae1-a89a-2eb4b551b141-maas-api-tls\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:45:02.265963 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:02.265930 2570 generic.go:356] "Generic (PLEG): container finished" podID="4b371265-bcb8-4ae1-a89a-2eb4b551b141" containerID="03863d3df3b1fd8551e386c85254fc4503fd2d44c2d33f9f873e9b0cb0639fa9" exitCode=0 Jul 06 15:45:02.266160 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:02.265979 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx" Jul 06 15:45:02.266160 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:02.265987 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx" event={"ID":"4b371265-bcb8-4ae1-a89a-2eb4b551b141","Type":"ContainerDied","Data":"03863d3df3b1fd8551e386c85254fc4503fd2d44c2d33f9f873e9b0cb0639fa9"} Jul 06 15:45:02.266160 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:02.266035 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx" event={"ID":"4b371265-bcb8-4ae1-a89a-2eb4b551b141","Type":"ContainerDied","Data":"0596b97d9b6e18ad1cf24dc22384680173823f3214e1290f9e31ce418aff942c"} Jul 06 15:45:02.266160 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:02.266061 2570 scope.go:117] "RemoveContainer" containerID="03863d3df3b1fd8551e386c85254fc4503fd2d44c2d33f9f873e9b0cb0639fa9" Jul 06 15:45:02.275313 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:02.275134 2570 scope.go:117] "RemoveContainer" containerID="03863d3df3b1fd8551e386c85254fc4503fd2d44c2d33f9f873e9b0cb0639fa9" Jul 06 15:45:02.275419 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:45:02.275396 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03863d3df3b1fd8551e386c85254fc4503fd2d44c2d33f9f873e9b0cb0639fa9\": container with ID starting with 03863d3df3b1fd8551e386c85254fc4503fd2d44c2d33f9f873e9b0cb0639fa9 not found: ID does not exist" containerID="03863d3df3b1fd8551e386c85254fc4503fd2d44c2d33f9f873e9b0cb0639fa9" Jul 06 15:45:02.275464 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:02.275427 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03863d3df3b1fd8551e386c85254fc4503fd2d44c2d33f9f873e9b0cb0639fa9"} err="failed to get container status \"03863d3df3b1fd8551e386c85254fc4503fd2d44c2d33f9f873e9b0cb0639fa9\": rpc error: code = NotFound desc = could not find container \"03863d3df3b1fd8551e386c85254fc4503fd2d44c2d33f9f873e9b0cb0639fa9\": container with ID starting with 03863d3df3b1fd8551e386c85254fc4503fd2d44c2d33f9f873e9b0cb0639fa9 not found: ID does not exist" Jul 06 15:45:02.307641 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:02.307607 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx"] Jul 06 15:45:02.313215 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:02.313187 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-438fd2f6-595fd46bbd-9tspx"] Jul 06 15:45:02.799280 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:02.799248 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b371265-bcb8-4ae1-a89a-2eb4b551b141" path="/var/lib/kubelet/pods/4b371265-bcb8-4ae1-a89a-2eb4b551b141/volumes" Jul 06 15:45:06.917932 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:06.917898 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2"] Jul 06 15:45:06.918535 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:06.918090 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" podUID="dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb" containerName="istio-proxy" containerID="cri-o://e347a9fa440270ca685e54fde17b567956dc9283fdfde5509746e80db544aaa0" gracePeriod=30 Jul 06 15:45:06.971628 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:06.971598 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:07.110818 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:07.110783 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:08.605041 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:08.605012 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:11.235912 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:11.235871 2570 patch_prober.go:28] interesting pod/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 15:45:11.236365 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:11.235937 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" podUID="dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 15:45:12.184224 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.184148 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:45:12.316667 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.316634 2570 generic.go:356] "Generic (PLEG): container finished" podID="dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb" containerID="e347a9fa440270ca685e54fde17b567956dc9283fdfde5509746e80db544aaa0" exitCode=0 Jul 06 15:45:12.317136 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.316682 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" Jul 06 15:45:12.317136 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.316723 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" event={"ID":"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb","Type":"ContainerDied","Data":"e347a9fa440270ca685e54fde17b567956dc9283fdfde5509746e80db544aaa0"} Jul 06 15:45:12.317136 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.316784 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2" event={"ID":"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb","Type":"ContainerDied","Data":"6dbe7def0abb5f13d4b47afe5c884758fb625d6671cc12b9401618e2392f1fe6"} Jul 06 15:45:12.317136 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.316821 2570 scope.go:117] "RemoveContainer" containerID="e347a9fa440270ca685e54fde17b567956dc9283fdfde5509746e80db544aaa0" Jul 06 15:45:12.325265 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.325251 2570 scope.go:117] "RemoveContainer" containerID="e347a9fa440270ca685e54fde17b567956dc9283fdfde5509746e80db544aaa0" Jul 06 15:45:12.325490 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:45:12.325467 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e347a9fa440270ca685e54fde17b567956dc9283fdfde5509746e80db544aaa0\": container with ID starting with e347a9fa440270ca685e54fde17b567956dc9283fdfde5509746e80db544aaa0 not found: ID does not exist" containerID="e347a9fa440270ca685e54fde17b567956dc9283fdfde5509746e80db544aaa0" Jul 06 15:45:12.325603 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.325519 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e347a9fa440270ca685e54fde17b567956dc9283fdfde5509746e80db544aaa0"} err="failed to get container status \"e347a9fa440270ca685e54fde17b567956dc9283fdfde5509746e80db544aaa0\": rpc error: code = NotFound desc = could not find container \"e347a9fa440270ca685e54fde17b567956dc9283fdfde5509746e80db544aaa0\": container with ID starting with e347a9fa440270ca685e54fde17b567956dc9283fdfde5509746e80db544aaa0 not found: ID does not exist" Jul 06 15:45:12.356320 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.356296 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-workload-socket\") pod \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " Jul 06 15:45:12.356446 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.356332 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-workload-certs\") pod \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " Jul 06 15:45:12.356446 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.356359 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mw5sf\" (UniqueName: \"kubernetes.io/projected/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-kube-api-access-mw5sf\") pod \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " Jul 06 15:45:12.356446 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.356406 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-istio-podinfo\") pod \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " Jul 06 15:45:12.356446 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.356430 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-istiod-ca-cert\") pod \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " Jul 06 15:45:12.356764 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.356453 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-istio-data\") pod \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " Jul 06 15:45:12.356764 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.356532 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-istio-token\") pod \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " Jul 06 15:45:12.356764 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.356568 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-istio-envoy\") pod \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " Jul 06 15:45:12.356764 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.356578 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb" (UID: "dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:45:12.356764 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.356611 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-credential-socket\") pod \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\" (UID: \"dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb\") " Jul 06 15:45:12.356764 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.356654 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb" (UID: "dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:45:12.357074 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.356808 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-istio-data" (OuterVolumeSpecName: "istio-data") pod "dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb" (UID: "dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:45:12.357074 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.356908 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb" (UID: "dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:45:12.357074 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.356981 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-workload-certs\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:45:12.357074 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.357000 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-istio-data\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:45:12.357074 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.357011 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-workload-socket\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:45:12.357074 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.357010 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb" (UID: "dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:45:12.358800 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.358773 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb" (UID: "dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 15:45:12.358905 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.358780 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb" (UID: "dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:45:12.358905 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.358782 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-istio-token" (OuterVolumeSpecName: "istio-token") pod "dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb" (UID: "dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:45:12.358905 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.358857 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-kube-api-access-mw5sf" (OuterVolumeSpecName: "kube-api-access-mw5sf") pod "dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb" (UID: "dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb"). InnerVolumeSpecName "kube-api-access-mw5sf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:45:12.457982 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.457892 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-istio-token\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:45:12.457982 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.457924 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-istio-envoy\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:45:12.457982 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.457933 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-credential-socket\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:45:12.457982 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.457942 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw5sf\" (UniqueName: \"kubernetes.io/projected/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-kube-api-access-mw5sf\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:45:12.457982 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.457951 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-istio-podinfo\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:45:12.457982 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.457959 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb-istiod-ca-cert\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:45:12.653573 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.653538 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2"] Jul 06 15:45:12.658173 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.658141 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-438fd2f6-openshift-default-64bb7b9df7-sjcr2"] Jul 06 15:45:12.799310 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:12.799221 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb" path="/var/lib/kubelet/pods/dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb/volumes" Jul 06 15:45:13.908218 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:13.908181 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:14.100964 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.100931 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:14.246016 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.245983 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98"] Jul 06 15:45:14.246383 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.246370 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb" containerName="istio-proxy" Jul 06 15:45:14.246428 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.246385 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb" containerName="istio-proxy" Jul 06 15:45:14.246428 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.246403 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b371265-bcb8-4ae1-a89a-2eb4b551b141" containerName="maas-api" Jul 06 15:45:14.246428 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.246409 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b371265-bcb8-4ae1-a89a-2eb4b551b141" containerName="maas-api" Jul 06 15:45:14.246543 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.246472 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b371265-bcb8-4ae1-a89a-2eb4b551b141" containerName="maas-api" Jul 06 15:45:14.246543 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.246481 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd2585da-3b1d-4eb5-8c39-d6b8c9a30bcb" containerName="istio-proxy" Jul 06 15:45:14.250780 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.250765 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:14.259369 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.259349 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-7475f450-openshift-default-dockercfg-lq96k\"" Jul 06 15:45:14.277866 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.277837 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98"] Jul 06 15:45:14.348416 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.348386 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:14.378710 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.378660 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-istiod-ca-cert\") pod \"e2e-derive-7475f450-openshift-default-76b57757f-4mz98\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:14.378710 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.378709 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-istio-podinfo\") pod \"e2e-derive-7475f450-openshift-default-76b57757f-4mz98\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:14.378933 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.378764 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-workload-certs\") pod \"e2e-derive-7475f450-openshift-default-76b57757f-4mz98\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:14.378933 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.378798 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-credential-socket\") pod \"e2e-derive-7475f450-openshift-default-76b57757f-4mz98\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:14.378933 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.378827 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm2f6\" (UniqueName: \"kubernetes.io/projected/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-kube-api-access-zm2f6\") pod \"e2e-derive-7475f450-openshift-default-76b57757f-4mz98\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:14.378933 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.378849 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-workload-socket\") pod \"e2e-derive-7475f450-openshift-default-76b57757f-4mz98\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:14.378933 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.378869 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-istio-data\") pod \"e2e-derive-7475f450-openshift-default-76b57757f-4mz98\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:14.379088 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.378937 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-istio-envoy\") pod \"e2e-derive-7475f450-openshift-default-76b57757f-4mz98\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:14.379088 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.378973 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-istio-token\") pod \"e2e-derive-7475f450-openshift-default-76b57757f-4mz98\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:14.479896 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.479801 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-istiod-ca-cert\") pod \"e2e-derive-7475f450-openshift-default-76b57757f-4mz98\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:14.479896 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.479845 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-istio-podinfo\") pod \"e2e-derive-7475f450-openshift-default-76b57757f-4mz98\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:14.479896 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.479867 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-workload-certs\") pod \"e2e-derive-7475f450-openshift-default-76b57757f-4mz98\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:14.479896 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.479886 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-credential-socket\") pod \"e2e-derive-7475f450-openshift-default-76b57757f-4mz98\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:14.480221 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.479917 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm2f6\" (UniqueName: \"kubernetes.io/projected/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-kube-api-access-zm2f6\") pod \"e2e-derive-7475f450-openshift-default-76b57757f-4mz98\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:14.480221 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.479936 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-workload-socket\") pod \"e2e-derive-7475f450-openshift-default-76b57757f-4mz98\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:14.480221 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.479956 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-istio-data\") pod \"e2e-derive-7475f450-openshift-default-76b57757f-4mz98\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:14.480221 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.479988 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-istio-envoy\") pod \"e2e-derive-7475f450-openshift-default-76b57757f-4mz98\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:14.480221 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.480016 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-istio-token\") pod \"e2e-derive-7475f450-openshift-default-76b57757f-4mz98\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:14.480480 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.480328 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-credential-socket\") pod \"e2e-derive-7475f450-openshift-default-76b57757f-4mz98\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:14.480480 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.480405 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-workload-socket\") pod \"e2e-derive-7475f450-openshift-default-76b57757f-4mz98\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:14.480480 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.480440 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-istio-data\") pod \"e2e-derive-7475f450-openshift-default-76b57757f-4mz98\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:14.480638 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.480553 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-workload-certs\") pod \"e2e-derive-7475f450-openshift-default-76b57757f-4mz98\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:14.480717 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.480698 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-istiod-ca-cert\") pod \"e2e-derive-7475f450-openshift-default-76b57757f-4mz98\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:14.482270 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.482254 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-istio-envoy\") pod \"e2e-derive-7475f450-openshift-default-76b57757f-4mz98\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:14.482484 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.482468 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-istio-podinfo\") pod \"e2e-derive-7475f450-openshift-default-76b57757f-4mz98\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:14.497513 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.497477 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm2f6\" (UniqueName: \"kubernetes.io/projected/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-kube-api-access-zm2f6\") pod \"e2e-derive-7475f450-openshift-default-76b57757f-4mz98\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:14.498293 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.498275 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-istio-token\") pod \"e2e-derive-7475f450-openshift-default-76b57757f-4mz98\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:14.561420 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.561382 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:14.565890 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.565664 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:14.708600 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:45:14.708568 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea6efe33_b384_485c_b4d3_0d863f7f4d2c.slice/crio-544a7a0a53d91fd69bfa4ac3deea43a6046329c5ea85648143f7621e8854f296 WatchSource:0}: Error finding container 544a7a0a53d91fd69bfa4ac3deea43a6046329c5ea85648143f7621e8854f296: Status 404 returned error can't find the container with id 544a7a0a53d91fd69bfa4ac3deea43a6046329c5ea85648143f7621e8854f296 Jul 06 15:45:14.711094 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.711058 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:45:14.711176 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.711129 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:45:14.711176 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.711165 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:45:14.712863 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:14.712842 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98"] Jul 06 15:45:15.333725 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:15.333690 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" event={"ID":"ea6efe33-b384-485c-b4d3-0d863f7f4d2c","Type":"ContainerStarted","Data":"17b57a9a391a7d65a43f651214fa0235ccde03e796b3005b9f5eacf6591c651e"} Jul 06 15:45:15.334089 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:15.333738 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" event={"ID":"ea6efe33-b384-485c-b4d3-0d863f7f4d2c","Type":"ContainerStarted","Data":"544a7a0a53d91fd69bfa4ac3deea43a6046329c5ea85648143f7621e8854f296"} Jul 06 15:45:15.360736 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:15.360655 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" podStartSLOduration=1.360641258 podStartE2EDuration="1.360641258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:45:15.360031893 +0000 UTC m=+1661.973780428" watchObservedRunningTime="2026-07-06 15:45:15.360641258 +0000 UTC m=+1661.974389793" Jul 06 15:45:15.562614 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:15.562577 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:15.567433 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:15.567408 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:15.595725 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:15.595654 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:15.726557 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:15.726525 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-7475f450-656f6dff74-fjgjb"] Jul 06 15:45:15.730180 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:15.730165 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-7475f450-656f6dff74-fjgjb" Jul 06 15:45:15.733554 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:15.733534 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-7475f450-serving-cert\"" Jul 06 15:45:15.751671 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:15.751642 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-7475f450-656f6dff74-fjgjb"] Jul 06 15:45:15.767938 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:15.767913 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:15.892148 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:15.892052 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58641547-b863-400b-a4a9-0ae4ce5f1961-maas-api-tls\") pod \"maas-api-e2e-derive-7475f450-656f6dff74-fjgjb\" (UID: \"58641547-b863-400b-a4a9-0ae4ce5f1961\") " pod="opendatahub/maas-api-e2e-derive-7475f450-656f6dff74-fjgjb" Jul 06 15:45:15.892365 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:15.892336 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dff2\" (UniqueName: \"kubernetes.io/projected/58641547-b863-400b-a4a9-0ae4ce5f1961-kube-api-access-7dff2\") pod \"maas-api-e2e-derive-7475f450-656f6dff74-fjgjb\" (UID: \"58641547-b863-400b-a4a9-0ae4ce5f1961\") " pod="opendatahub/maas-api-e2e-derive-7475f450-656f6dff74-fjgjb" Jul 06 15:45:15.892409 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:15.892382 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58641547-b863-400b-a4a9-0ae4ce5f1961-openshift-service-ca\") pod \"maas-api-e2e-derive-7475f450-656f6dff74-fjgjb\" (UID: \"58641547-b863-400b-a4a9-0ae4ce5f1961\") " pod="opendatahub/maas-api-e2e-derive-7475f450-656f6dff74-fjgjb" Jul 06 15:45:15.900579 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:15.900555 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:15.994052 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:15.994007 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58641547-b863-400b-a4a9-0ae4ce5f1961-maas-api-tls\") pod \"maas-api-e2e-derive-7475f450-656f6dff74-fjgjb\" (UID: \"58641547-b863-400b-a4a9-0ae4ce5f1961\") " pod="opendatahub/maas-api-e2e-derive-7475f450-656f6dff74-fjgjb" Jul 06 15:45:15.994240 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:15.994081 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dff2\" (UniqueName: \"kubernetes.io/projected/58641547-b863-400b-a4a9-0ae4ce5f1961-kube-api-access-7dff2\") pod \"maas-api-e2e-derive-7475f450-656f6dff74-fjgjb\" (UID: \"58641547-b863-400b-a4a9-0ae4ce5f1961\") " pod="opendatahub/maas-api-e2e-derive-7475f450-656f6dff74-fjgjb" Jul 06 15:45:15.994240 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:15.994120 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58641547-b863-400b-a4a9-0ae4ce5f1961-openshift-service-ca\") pod \"maas-api-e2e-derive-7475f450-656f6dff74-fjgjb\" (UID: \"58641547-b863-400b-a4a9-0ae4ce5f1961\") " pod="opendatahub/maas-api-e2e-derive-7475f450-656f6dff74-fjgjb" Jul 06 15:45:15.994240 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:45:15.994167 2570 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-7475f450-serving-cert: secret "maas-api-e2e-derive-7475f450-serving-cert" not found Jul 06 15:45:15.994386 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:45:15.994249 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58641547-b863-400b-a4a9-0ae4ce5f1961-maas-api-tls podName:58641547-b863-400b-a4a9-0ae4ce5f1961 nodeName:}" failed. No retries permitted until 2026-07-06 15:45:16.494227044 +0000 UTC m=+1663.107975558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/58641547-b863-400b-a4a9-0ae4ce5f1961-maas-api-tls") pod "maas-api-e2e-derive-7475f450-656f6dff74-fjgjb" (UID: "58641547-b863-400b-a4a9-0ae4ce5f1961") : secret "maas-api-e2e-derive-7475f450-serving-cert" not found Jul 06 15:45:15.994830 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:15.994811 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58641547-b863-400b-a4a9-0ae4ce5f1961-openshift-service-ca\") pod \"maas-api-e2e-derive-7475f450-656f6dff74-fjgjb\" (UID: \"58641547-b863-400b-a4a9-0ae4ce5f1961\") " pod="opendatahub/maas-api-e2e-derive-7475f450-656f6dff74-fjgjb" Jul 06 15:45:16.022045 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:16.022015 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dff2\" (UniqueName: \"kubernetes.io/projected/58641547-b863-400b-a4a9-0ae4ce5f1961-kube-api-access-7dff2\") pod \"maas-api-e2e-derive-7475f450-656f6dff74-fjgjb\" (UID: \"58641547-b863-400b-a4a9-0ae4ce5f1961\") " pod="opendatahub/maas-api-e2e-derive-7475f450-656f6dff74-fjgjb" Jul 06 15:45:16.339220 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:16.339190 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:16.340338 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:16.340320 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:16.432283 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:16.432252 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:16.498640 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:16.498599 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58641547-b863-400b-a4a9-0ae4ce5f1961-maas-api-tls\") pod \"maas-api-e2e-derive-7475f450-656f6dff74-fjgjb\" (UID: \"58641547-b863-400b-a4a9-0ae4ce5f1961\") " pod="opendatahub/maas-api-e2e-derive-7475f450-656f6dff74-fjgjb" Jul 06 15:45:16.500954 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:16.500930 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58641547-b863-400b-a4a9-0ae4ce5f1961-maas-api-tls\") pod \"maas-api-e2e-derive-7475f450-656f6dff74-fjgjb\" (UID: \"58641547-b863-400b-a4a9-0ae4ce5f1961\") " pod="opendatahub/maas-api-e2e-derive-7475f450-656f6dff74-fjgjb" Jul 06 15:45:16.641120 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:16.641039 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-7475f450-656f6dff74-fjgjb" Jul 06 15:45:16.787493 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:16.787463 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-7475f450-656f6dff74-fjgjb"] Jul 06 15:45:16.789387 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:45:16.789360 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58641547_b863_400b_a4a9_0ae4ce5f1961.slice/crio-0a91abf01ef8610fab106d8f8322755fb92247046c3862c1a25c64e9df4a7092 WatchSource:0}: Error finding container 0a91abf01ef8610fab106d8f8322755fb92247046c3862c1a25c64e9df4a7092: Status 404 returned error can't find the container with id 0a91abf01ef8610fab106d8f8322755fb92247046c3862c1a25c64e9df4a7092 Jul 06 15:45:17.185010 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:17.184979 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:17.345812 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:17.345769 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-7475f450-656f6dff74-fjgjb" event={"ID":"58641547-b863-400b-a4a9-0ae4ce5f1961","Type":"ContainerStarted","Data":"ffbbf70b21813d789d783174e226a0de3749706ee04f936a5786f92d9708aba4"} Jul 06 15:45:17.346212 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:17.345824 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-7475f450-656f6dff74-fjgjb" event={"ID":"58641547-b863-400b-a4a9-0ae4ce5f1961","Type":"ContainerStarted","Data":"0a91abf01ef8610fab106d8f8322755fb92247046c3862c1a25c64e9df4a7092"} Jul 06 15:45:17.346212 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:17.345865 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-7475f450-656f6dff74-fjgjb" Jul 06 15:45:17.591908 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:17.591869 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:17.765153 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:17.765117 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:20.079511 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:20.079473 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:20.278446 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:20.278391 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-7475f450-656f6dff74-fjgjb" podStartSLOduration=4.865405504 podStartE2EDuration="5.278374911s" totalImagesPullingTime="412.969407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:45:17.386884044 +0000 UTC m=+1664.000632579" watchObservedRunningTime="2026-07-06 15:45:20.278374911 +0000 UTC m=+1666.892123472" Jul 06 15:45:20.280378 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:20.280357 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98"] Jul 06 15:45:20.280603 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:20.280572 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" podUID="ea6efe33-b384-485c-b4d3-0d863f7f4d2c" containerName="istio-proxy" containerID="cri-o://17b57a9a391a7d65a43f651214fa0235ccde03e796b3005b9f5eacf6591c651e" gracePeriod=30 Jul 06 15:45:20.311364 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:20.311336 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:20.426851 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:20.426753 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-7475f450-656f6dff74-fjgjb"] Jul 06 15:45:20.427239 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:20.427052 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-7475f450-656f6dff74-fjgjb" podUID="58641547-b863-400b-a4a9-0ae4ce5f1961" containerName="maas-api" containerID="cri-o://ffbbf70b21813d789d783174e226a0de3749706ee04f936a5786f92d9708aba4" gracePeriod=30 Jul 06 15:45:20.433948 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:20.433918 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:20.532959 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:20.532913 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:20.676341 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:20.676312 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-7475f450-656f6dff74-fjgjb" Jul 06 15:45:20.738572 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:20.738465 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dff2\" (UniqueName: \"kubernetes.io/projected/58641547-b863-400b-a4a9-0ae4ce5f1961-kube-api-access-7dff2\") pod \"58641547-b863-400b-a4a9-0ae4ce5f1961\" (UID: \"58641547-b863-400b-a4a9-0ae4ce5f1961\") " Jul 06 15:45:20.738572 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:20.738554 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58641547-b863-400b-a4a9-0ae4ce5f1961-openshift-service-ca\") pod \"58641547-b863-400b-a4a9-0ae4ce5f1961\" (UID: \"58641547-b863-400b-a4a9-0ae4ce5f1961\") " Jul 06 15:45:20.738787 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:20.738592 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58641547-b863-400b-a4a9-0ae4ce5f1961-maas-api-tls\") pod \"58641547-b863-400b-a4a9-0ae4ce5f1961\" (UID: \"58641547-b863-400b-a4a9-0ae4ce5f1961\") " Jul 06 15:45:20.738896 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:20.738872 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58641547-b863-400b-a4a9-0ae4ce5f1961-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "58641547-b863-400b-a4a9-0ae4ce5f1961" (UID: "58641547-b863-400b-a4a9-0ae4ce5f1961"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:45:20.740542 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:20.740514 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58641547-b863-400b-a4a9-0ae4ce5f1961-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "58641547-b863-400b-a4a9-0ae4ce5f1961" (UID: "58641547-b863-400b-a4a9-0ae4ce5f1961"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:45:20.740647 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:20.740581 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58641547-b863-400b-a4a9-0ae4ce5f1961-kube-api-access-7dff2" (OuterVolumeSpecName: "kube-api-access-7dff2") pod "58641547-b863-400b-a4a9-0ae4ce5f1961" (UID: "58641547-b863-400b-a4a9-0ae4ce5f1961"). InnerVolumeSpecName "kube-api-access-7dff2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:45:20.839549 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:20.839481 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dff2\" (UniqueName: \"kubernetes.io/projected/58641547-b863-400b-a4a9-0ae4ce5f1961-kube-api-access-7dff2\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:45:20.839549 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:20.839546 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58641547-b863-400b-a4a9-0ae4ce5f1961-openshift-service-ca\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:45:20.839821 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:20.839563 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58641547-b863-400b-a4a9-0ae4ce5f1961-maas-api-tls\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:45:21.366423 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:21.366384 2570 generic.go:356] "Generic (PLEG): container finished" podID="58641547-b863-400b-a4a9-0ae4ce5f1961" containerID="ffbbf70b21813d789d783174e226a0de3749706ee04f936a5786f92d9708aba4" exitCode=0 Jul 06 15:45:21.366875 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:21.366432 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-7475f450-656f6dff74-fjgjb" Jul 06 15:45:21.366875 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:21.366474 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-7475f450-656f6dff74-fjgjb" event={"ID":"58641547-b863-400b-a4a9-0ae4ce5f1961","Type":"ContainerDied","Data":"ffbbf70b21813d789d783174e226a0de3749706ee04f936a5786f92d9708aba4"} Jul 06 15:45:21.366875 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:21.366555 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-7475f450-656f6dff74-fjgjb" event={"ID":"58641547-b863-400b-a4a9-0ae4ce5f1961","Type":"ContainerDied","Data":"0a91abf01ef8610fab106d8f8322755fb92247046c3862c1a25c64e9df4a7092"} Jul 06 15:45:21.366875 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:21.366580 2570 scope.go:117] "RemoveContainer" containerID="ffbbf70b21813d789d783174e226a0de3749706ee04f936a5786f92d9708aba4" Jul 06 15:45:21.377649 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:21.377616 2570 scope.go:117] "RemoveContainer" containerID="ffbbf70b21813d789d783174e226a0de3749706ee04f936a5786f92d9708aba4" Jul 06 15:45:21.378037 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:45:21.378018 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffbbf70b21813d789d783174e226a0de3749706ee04f936a5786f92d9708aba4\": container with ID starting with ffbbf70b21813d789d783174e226a0de3749706ee04f936a5786f92d9708aba4 not found: ID does not exist" containerID="ffbbf70b21813d789d783174e226a0de3749706ee04f936a5786f92d9708aba4" Jul 06 15:45:21.378087 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:21.378046 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffbbf70b21813d789d783174e226a0de3749706ee04f936a5786f92d9708aba4"} err="failed to get container status \"ffbbf70b21813d789d783174e226a0de3749706ee04f936a5786f92d9708aba4\": rpc error: code = NotFound desc = could not find container \"ffbbf70b21813d789d783174e226a0de3749706ee04f936a5786f92d9708aba4\": container with ID starting with ffbbf70b21813d789d783174e226a0de3749706ee04f936a5786f92d9708aba4 not found: ID does not exist" Jul 06 15:45:21.385341 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:21.385320 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-7475f450-656f6dff74-fjgjb"] Jul 06 15:45:21.388693 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:21.388673 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-7475f450-656f6dff74-fjgjb"] Jul 06 15:45:22.300132 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:22.300105 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:22.799673 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:22.799645 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58641547-b863-400b-a4a9-0ae4ce5f1961" path="/var/lib/kubelet/pods/58641547-b863-400b-a4a9-0ae4ce5f1961/volumes" Jul 06 15:45:25.542109 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:25.542038 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:25.580820 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:25.580787 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zm2f6\" (UniqueName: \"kubernetes.io/projected/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-kube-api-access-zm2f6\") pod \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " Jul 06 15:45:25.580820 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:25.580827 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-istio-token\") pod \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " Jul 06 15:45:25.581070 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:25.580864 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-istiod-ca-cert\") pod \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " Jul 06 15:45:25.581070 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:25.580884 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-workload-certs\") pod \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " Jul 06 15:45:25.581070 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:25.580901 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-workload-socket\") pod \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " Jul 06 15:45:25.581070 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:25.580928 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-istio-podinfo\") pod \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " Jul 06 15:45:25.581070 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:25.580955 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-istio-data\") pod \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " Jul 06 15:45:25.581070 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:25.581028 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-credential-socket\") pod \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " Jul 06 15:45:25.581349 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:25.581093 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-istio-envoy\") pod \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\" (UID: \"ea6efe33-b384-485c-b4d3-0d863f7f4d2c\") " Jul 06 15:45:25.581349 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:25.581221 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ea6efe33-b384-485c-b4d3-0d863f7f4d2c" (UID: "ea6efe33-b384-485c-b4d3-0d863f7f4d2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:45:25.581460 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:25.581361 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ea6efe33-b384-485c-b4d3-0d863f7f4d2c" (UID: "ea6efe33-b384-485c-b4d3-0d863f7f4d2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:45:25.581460 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:25.581395 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-istio-data" (OuterVolumeSpecName: "istio-data") pod "ea6efe33-b384-485c-b4d3-0d863f7f4d2c" (UID: "ea6efe33-b384-485c-b4d3-0d863f7f4d2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:45:25.581460 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:25.581405 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ea6efe33-b384-485c-b4d3-0d863f7f4d2c" (UID: "ea6efe33-b384-485c-b4d3-0d863f7f4d2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:45:25.581650 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:25.581591 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-workload-certs\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:45:25.581650 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:25.581611 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-workload-socket\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:45:25.581650 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:25.581615 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ea6efe33-b384-485c-b4d3-0d863f7f4d2c" (UID: "ea6efe33-b384-485c-b4d3-0d863f7f4d2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:45:25.581650 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:25.581625 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-istio-data\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:45:25.581650 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:25.581640 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-credential-socket\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:45:25.583172 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:25.583149 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-istio-token" (OuterVolumeSpecName: "istio-token") pod "ea6efe33-b384-485c-b4d3-0d863f7f4d2c" (UID: "ea6efe33-b384-485c-b4d3-0d863f7f4d2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:45:25.583323 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:25.583304 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-kube-api-access-zm2f6" (OuterVolumeSpecName: "kube-api-access-zm2f6") pod "ea6efe33-b384-485c-b4d3-0d863f7f4d2c" (UID: "ea6efe33-b384-485c-b4d3-0d863f7f4d2c"). InnerVolumeSpecName "kube-api-access-zm2f6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:45:25.583787 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:25.583768 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ea6efe33-b384-485c-b4d3-0d863f7f4d2c" (UID: "ea6efe33-b384-485c-b4d3-0d863f7f4d2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:45:25.583787 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:25.583769 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ea6efe33-b384-485c-b4d3-0d863f7f4d2c" (UID: "ea6efe33-b384-485c-b4d3-0d863f7f4d2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 15:45:25.682924 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:25.682891 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-istio-envoy\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:45:25.682924 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:25.682919 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm2f6\" (UniqueName: \"kubernetes.io/projected/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-kube-api-access-zm2f6\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:45:25.682924 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:25.682930 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-istio-token\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:45:25.683161 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:25.682939 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-istiod-ca-cert\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:45:25.683161 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:25.682947 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea6efe33-b384-485c-b4d3-0d863f7f4d2c-istio-podinfo\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:45:26.393013 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:26.392979 2570 generic.go:356] "Generic (PLEG): container finished" podID="ea6efe33-b384-485c-b4d3-0d863f7f4d2c" containerID="17b57a9a391a7d65a43f651214fa0235ccde03e796b3005b9f5eacf6591c651e" exitCode=0 Jul 06 15:45:26.393191 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:26.393055 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" Jul 06 15:45:26.393191 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:26.393060 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" event={"ID":"ea6efe33-b384-485c-b4d3-0d863f7f4d2c","Type":"ContainerDied","Data":"17b57a9a391a7d65a43f651214fa0235ccde03e796b3005b9f5eacf6591c651e"} Jul 06 15:45:26.393191 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:26.393136 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98" event={"ID":"ea6efe33-b384-485c-b4d3-0d863f7f4d2c","Type":"ContainerDied","Data":"544a7a0a53d91fd69bfa4ac3deea43a6046329c5ea85648143f7621e8854f296"} Jul 06 15:45:26.393191 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:26.393164 2570 scope.go:117] "RemoveContainer" containerID="17b57a9a391a7d65a43f651214fa0235ccde03e796b3005b9f5eacf6591c651e" Jul 06 15:45:26.402690 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:26.402677 2570 scope.go:117] "RemoveContainer" containerID="17b57a9a391a7d65a43f651214fa0235ccde03e796b3005b9f5eacf6591c651e" Jul 06 15:45:26.403102 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:45:26.403072 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17b57a9a391a7d65a43f651214fa0235ccde03e796b3005b9f5eacf6591c651e\": container with ID starting with 17b57a9a391a7d65a43f651214fa0235ccde03e796b3005b9f5eacf6591c651e not found: ID does not exist" containerID="17b57a9a391a7d65a43f651214fa0235ccde03e796b3005b9f5eacf6591c651e" Jul 06 15:45:26.403203 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:26.403122 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17b57a9a391a7d65a43f651214fa0235ccde03e796b3005b9f5eacf6591c651e"} err="failed to get container status \"17b57a9a391a7d65a43f651214fa0235ccde03e796b3005b9f5eacf6591c651e\": rpc error: code = NotFound desc = could not find container \"17b57a9a391a7d65a43f651214fa0235ccde03e796b3005b9f5eacf6591c651e\": container with ID starting with 17b57a9a391a7d65a43f651214fa0235ccde03e796b3005b9f5eacf6591c651e not found: ID does not exist" Jul 06 15:45:26.427568 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:26.427531 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98"] Jul 06 15:45:26.445890 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:26.445854 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-7475f450-openshift-default-76b57757f-4mz98"] Jul 06 15:45:26.800412 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:26.800332 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea6efe33-b384-485c-b4d3-0d863f7f4d2c" path="/var/lib/kubelet/pods/ea6efe33-b384-485c-b4d3-0d863f7f4d2c/volumes" Jul 06 15:45:27.371836 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:27.371805 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:27.480082 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:27.480051 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:27.992308 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:27.992275 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:28.119818 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:28.119787 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:28.266458 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:28.266377 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:30.009319 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:30.009277 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:36.931816 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:36.931777 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:38.243111 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:38.243077 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:38.568047 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:45:38.567949 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:46:04.676838 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:46:04.676727 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:46:12.429282 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:46:12.429250 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:46:12.530454 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:46:12.530421 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:46:27.358442 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:46:27.358408 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:46:33.701030 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:46:33.700995 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:46:33.861512 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:46:33.861465 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:46:34.032601 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:46:34.032497 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:46:44.466609 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:46:44.466572 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:46:45.266283 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:46:45.266250 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:46:49.479090 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:46:49.479051 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:46:49.579652 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:46:49.579622 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:46:51.347219 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:46:51.347178 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:46:51.458677 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:46:51.458651 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:46:51.570328 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:46:51.570293 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:46:51.673015 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:46:51.672918 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:46:53.655386 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:46:53.655348 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:46:53.722240 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:46:53.722209 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:46:54.334888 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:46:54.334853 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:00.554934 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:00.554905 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:00.649316 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:00.649281 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:01.237018 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:01.236966 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:07.347112 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:07.347061 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:07.441867 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:07.441836 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:07.592175 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:07.592143 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:07.901593 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:07.901558 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4"] Jul 06 15:47:07.901927 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:07.901915 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea6efe33-b384-485c-b4d3-0d863f7f4d2c" containerName="istio-proxy" Jul 06 15:47:07.901979 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:07.901928 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea6efe33-b384-485c-b4d3-0d863f7f4d2c" containerName="istio-proxy" Jul 06 15:47:07.901979 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:07.901941 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58641547-b863-400b-a4a9-0ae4ce5f1961" containerName="maas-api" Jul 06 15:47:07.901979 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:07.901947 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="58641547-b863-400b-a4a9-0ae4ce5f1961" containerName="maas-api" Jul 06 15:47:07.902074 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:07.902016 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="58641547-b863-400b-a4a9-0ae4ce5f1961" containerName="maas-api" Jul 06 15:47:07.902074 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:07.902026 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea6efe33-b384-485c-b4d3-0d863f7f4d2c" containerName="istio-proxy" Jul 06 15:47:07.904994 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:07.904974 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:07.908559 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:07.908540 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7736d688-openshift-default-dockercfg-7rtvp\"" Jul 06 15:47:07.931438 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:07.931413 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:07.939677 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:07.939650 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4"] Jul 06 15:47:08.068970 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.068931 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfe8906-f271-4fd7-adca-60975b73cd8a-credential-socket\") pod \"e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:08.069151 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.068985 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfe8906-f271-4fd7-adca-60975b73cd8a-workload-socket\") pod \"e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:08.069151 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.069038 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dfe8906-f271-4fd7-adca-60975b73cd8a-istiod-ca-cert\") pod \"e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:08.069151 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.069073 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l69rs\" (UniqueName: \"kubernetes.io/projected/5dfe8906-f271-4fd7-adca-60975b73cd8a-kube-api-access-l69rs\") pod \"e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:08.069151 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.069101 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dfe8906-f271-4fd7-adca-60975b73cd8a-istio-podinfo\") pod \"e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:08.069151 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.069150 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dfe8906-f271-4fd7-adca-60975b73cd8a-workload-certs\") pod \"e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:08.069318 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.069182 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dfe8906-f271-4fd7-adca-60975b73cd8a-istio-envoy\") pod \"e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:08.069318 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.069225 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dfe8906-f271-4fd7-adca-60975b73cd8a-istio-data\") pod \"e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:08.069318 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.069246 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dfe8906-f271-4fd7-adca-60975b73cd8a-istio-token\") pod \"e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:08.161098 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.161001 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:08.169825 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.169797 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dfe8906-f271-4fd7-adca-60975b73cd8a-istio-data\") pod \"e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:08.169982 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.169832 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dfe8906-f271-4fd7-adca-60975b73cd8a-istio-token\") pod \"e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:08.169982 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.169867 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfe8906-f271-4fd7-adca-60975b73cd8a-credential-socket\") pod \"e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:08.169982 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.169893 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfe8906-f271-4fd7-adca-60975b73cd8a-workload-socket\") pod \"e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:08.169982 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.169912 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dfe8906-f271-4fd7-adca-60975b73cd8a-istiod-ca-cert\") pod \"e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:08.169982 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.169931 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l69rs\" (UniqueName: \"kubernetes.io/projected/5dfe8906-f271-4fd7-adca-60975b73cd8a-kube-api-access-l69rs\") pod \"e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:08.169982 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.169953 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dfe8906-f271-4fd7-adca-60975b73cd8a-istio-podinfo\") pod \"e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:08.170343 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.170007 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dfe8906-f271-4fd7-adca-60975b73cd8a-workload-certs\") pod \"e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:08.170343 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.170052 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dfe8906-f271-4fd7-adca-60975b73cd8a-istio-envoy\") pod \"e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:08.170343 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.170262 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dfe8906-f271-4fd7-adca-60975b73cd8a-istio-data\") pod \"e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:08.170460 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.170375 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfe8906-f271-4fd7-adca-60975b73cd8a-workload-socket\") pod \"e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:08.170526 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.170474 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfe8906-f271-4fd7-adca-60975b73cd8a-credential-socket\") pod \"e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:08.170742 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.170719 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dfe8906-f271-4fd7-adca-60975b73cd8a-workload-certs\") pod \"e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:08.170742 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.170731 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dfe8906-f271-4fd7-adca-60975b73cd8a-istiod-ca-cert\") pod \"e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:08.172455 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.172434 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dfe8906-f271-4fd7-adca-60975b73cd8a-istio-envoy\") pod \"e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:08.172619 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.172600 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dfe8906-f271-4fd7-adca-60975b73cd8a-istio-podinfo\") pod \"e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:08.179392 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.179370 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dfe8906-f271-4fd7-adca-60975b73cd8a-istio-token\") pod \"e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:08.179552 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.179533 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l69rs\" (UniqueName: \"kubernetes.io/projected/5dfe8906-f271-4fd7-adca-60975b73cd8a-kube-api-access-l69rs\") pod \"e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:08.215166 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.215141 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:08.358900 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.358873 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4"] Jul 06 15:47:08.361273 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:47:08.361244 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dfe8906_f271_4fd7_adca_60975b73cd8a.slice/crio-c65f334055c3a9fbb416693302af25d3e7e5dccd3aff5dfb468d80bb5af6851d WatchSource:0}: Error finding container c65f334055c3a9fbb416693302af25d3e7e5dccd3aff5dfb468d80bb5af6851d: Status 404 returned error can't find the container with id c65f334055c3a9fbb416693302af25d3e7e5dccd3aff5dfb468d80bb5af6851d Jul 06 15:47:08.363677 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.363635 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:47:08.363772 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.363731 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:47:08.363812 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.363784 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:47:08.880864 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.880823 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" event={"ID":"5dfe8906-f271-4fd7-adca-60975b73cd8a","Type":"ContainerStarted","Data":"60d813c44ab5c11c2c1ad010807008039a1580904e1bdea9939be6ef8e103a78"} Jul 06 15:47:08.881127 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.880884 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" event={"ID":"5dfe8906-f271-4fd7-adca-60975b73cd8a","Type":"ContainerStarted","Data":"c65f334055c3a9fbb416693302af25d3e7e5dccd3aff5dfb468d80bb5af6851d"} Jul 06 15:47:08.912836 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:08.912786 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" podStartSLOduration=1.912768823 podStartE2EDuration="1.912768823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:47:08.911981746 +0000 UTC m=+1775.525730281" watchObservedRunningTime="2026-07-06 15:47:08.912768823 +0000 UTC m=+1775.526517424" Jul 06 15:47:09.216251 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:09.216164 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:09.220910 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:09.220887 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:09.888465 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:09.888429 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:09.889287 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:09.889265 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:11.101706 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:11.101677 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:11.227450 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:11.227417 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-7736d688-9d98b4667-2b5dh"] Jul 06 15:47:11.231007 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:11.230991 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-7736d688-9d98b4667-2b5dh" Jul 06 15:47:11.239847 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:47:11.239818 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-e2e-mt-7736d688-9d98b4667-2b5dh\" is forbidden: User \"system:node:ip-10-0-130-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-108.ec2.internal' and this object" podUID="dc887352-4d3f-4a56-8d4d-3458ba383401" pod="opendatahub/maas-api-e2e-mt-7736d688-9d98b4667-2b5dh" Jul 06 15:47:11.240034 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:47:11.240012 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"maas-api-e2e-mt-7736d688-serving-cert\" is forbidden: User \"system:node:ip-10-0-130-108.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-108.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-7736d688-serving-cert\"" type="*v1.Secret" Jul 06 15:47:11.258637 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:11.258609 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-7736d688-9d98b4667-2b5dh"] Jul 06 15:47:11.304319 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:11.304286 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:11.404796 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:11.404691 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc887352-4d3f-4a56-8d4d-3458ba383401-openshift-service-ca\") pod \"maas-api-e2e-mt-7736d688-9d98b4667-2b5dh\" (UID: \"dc887352-4d3f-4a56-8d4d-3458ba383401\") " pod="opendatahub/maas-api-e2e-mt-7736d688-9d98b4667-2b5dh" Jul 06 15:47:11.404961 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:11.404836 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgsvb\" (UniqueName: \"kubernetes.io/projected/dc887352-4d3f-4a56-8d4d-3458ba383401-kube-api-access-jgsvb\") pod \"maas-api-e2e-mt-7736d688-9d98b4667-2b5dh\" (UID: \"dc887352-4d3f-4a56-8d4d-3458ba383401\") " pod="opendatahub/maas-api-e2e-mt-7736d688-9d98b4667-2b5dh" Jul 06 15:47:11.404961 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:11.404933 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc887352-4d3f-4a56-8d4d-3458ba383401-maas-api-tls\") pod \"maas-api-e2e-mt-7736d688-9d98b4667-2b5dh\" (UID: \"dc887352-4d3f-4a56-8d4d-3458ba383401\") " pod="opendatahub/maas-api-e2e-mt-7736d688-9d98b4667-2b5dh" Jul 06 15:47:11.506229 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:11.506192 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc887352-4d3f-4a56-8d4d-3458ba383401-openshift-service-ca\") pod \"maas-api-e2e-mt-7736d688-9d98b4667-2b5dh\" (UID: \"dc887352-4d3f-4a56-8d4d-3458ba383401\") " pod="opendatahub/maas-api-e2e-mt-7736d688-9d98b4667-2b5dh" Jul 06 15:47:11.506429 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:11.506282 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgsvb\" (UniqueName: \"kubernetes.io/projected/dc887352-4d3f-4a56-8d4d-3458ba383401-kube-api-access-jgsvb\") pod \"maas-api-e2e-mt-7736d688-9d98b4667-2b5dh\" (UID: \"dc887352-4d3f-4a56-8d4d-3458ba383401\") " pod="opendatahub/maas-api-e2e-mt-7736d688-9d98b4667-2b5dh" Jul 06 15:47:11.506429 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:11.506325 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc887352-4d3f-4a56-8d4d-3458ba383401-maas-api-tls\") pod \"maas-api-e2e-mt-7736d688-9d98b4667-2b5dh\" (UID: \"dc887352-4d3f-4a56-8d4d-3458ba383401\") " pod="opendatahub/maas-api-e2e-mt-7736d688-9d98b4667-2b5dh" Jul 06 15:47:11.506936 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:11.506909 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc887352-4d3f-4a56-8d4d-3458ba383401-openshift-service-ca\") pod \"maas-api-e2e-mt-7736d688-9d98b4667-2b5dh\" (UID: \"dc887352-4d3f-4a56-8d4d-3458ba383401\") " pod="opendatahub/maas-api-e2e-mt-7736d688-9d98b4667-2b5dh" Jul 06 15:47:11.514876 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:11.514849 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:11.523696 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:11.523672 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgsvb\" (UniqueName: \"kubernetes.io/projected/dc887352-4d3f-4a56-8d4d-3458ba383401-kube-api-access-jgsvb\") pod \"maas-api-e2e-mt-7736d688-9d98b4667-2b5dh\" (UID: \"dc887352-4d3f-4a56-8d4d-3458ba383401\") " pod="opendatahub/maas-api-e2e-mt-7736d688-9d98b4667-2b5dh" Jul 06 15:47:11.826544 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:11.826515 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:12.040544 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:12.040512 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:12.141825 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:12.141725 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:12.507294 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:47:12.507202 2570 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-7736d688-serving-cert: failed to sync secret cache: timed out waiting for the condition Jul 06 15:47:12.507458 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:47:12.507296 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc887352-4d3f-4a56-8d4d-3458ba383401-maas-api-tls podName:dc887352-4d3f-4a56-8d4d-3458ba383401 nodeName:}" failed. No retries permitted until 2026-07-06 15:47:13.007280654 +0000 UTC m=+1779.621029168 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dc887352-4d3f-4a56-8d4d-3458ba383401-maas-api-tls") pod "maas-api-e2e-mt-7736d688-9d98b4667-2b5dh" (UID: "dc887352-4d3f-4a56-8d4d-3458ba383401") : failed to sync secret cache: timed out waiting for the condition Jul 06 15:47:12.695843 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:12.695809 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-7736d688-serving-cert\"" Jul 06 15:47:13.023286 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:13.023253 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc887352-4d3f-4a56-8d4d-3458ba383401-maas-api-tls\") pod \"maas-api-e2e-mt-7736d688-9d98b4667-2b5dh\" (UID: \"dc887352-4d3f-4a56-8d4d-3458ba383401\") " pod="opendatahub/maas-api-e2e-mt-7736d688-9d98b4667-2b5dh" Jul 06 15:47:13.025725 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:13.025704 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc887352-4d3f-4a56-8d4d-3458ba383401-maas-api-tls\") pod \"maas-api-e2e-mt-7736d688-9d98b4667-2b5dh\" (UID: \"dc887352-4d3f-4a56-8d4d-3458ba383401\") " pod="opendatahub/maas-api-e2e-mt-7736d688-9d98b4667-2b5dh" Jul 06 15:47:13.041769 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:13.041719 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-7736d688-9d98b4667-2b5dh" Jul 06 15:47:13.179171 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:13.179147 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-7736d688-9d98b4667-2b5dh"] Jul 06 15:47:13.181089 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:47:13.181062 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc887352_4d3f_4a56_8d4d_3458ba383401.slice/crio-0d47a3afb9fe9c9fccf87ea405bc412b6bc9875e5855c8cd927ffc73f15bc0e4 WatchSource:0}: Error finding container 0d47a3afb9fe9c9fccf87ea405bc412b6bc9875e5855c8cd927ffc73f15bc0e4: Status 404 returned error can't find the container with id 0d47a3afb9fe9c9fccf87ea405bc412b6bc9875e5855c8cd927ffc73f15bc0e4 Jul 06 15:47:13.914751 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:13.914712 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-7736d688-9d98b4667-2b5dh" event={"ID":"dc887352-4d3f-4a56-8d4d-3458ba383401","Type":"ContainerStarted","Data":"d9c5bb6400d40e0febc4240957b871e57c8c1e4d561f766f156a14ebfc3c877d"} Jul 06 15:47:13.914967 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:13.914798 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-7736d688-9d98b4667-2b5dh" event={"ID":"dc887352-4d3f-4a56-8d4d-3458ba383401","Type":"ContainerStarted","Data":"0d47a3afb9fe9c9fccf87ea405bc412b6bc9875e5855c8cd927ffc73f15bc0e4"} Jul 06 15:47:13.914967 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:13.914833 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-7736d688-9d98b4667-2b5dh" Jul 06 15:47:13.937349 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:13.937298 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-7736d688-9d98b4667-2b5dh" podStartSLOduration=2.5581185680000003 podStartE2EDuration="2.937283486s" totalImagesPullingTime="379.164918ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:47:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:47:13.932653404 +0000 UTC m=+1780.546401940" watchObservedRunningTime="2026-07-06 15:47:13.937283486 +0000 UTC m=+1780.551032048" Jul 06 15:47:15.474380 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:15.474348 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:16.159302 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:16.159271 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:16.262995 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:16.262966 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:16.610124 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:16.610059 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:19.925380 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:19.925349 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-7736d688-9d98b4667-2b5dh" Jul 06 15:47:22.581046 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:22.580994 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:22.690554 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:22.690480 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:23.128037 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:23.128008 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:23.396554 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:23.396399 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:23.543901 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:23.543861 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-7736d688-9d98b4667-2b5dh"] Jul 06 15:47:23.544184 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:23.544149 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-7736d688-9d98b4667-2b5dh" podUID="dc887352-4d3f-4a56-8d4d-3458ba383401" containerName="maas-api" containerID="cri-o://d9c5bb6400d40e0febc4240957b871e57c8c1e4d561f766f156a14ebfc3c877d" gracePeriod=30 Jul 06 15:47:23.578412 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:23.578375 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:23.808360 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:23.808329 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-7736d688-9d98b4667-2b5dh" Jul 06 15:47:23.824857 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:23.824827 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgsvb\" (UniqueName: \"kubernetes.io/projected/dc887352-4d3f-4a56-8d4d-3458ba383401-kube-api-access-jgsvb\") pod \"dc887352-4d3f-4a56-8d4d-3458ba383401\" (UID: \"dc887352-4d3f-4a56-8d4d-3458ba383401\") " Jul 06 15:47:23.825004 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:23.824864 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc887352-4d3f-4a56-8d4d-3458ba383401-maas-api-tls\") pod \"dc887352-4d3f-4a56-8d4d-3458ba383401\" (UID: \"dc887352-4d3f-4a56-8d4d-3458ba383401\") " Jul 06 15:47:23.825004 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:23.824886 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc887352-4d3f-4a56-8d4d-3458ba383401-openshift-service-ca\") pod \"dc887352-4d3f-4a56-8d4d-3458ba383401\" (UID: \"dc887352-4d3f-4a56-8d4d-3458ba383401\") " Jul 06 15:47:23.825349 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:23.825314 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc887352-4d3f-4a56-8d4d-3458ba383401-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dc887352-4d3f-4a56-8d4d-3458ba383401" (UID: "dc887352-4d3f-4a56-8d4d-3458ba383401"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:47:23.826927 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:23.826901 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc887352-4d3f-4a56-8d4d-3458ba383401-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dc887352-4d3f-4a56-8d4d-3458ba383401" (UID: "dc887352-4d3f-4a56-8d4d-3458ba383401"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:47:23.827057 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:23.827037 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc887352-4d3f-4a56-8d4d-3458ba383401-kube-api-access-jgsvb" (OuterVolumeSpecName: "kube-api-access-jgsvb") pod "dc887352-4d3f-4a56-8d4d-3458ba383401" (UID: "dc887352-4d3f-4a56-8d4d-3458ba383401"). InnerVolumeSpecName "kube-api-access-jgsvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:47:23.926290 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:23.926202 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgsvb\" (UniqueName: \"kubernetes.io/projected/dc887352-4d3f-4a56-8d4d-3458ba383401-kube-api-access-jgsvb\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:47:23.926290 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:23.926246 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc887352-4d3f-4a56-8d4d-3458ba383401-maas-api-tls\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:47:23.926290 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:23.926260 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc887352-4d3f-4a56-8d4d-3458ba383401-openshift-service-ca\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:47:23.972315 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:23.972282 2570 generic.go:356] "Generic (PLEG): container finished" podID="dc887352-4d3f-4a56-8d4d-3458ba383401" containerID="d9c5bb6400d40e0febc4240957b871e57c8c1e4d561f766f156a14ebfc3c877d" exitCode=0 Jul 06 15:47:23.972492 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:23.972345 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-7736d688-9d98b4667-2b5dh" Jul 06 15:47:23.972492 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:23.972371 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-7736d688-9d98b4667-2b5dh" event={"ID":"dc887352-4d3f-4a56-8d4d-3458ba383401","Type":"ContainerDied","Data":"d9c5bb6400d40e0febc4240957b871e57c8c1e4d561f766f156a14ebfc3c877d"} Jul 06 15:47:23.972492 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:23.972434 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-7736d688-9d98b4667-2b5dh" event={"ID":"dc887352-4d3f-4a56-8d4d-3458ba383401","Type":"ContainerDied","Data":"0d47a3afb9fe9c9fccf87ea405bc412b6bc9875e5855c8cd927ffc73f15bc0e4"} Jul 06 15:47:23.972492 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:23.972465 2570 scope.go:117] "RemoveContainer" containerID="d9c5bb6400d40e0febc4240957b871e57c8c1e4d561f766f156a14ebfc3c877d" Jul 06 15:47:23.981532 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:23.981445 2570 scope.go:117] "RemoveContainer" containerID="d9c5bb6400d40e0febc4240957b871e57c8c1e4d561f766f156a14ebfc3c877d" Jul 06 15:47:23.981793 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:47:23.981768 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9c5bb6400d40e0febc4240957b871e57c8c1e4d561f766f156a14ebfc3c877d\": container with ID starting with d9c5bb6400d40e0febc4240957b871e57c8c1e4d561f766f156a14ebfc3c877d not found: ID does not exist" containerID="d9c5bb6400d40e0febc4240957b871e57c8c1e4d561f766f156a14ebfc3c877d" Jul 06 15:47:23.981888 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:23.981796 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9c5bb6400d40e0febc4240957b871e57c8c1e4d561f766f156a14ebfc3c877d"} err="failed to get container status \"d9c5bb6400d40e0febc4240957b871e57c8c1e4d561f766f156a14ebfc3c877d\": rpc error: code = NotFound desc = could not find container \"d9c5bb6400d40e0febc4240957b871e57c8c1e4d561f766f156a14ebfc3c877d\": container with ID starting with d9c5bb6400d40e0febc4240957b871e57c8c1e4d561f766f156a14ebfc3c877d not found: ID does not exist" Jul 06 15:47:24.008035 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:24.007997 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-7736d688-9d98b4667-2b5dh"] Jul 06 15:47:24.011090 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:24.011057 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-7736d688-9d98b4667-2b5dh"] Jul 06 15:47:24.800585 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:24.800556 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc887352-4d3f-4a56-8d4d-3458ba383401" path="/var/lib/kubelet/pods/dc887352-4d3f-4a56-8d4d-3458ba383401/volumes" Jul 06 15:47:25.152054 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:25.151328 2570 patch_prober.go:28] interesting pod/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": read tcp 10.134.0.2:53174->10.134.0.43:15021: read: connection reset by peer" start-of-body= Jul 06 15:47:25.152054 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:25.151405 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" podUID="5dfe8906-f271-4fd7-adca-60975b73cd8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": read tcp 10.134.0.2:53174->10.134.0.43:15021: read: connection reset by peer" Jul 06 15:47:25.984311 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:25.984277 2570 generic.go:356] "Generic (PLEG): container finished" podID="5dfe8906-f271-4fd7-adca-60975b73cd8a" containerID="60d813c44ab5c11c2c1ad010807008039a1580904e1bdea9939be6ef8e103a78" exitCode=0 Jul 06 15:47:25.984533 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:25.984342 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" event={"ID":"5dfe8906-f271-4fd7-adca-60975b73cd8a","Type":"ContainerDied","Data":"60d813c44ab5c11c2c1ad010807008039a1580904e1bdea9939be6ef8e103a78"} Jul 06 15:47:25.984837 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:25.984807 2570 scope.go:117] "RemoveContainer" containerID="60d813c44ab5c11c2c1ad010807008039a1580904e1bdea9939be6ef8e103a78" Jul 06 15:47:25.985876 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:25.985844 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:47:25.985962 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:25.985913 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:47:25.985962 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:25.985946 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:47:26.991740 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:26.991692 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" event={"ID":"5dfe8906-f271-4fd7-adca-60975b73cd8a","Type":"ContainerStarted","Data":"e863ebd9c1b671950d68e5e026d8a556af96961c88a48087788eb233d8f8a4d3"} Jul 06 15:47:27.216009 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:27.215969 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:27.216182 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:27.216142 2570 patch_prober.go:28] interesting pod/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 06 15:47:27.216227 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:27.216199 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" podUID="5dfe8906-f271-4fd7-adca-60975b73cd8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 06 15:47:28.215602 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:28.215567 2570 patch_prober.go:28] interesting pod/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 06 15:47:28.216002 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:28.215628 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" podUID="5dfe8906-f271-4fd7-adca-60975b73cd8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 06 15:47:29.215644 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:29.215612 2570 patch_prober.go:28] interesting pod/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 06 15:47:29.216035 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:29.215667 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" podUID="5dfe8906-f271-4fd7-adca-60975b73cd8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 06 15:47:29.492324 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:29.492233 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:29.593643 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:29.593602 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:29.796336 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:29.796247 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4"] Jul 06 15:47:29.796518 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:29.796464 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" podUID="5dfe8906-f271-4fd7-adca-60975b73cd8a" containerName="istio-proxy" containerID="cri-o://e863ebd9c1b671950d68e5e026d8a556af96961c88a48087788eb233d8f8a4d3" gracePeriod=30 Jul 06 15:47:29.820280 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:29.820253 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:30.070858 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:30.070768 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:30.213938 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:30.213897 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:30.391141 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:30.391100 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:30.528619 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:30.528588 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:34.763808 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:34.763774 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-mlw9w_91e0c855-5b32-4ca5-bab6-8e3aae20c1e0/console-operator/2.log" Jul 06 15:47:34.766713 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:34.766693 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-mlw9w_91e0c855-5b32-4ca5-bab6-8e3aae20c1e0/console-operator/2.log" Jul 06 15:47:34.767979 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:34.767957 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hxpw_3eca6180-d678-4729-ad6f-979a025af841/ovn-acl-logging/0.log" Jul 06 15:47:34.770603 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:34.770582 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hxpw_3eca6180-d678-4729-ad6f-979a025af841/ovn-acl-logging/0.log" Jul 06 15:47:36.036731 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.036705 2570 generic.go:356] "Generic (PLEG): container finished" podID="5dfe8906-f271-4fd7-adca-60975b73cd8a" containerID="e863ebd9c1b671950d68e5e026d8a556af96961c88a48087788eb233d8f8a4d3" exitCode=0 Jul 06 15:47:36.037115 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.036779 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" event={"ID":"5dfe8906-f271-4fd7-adca-60975b73cd8a","Type":"ContainerDied","Data":"e863ebd9c1b671950d68e5e026d8a556af96961c88a48087788eb233d8f8a4d3"} Jul 06 15:47:36.037115 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.036847 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" event={"ID":"5dfe8906-f271-4fd7-adca-60975b73cd8a","Type":"ContainerDied","Data":"c65f334055c3a9fbb416693302af25d3e7e5dccd3aff5dfb468d80bb5af6851d"} Jul 06 15:47:36.037115 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.036873 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c65f334055c3a9fbb416693302af25d3e7e5dccd3aff5dfb468d80bb5af6851d" Jul 06 15:47:36.037115 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.036891 2570 scope.go:117] "RemoveContainer" containerID="60d813c44ab5c11c2c1ad010807008039a1580904e1bdea9939be6ef8e103a78" Jul 06 15:47:36.050963 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.050946 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:36.145853 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.145773 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dfe8906-f271-4fd7-adca-60975b73cd8a-istio-envoy\") pod \"5dfe8906-f271-4fd7-adca-60975b73cd8a\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " Jul 06 15:47:36.145853 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.145814 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dfe8906-f271-4fd7-adca-60975b73cd8a-istiod-ca-cert\") pod \"5dfe8906-f271-4fd7-adca-60975b73cd8a\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " Jul 06 15:47:36.146067 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.145856 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dfe8906-f271-4fd7-adca-60975b73cd8a-istio-podinfo\") pod \"5dfe8906-f271-4fd7-adca-60975b73cd8a\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " Jul 06 15:47:36.146067 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.145883 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l69rs\" (UniqueName: \"kubernetes.io/projected/5dfe8906-f271-4fd7-adca-60975b73cd8a-kube-api-access-l69rs\") pod \"5dfe8906-f271-4fd7-adca-60975b73cd8a\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " Jul 06 15:47:36.146067 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.145920 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfe8906-f271-4fd7-adca-60975b73cd8a-credential-socket\") pod \"5dfe8906-f271-4fd7-adca-60975b73cd8a\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " Jul 06 15:47:36.146067 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.145944 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfe8906-f271-4fd7-adca-60975b73cd8a-workload-socket\") pod \"5dfe8906-f271-4fd7-adca-60975b73cd8a\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " Jul 06 15:47:36.146067 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.146013 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dfe8906-f271-4fd7-adca-60975b73cd8a-istio-token\") pod \"5dfe8906-f271-4fd7-adca-60975b73cd8a\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " Jul 06 15:47:36.146332 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.146099 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dfe8906-f271-4fd7-adca-60975b73cd8a-workload-certs\") pod \"5dfe8906-f271-4fd7-adca-60975b73cd8a\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " Jul 06 15:47:36.146332 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.146145 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dfe8906-f271-4fd7-adca-60975b73cd8a-istio-data\") pod \"5dfe8906-f271-4fd7-adca-60975b73cd8a\" (UID: \"5dfe8906-f271-4fd7-adca-60975b73cd8a\") " Jul 06 15:47:36.146332 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.146235 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dfe8906-f271-4fd7-adca-60975b73cd8a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5dfe8906-f271-4fd7-adca-60975b73cd8a" (UID: "5dfe8906-f271-4fd7-adca-60975b73cd8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:47:36.146332 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.146310 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dfe8906-f271-4fd7-adca-60975b73cd8a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5dfe8906-f271-4fd7-adca-60975b73cd8a" (UID: "5dfe8906-f271-4fd7-adca-60975b73cd8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:47:36.148331 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.146983 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dfe8906-f271-4fd7-adca-60975b73cd8a-istiod-ca-cert\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:47:36.148331 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.147011 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfe8906-f271-4fd7-adca-60975b73cd8a-workload-socket\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:47:36.148331 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.146982 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dfe8906-f271-4fd7-adca-60975b73cd8a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5dfe8906-f271-4fd7-adca-60975b73cd8a" (UID: "5dfe8906-f271-4fd7-adca-60975b73cd8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:47:36.148331 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.147157 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dfe8906-f271-4fd7-adca-60975b73cd8a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5dfe8906-f271-4fd7-adca-60975b73cd8a" (UID: "5dfe8906-f271-4fd7-adca-60975b73cd8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:47:36.148331 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.147689 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dfe8906-f271-4fd7-adca-60975b73cd8a-istio-data" (OuterVolumeSpecName: "istio-data") pod "5dfe8906-f271-4fd7-adca-60975b73cd8a" (UID: "5dfe8906-f271-4fd7-adca-60975b73cd8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:47:36.148731 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.148387 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dfe8906-f271-4fd7-adca-60975b73cd8a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5dfe8906-f271-4fd7-adca-60975b73cd8a" (UID: "5dfe8906-f271-4fd7-adca-60975b73cd8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:47:36.148835 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.148810 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5dfe8906-f271-4fd7-adca-60975b73cd8a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5dfe8906-f271-4fd7-adca-60975b73cd8a" (UID: "5dfe8906-f271-4fd7-adca-60975b73cd8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 15:47:36.155483 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.155460 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dfe8906-f271-4fd7-adca-60975b73cd8a-istio-token" (OuterVolumeSpecName: "istio-token") pod "5dfe8906-f271-4fd7-adca-60975b73cd8a" (UID: "5dfe8906-f271-4fd7-adca-60975b73cd8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:47:36.155615 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.155528 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dfe8906-f271-4fd7-adca-60975b73cd8a-kube-api-access-l69rs" (OuterVolumeSpecName: "kube-api-access-l69rs") pod "5dfe8906-f271-4fd7-adca-60975b73cd8a" (UID: "5dfe8906-f271-4fd7-adca-60975b73cd8a"). InnerVolumeSpecName "kube-api-access-l69rs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:47:36.248555 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.248495 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dfe8906-f271-4fd7-adca-60975b73cd8a-istio-data\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:47:36.248555 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.248549 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dfe8906-f271-4fd7-adca-60975b73cd8a-istio-envoy\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:47:36.248555 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.248558 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dfe8906-f271-4fd7-adca-60975b73cd8a-istio-podinfo\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:47:36.248783 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.248568 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l69rs\" (UniqueName: \"kubernetes.io/projected/5dfe8906-f271-4fd7-adca-60975b73cd8a-kube-api-access-l69rs\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:47:36.248783 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.248578 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfe8906-f271-4fd7-adca-60975b73cd8a-credential-socket\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:47:36.248783 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.248586 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dfe8906-f271-4fd7-adca-60975b73cd8a-istio-token\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:47:36.248783 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.248595 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dfe8906-f271-4fd7-adca-60975b73cd8a-workload-certs\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:47:36.633091 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.633055 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:36.774274 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.774240 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:36.891188 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:36.891090 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:37.029914 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:37.029879 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:37.042984 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:37.042959 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4" Jul 06 15:47:37.122388 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:37.122355 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4"] Jul 06 15:47:37.125623 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:37.125597 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7736d688-openshift-default-68cccfb567-6q6t4"] Jul 06 15:47:38.799015 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:38.798981 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dfe8906-f271-4fd7-adca-60975b73cd8a" path="/var/lib/kubelet/pods/5dfe8906-f271-4fd7-adca-60975b73cd8a/volumes" Jul 06 15:47:43.074721 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:43.074677 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:43.172845 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:43.172816 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:50.202700 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.202657 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:50.576538 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.576490 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf"] Jul 06 15:47:50.576915 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.576901 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dfe8906-f271-4fd7-adca-60975b73cd8a" containerName="istio-proxy" Jul 06 15:47:50.576963 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.576917 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dfe8906-f271-4fd7-adca-60975b73cd8a" containerName="istio-proxy" Jul 06 15:47:50.576963 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.576926 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dfe8906-f271-4fd7-adca-60975b73cd8a" containerName="istio-proxy" Jul 06 15:47:50.576963 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.576931 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dfe8906-f271-4fd7-adca-60975b73cd8a" containerName="istio-proxy" Jul 06 15:47:50.576963 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.576939 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc887352-4d3f-4a56-8d4d-3458ba383401" containerName="maas-api" Jul 06 15:47:50.576963 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.576944 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc887352-4d3f-4a56-8d4d-3458ba383401" containerName="maas-api" Jul 06 15:47:50.577117 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.576999 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dfe8906-f271-4fd7-adca-60975b73cd8a" containerName="istio-proxy" Jul 06 15:47:50.577117 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.577009 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc887352-4d3f-4a56-8d4d-3458ba383401" containerName="maas-api" Jul 06 15:47:50.581720 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.581697 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:47:50.585358 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.585339 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6bb1764c-openshift-default-dockercfg-wj7ld\"" Jul 06 15:47:50.597377 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.597347 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf"] Jul 06 15:47:50.602369 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.602345 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:50.689128 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.689080 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-istiod-ca-cert\") pod \"e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:47:50.689128 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.689135 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-istio-token\") pod \"e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:47:50.689417 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.689240 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-workload-certs\") pod \"e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:47:50.689417 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.689296 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-istio-data\") pod \"e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:47:50.689417 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.689326 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-istio-envoy\") pod \"e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:47:50.689417 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.689399 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2vtv\" (UniqueName: \"kubernetes.io/projected/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-kube-api-access-d2vtv\") pod \"e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:47:50.689633 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.689464 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-workload-socket\") pod \"e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:47:50.689633 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.689544 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-credential-socket\") pod \"e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:47:50.689633 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.689567 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-istio-podinfo\") pod \"e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:47:50.790863 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.790816 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-istiod-ca-cert\") pod \"e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:47:50.790863 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.790872 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-istio-token\") pod \"e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:47:50.791141 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.790908 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-workload-certs\") pod \"e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:47:50.791141 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.790933 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-istio-data\") pod \"e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:47:50.791141 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.790950 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-istio-envoy\") pod \"e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:47:50.791141 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.790988 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2vtv\" (UniqueName: \"kubernetes.io/projected/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-kube-api-access-d2vtv\") pod \"e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:47:50.791141 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.791048 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-workload-socket\") pod \"e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:47:50.791141 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.791089 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-credential-socket\") pod \"e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:47:50.791141 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.791114 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-istio-podinfo\") pod \"e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:47:50.791527 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.791489 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-workload-certs\") pod \"e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:47:50.791598 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.791531 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-istio-data\") pod \"e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:47:50.791598 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.791540 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-credential-socket\") pod \"e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:47:50.791705 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.791594 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-workload-socket\") pod \"e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:47:50.791705 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.791664 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-istiod-ca-cert\") pod \"e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:47:50.793473 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.793444 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-istio-envoy\") pod \"e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:47:50.793783 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.793764 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-istio-podinfo\") pod \"e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:47:50.808265 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.808239 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-istio-token\") pod \"e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:47:50.808480 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.808457 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2vtv\" (UniqueName: \"kubernetes.io/projected/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-kube-api-access-d2vtv\") pod \"e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:47:50.833704 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.833578 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:50.891944 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:50.891902 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:47:51.047345 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:51.047320 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf"] Jul 06 15:47:51.050084 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:47:51.050056 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50ca9494_74db_4dd0_bc3b_d45e5fab17b1.slice/crio-693158d96c1b638bbf9679a84873b556b14e6a0b3f07a78488698bd279de8c54 WatchSource:0}: Error finding container 693158d96c1b638bbf9679a84873b556b14e6a0b3f07a78488698bd279de8c54: Status 404 returned error can't find the container with id 693158d96c1b638bbf9679a84873b556b14e6a0b3f07a78488698bd279de8c54 Jul 06 15:47:51.052214 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:51.052179 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:47:51.052281 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:51.052256 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:47:51.052318 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:51.052307 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:47:51.109380 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:51.109351 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" event={"ID":"50ca9494-74db-4dd0-bc3b-d45e5fab17b1","Type":"ContainerStarted","Data":"693158d96c1b638bbf9679a84873b556b14e6a0b3f07a78488698bd279de8c54"} Jul 06 15:47:52.116079 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:52.116038 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" event={"ID":"50ca9494-74db-4dd0-bc3b-d45e5fab17b1","Type":"ContainerStarted","Data":"ce58da4053da0b52e5da520d047dffc2bec3615149e995c09bf5154fed290f73"} Jul 06 15:47:52.144227 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:52.144166 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" podStartSLOduration=2.14414501 podStartE2EDuration="2.14414501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:47:52.14356748 +0000 UTC m=+1818.757316017" watchObservedRunningTime="2026-07-06 15:47:52.14414501 +0000 UTC m=+1818.757893553" Jul 06 15:47:52.892670 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:52.892623 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:47:52.897742 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:52.897719 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:47:53.121535 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:53.121493 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:47:53.122418 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:53.122398 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:47:54.239291 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:54.239260 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:54.338262 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:54.338220 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:54.410866 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:54.410836 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8"] Jul 06 15:47:54.411303 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:54.411290 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dfe8906-f271-4fd7-adca-60975b73cd8a" containerName="istio-proxy" Jul 06 15:47:54.414464 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:54.414447 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8" Jul 06 15:47:54.419139 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:54.419119 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-6bb1764c-serving-cert\"" Jul 06 15:47:54.431061 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:54.431037 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8"] Jul 06 15:47:54.464704 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:54.464670 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:54.528248 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:54.528155 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b52eeda2-cce7-45d0-9ebe-40f030e061db-maas-api-tls\") pod \"maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8\" (UID: \"b52eeda2-cce7-45d0-9ebe-40f030e061db\") " pod="opendatahub/maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8" Jul 06 15:47:54.528248 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:54.528194 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b52eeda2-cce7-45d0-9ebe-40f030e061db-openshift-service-ca\") pod \"maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8\" (UID: \"b52eeda2-cce7-45d0-9ebe-40f030e061db\") " pod="opendatahub/maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8" Jul 06 15:47:54.528535 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:54.528336 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgbtb\" (UniqueName: \"kubernetes.io/projected/b52eeda2-cce7-45d0-9ebe-40f030e061db-kube-api-access-vgbtb\") pod \"maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8\" (UID: \"b52eeda2-cce7-45d0-9ebe-40f030e061db\") " pod="opendatahub/maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8" Jul 06 15:47:54.629486 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:54.629450 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b52eeda2-cce7-45d0-9ebe-40f030e061db-maas-api-tls\") pod \"maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8\" (UID: \"b52eeda2-cce7-45d0-9ebe-40f030e061db\") " pod="opendatahub/maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8" Jul 06 15:47:54.629682 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:54.629488 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b52eeda2-cce7-45d0-9ebe-40f030e061db-openshift-service-ca\") pod \"maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8\" (UID: \"b52eeda2-cce7-45d0-9ebe-40f030e061db\") " pod="opendatahub/maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8" Jul 06 15:47:54.629682 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:54.629585 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgbtb\" (UniqueName: \"kubernetes.io/projected/b52eeda2-cce7-45d0-9ebe-40f030e061db-kube-api-access-vgbtb\") pod \"maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8\" (UID: \"b52eeda2-cce7-45d0-9ebe-40f030e061db\") " pod="opendatahub/maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8" Jul 06 15:47:54.630235 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:54.630214 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b52eeda2-cce7-45d0-9ebe-40f030e061db-openshift-service-ca\") pod \"maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8\" (UID: \"b52eeda2-cce7-45d0-9ebe-40f030e061db\") " pod="opendatahub/maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8" Jul 06 15:47:54.631935 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:54.631913 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b52eeda2-cce7-45d0-9ebe-40f030e061db-maas-api-tls\") pod \"maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8\" (UID: \"b52eeda2-cce7-45d0-9ebe-40f030e061db\") " pod="opendatahub/maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8" Jul 06 15:47:54.643969 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:54.643939 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgbtb\" (UniqueName: \"kubernetes.io/projected/b52eeda2-cce7-45d0-9ebe-40f030e061db-kube-api-access-vgbtb\") pod \"maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8\" (UID: \"b52eeda2-cce7-45d0-9ebe-40f030e061db\") " pod="opendatahub/maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8" Jul 06 15:47:54.725807 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:54.725762 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8" Jul 06 15:47:54.907958 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:54.907927 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8"] Jul 06 15:47:54.908108 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:47:54.908089 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb52eeda2_cce7_45d0_9ebe_40f030e061db.slice/crio-9ac28cdb57d349d9c29efe3d895d3512d99cda6ad4dd8ee2bdc6cf38de8e60b7 WatchSource:0}: Error finding container 9ac28cdb57d349d9c29efe3d895d3512d99cda6ad4dd8ee2bdc6cf38de8e60b7: Status 404 returned error can't find the container with id 9ac28cdb57d349d9c29efe3d895d3512d99cda6ad4dd8ee2bdc6cf38de8e60b7 Jul 06 15:47:55.133273 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:55.133237 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8" event={"ID":"b52eeda2-cce7-45d0-9ebe-40f030e061db","Type":"ContainerStarted","Data":"9ac28cdb57d349d9c29efe3d895d3512d99cda6ad4dd8ee2bdc6cf38de8e60b7"} Jul 06 15:47:56.139672 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:56.139633 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8" event={"ID":"b52eeda2-cce7-45d0-9ebe-40f030e061db","Type":"ContainerStarted","Data":"cd8c3473dd346179bdee9db539bc361b5d404515ca76ba3eb9cd1952bf5f1835"} Jul 06 15:47:56.140065 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:56.139698 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8" Jul 06 15:47:56.171454 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:56.171405 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8" podStartSLOduration=1.71177395 podStartE2EDuration="2.171388417s" totalImagesPullingTime="459.614467ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:47:56.171185461 +0000 UTC m=+1822.784934008" watchObservedRunningTime="2026-07-06 15:47:56.171388417 +0000 UTC m=+1822.785136952" Jul 06 15:47:56.185296 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:56.185275 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:56.290043 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:56.290011 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:58.759117 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:47:58.759080 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:48:02.155418 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:02.155377 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8" Jul 06 15:48:06.828239 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:06.828202 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:48:06.928244 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:06.928210 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:48:08.860945 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:08.860490 2570 patch_prober.go:28] interesting pod/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": read tcp 10.134.0.2:50650->10.134.0.45:15021: read: connection reset by peer" start-of-body= Jul 06 15:48:08.860945 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:08.860579 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" podUID="50ca9494-74db-4dd0-bc3b-d45e5fab17b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": read tcp 10.134.0.2:50650->10.134.0.45:15021: read: connection reset by peer" Jul 06 15:48:09.206223 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:09.206124 2570 generic.go:356] "Generic (PLEG): container finished" podID="50ca9494-74db-4dd0-bc3b-d45e5fab17b1" containerID="ce58da4053da0b52e5da520d047dffc2bec3615149e995c09bf5154fed290f73" exitCode=0 Jul 06 15:48:09.206223 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:09.206196 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" event={"ID":"50ca9494-74db-4dd0-bc3b-d45e5fab17b1","Type":"ContainerDied","Data":"ce58da4053da0b52e5da520d047dffc2bec3615149e995c09bf5154fed290f73"} Jul 06 15:48:09.206627 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:09.206613 2570 scope.go:117] "RemoveContainer" containerID="ce58da4053da0b52e5da520d047dffc2bec3615149e995c09bf5154fed290f73" Jul 06 15:48:09.207594 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:09.207569 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:48:09.207663 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:09.207631 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:48:09.207707 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:09.207660 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:48:10.213170 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:10.213134 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" event={"ID":"50ca9494-74db-4dd0-bc3b-d45e5fab17b1","Type":"ContainerStarted","Data":"0c1c5b57963dd9794b8b2923e3db9ce4a0132a427d37288ed2655e2d325c7db9"} Jul 06 15:48:10.892855 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:10.892813 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:48:10.893052 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:10.892991 2570 patch_prober.go:28] interesting pod/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 06 15:48:10.893052 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:10.893038 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" podUID="50ca9494-74db-4dd0-bc3b-d45e5fab17b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 06 15:48:11.892417 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:11.892382 2570 patch_prober.go:28] interesting pod/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 06 15:48:11.892863 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:11.892434 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" podUID="50ca9494-74db-4dd0-bc3b-d45e5fab17b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 06 15:48:12.893313 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:12.893274 2570 patch_prober.go:28] interesting pod/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 06 15:48:12.893754 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:12.893327 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" podUID="50ca9494-74db-4dd0-bc3b-d45e5fab17b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 06 15:48:13.892878 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:13.892841 2570 patch_prober.go:28] interesting pod/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 06 15:48:13.893063 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:13.892895 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" podUID="50ca9494-74db-4dd0-bc3b-d45e5fab17b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 06 15:48:14.896164 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:14.896130 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:48:14.896662 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:14.896517 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:48:14.897064 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:14.897044 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:48:16.928090 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:16.928058 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:48:17.039994 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:17.039956 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:48:19.043200 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:19.043172 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:48:41.549602 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:41.549561 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:48:41.923918 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:41.923885 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8"] Jul 06 15:48:41.924185 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:41.924159 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8" podUID="b52eeda2-cce7-45d0-9ebe-40f030e061db" containerName="maas-api" containerID="cri-o://cd8c3473dd346179bdee9db539bc361b5d404515ca76ba3eb9cd1952bf5f1835" gracePeriod=30 Jul 06 15:48:41.957217 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:41.957193 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:48:42.172002 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:42.171978 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8" Jul 06 15:48:42.203887 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:42.203796 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b52eeda2-cce7-45d0-9ebe-40f030e061db-maas-api-tls\") pod \"b52eeda2-cce7-45d0-9ebe-40f030e061db\" (UID: \"b52eeda2-cce7-45d0-9ebe-40f030e061db\") " Jul 06 15:48:42.203887 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:42.203872 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgbtb\" (UniqueName: \"kubernetes.io/projected/b52eeda2-cce7-45d0-9ebe-40f030e061db-kube-api-access-vgbtb\") pod \"b52eeda2-cce7-45d0-9ebe-40f030e061db\" (UID: \"b52eeda2-cce7-45d0-9ebe-40f030e061db\") " Jul 06 15:48:42.204114 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:42.203908 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b52eeda2-cce7-45d0-9ebe-40f030e061db-openshift-service-ca\") pod \"b52eeda2-cce7-45d0-9ebe-40f030e061db\" (UID: \"b52eeda2-cce7-45d0-9ebe-40f030e061db\") " Jul 06 15:48:42.204291 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:42.204267 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b52eeda2-cce7-45d0-9ebe-40f030e061db-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b52eeda2-cce7-45d0-9ebe-40f030e061db" (UID: "b52eeda2-cce7-45d0-9ebe-40f030e061db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:48:42.206022 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:42.205998 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b52eeda2-cce7-45d0-9ebe-40f030e061db-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b52eeda2-cce7-45d0-9ebe-40f030e061db" (UID: "b52eeda2-cce7-45d0-9ebe-40f030e061db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:48:42.206392 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:42.206367 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b52eeda2-cce7-45d0-9ebe-40f030e061db-kube-api-access-vgbtb" (OuterVolumeSpecName: "kube-api-access-vgbtb") pod "b52eeda2-cce7-45d0-9ebe-40f030e061db" (UID: "b52eeda2-cce7-45d0-9ebe-40f030e061db"). InnerVolumeSpecName "kube-api-access-vgbtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:48:42.305554 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:42.305490 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b52eeda2-cce7-45d0-9ebe-40f030e061db-maas-api-tls\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:48:42.305554 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:42.305548 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgbtb\" (UniqueName: \"kubernetes.io/projected/b52eeda2-cce7-45d0-9ebe-40f030e061db-kube-api-access-vgbtb\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:48:42.305554 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:42.305561 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b52eeda2-cce7-45d0-9ebe-40f030e061db-openshift-service-ca\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:48:42.372298 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:42.372259 2570 generic.go:356] "Generic (PLEG): container finished" podID="b52eeda2-cce7-45d0-9ebe-40f030e061db" containerID="cd8c3473dd346179bdee9db539bc361b5d404515ca76ba3eb9cd1952bf5f1835" exitCode=0 Jul 06 15:48:42.372298 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:42.372301 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8" Jul 06 15:48:42.372539 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:42.372320 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8" event={"ID":"b52eeda2-cce7-45d0-9ebe-40f030e061db","Type":"ContainerDied","Data":"cd8c3473dd346179bdee9db539bc361b5d404515ca76ba3eb9cd1952bf5f1835"} Jul 06 15:48:42.372539 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:42.372368 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8" event={"ID":"b52eeda2-cce7-45d0-9ebe-40f030e061db","Type":"ContainerDied","Data":"9ac28cdb57d349d9c29efe3d895d3512d99cda6ad4dd8ee2bdc6cf38de8e60b7"} Jul 06 15:48:42.372539 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:42.372393 2570 scope.go:117] "RemoveContainer" containerID="cd8c3473dd346179bdee9db539bc361b5d404515ca76ba3eb9cd1952bf5f1835" Jul 06 15:48:42.381799 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:42.381780 2570 scope.go:117] "RemoveContainer" containerID="cd8c3473dd346179bdee9db539bc361b5d404515ca76ba3eb9cd1952bf5f1835" Jul 06 15:48:42.382075 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:48:42.382051 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd8c3473dd346179bdee9db539bc361b5d404515ca76ba3eb9cd1952bf5f1835\": container with ID starting with cd8c3473dd346179bdee9db539bc361b5d404515ca76ba3eb9cd1952bf5f1835 not found: ID does not exist" containerID="cd8c3473dd346179bdee9db539bc361b5d404515ca76ba3eb9cd1952bf5f1835" Jul 06 15:48:42.382131 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:42.382095 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd8c3473dd346179bdee9db539bc361b5d404515ca76ba3eb9cd1952bf5f1835"} err="failed to get container status \"cd8c3473dd346179bdee9db539bc361b5d404515ca76ba3eb9cd1952bf5f1835\": rpc error: code = NotFound desc = could not find container \"cd8c3473dd346179bdee9db539bc361b5d404515ca76ba3eb9cd1952bf5f1835\": container with ID starting with cd8c3473dd346179bdee9db539bc361b5d404515ca76ba3eb9cd1952bf5f1835 not found: ID does not exist" Jul 06 15:48:42.396438 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:42.396410 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8"] Jul 06 15:48:42.400343 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:42.400320 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8"] Jul 06 15:48:42.799539 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:42.799484 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b52eeda2-cce7-45d0-9ebe-40f030e061db" path="/var/lib/kubelet/pods/b52eeda2-cce7-45d0-9ebe-40f030e061db/volumes" Jul 06 15:48:45.146164 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:45.146100 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-api-e2e-mt-6bb1764c-547978b96c-5xcm8" podUID="b52eeda2-cce7-45d0-9ebe-40f030e061db" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 06 15:48:47.713107 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:47.713056 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:48:52.763571 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:48:52.763531 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:49:29.485837 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:29.485757 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:49:30.852883 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:30.852856 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:49:36.977990 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:36.977957 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:49:42.754175 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:42.754143 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:49:42.838021 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:42.837988 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:49:43.045932 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:43.045843 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf"] Jul 06 15:49:43.046138 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:43.046098 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" podUID="50ca9494-74db-4dd0-bc3b-d45e5fab17b1" containerName="istio-proxy" containerID="cri-o://0c1c5b57963dd9794b8b2923e3db9ce4a0132a427d37288ed2655e2d325c7db9" gracePeriod=30 Jul 06 15:49:43.066598 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:43.066568 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:49:43.377664 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:43.377629 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:49:44.525992 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:44.525939 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:49:44.617754 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:44.617718 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:49:44.717796 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:44.717771 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:49:44.898664 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:44.898626 2570 patch_prober.go:28] interesting pod/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 15:49:44.898840 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:44.898692 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" podUID="50ca9494-74db-4dd0-bc3b-d45e5fab17b1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 15:49:45.503148 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:45.503109 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:49:45.593210 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:45.593175 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:49:45.697300 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:45.697268 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:49:47.342306 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:47.342270 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:49:48.297596 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.297573 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:49:48.347419 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.347383 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-workload-certs\") pod \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " Jul 06 15:49:48.347950 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.347439 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-workload-socket\") pod \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " Jul 06 15:49:48.347950 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.347470 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-istiod-ca-cert\") pod \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " Jul 06 15:49:48.347950 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.347550 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-credential-socket\") pod \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " Jul 06 15:49:48.347950 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.347591 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-istio-data\") pod \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " Jul 06 15:49:48.347950 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.347620 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-istio-token\") pod \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " Jul 06 15:49:48.347950 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.347643 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-istio-podinfo\") pod \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " Jul 06 15:49:48.347950 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.347730 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "50ca9494-74db-4dd0-bc3b-d45e5fab17b1" (UID: "50ca9494-74db-4dd0-bc3b-d45e5fab17b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:49:48.347950 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.347740 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-istio-envoy\") pod \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " Jul 06 15:49:48.347950 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.347787 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "50ca9494-74db-4dd0-bc3b-d45e5fab17b1" (UID: "50ca9494-74db-4dd0-bc3b-d45e5fab17b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:49:48.347950 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.347801 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2vtv\" (UniqueName: \"kubernetes.io/projected/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-kube-api-access-d2vtv\") pod \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\" (UID: \"50ca9494-74db-4dd0-bc3b-d45e5fab17b1\") " Jul 06 15:49:48.348481 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.348003 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "50ca9494-74db-4dd0-bc3b-d45e5fab17b1" (UID: "50ca9494-74db-4dd0-bc3b-d45e5fab17b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:49:48.348481 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.348244 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-workload-socket\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:49:48.348481 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.348264 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-istiod-ca-cert\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:49:48.348481 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.348278 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-workload-certs\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:49:48.348775 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.348739 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-istio-data" (OuterVolumeSpecName: "istio-data") pod "50ca9494-74db-4dd0-bc3b-d45e5fab17b1" (UID: "50ca9494-74db-4dd0-bc3b-d45e5fab17b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:49:48.348904 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.348788 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "50ca9494-74db-4dd0-bc3b-d45e5fab17b1" (UID: "50ca9494-74db-4dd0-bc3b-d45e5fab17b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:49:48.350374 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.350347 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "50ca9494-74db-4dd0-bc3b-d45e5fab17b1" (UID: "50ca9494-74db-4dd0-bc3b-d45e5fab17b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:49:48.350374 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.350363 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-kube-api-access-d2vtv" (OuterVolumeSpecName: "kube-api-access-d2vtv") pod "50ca9494-74db-4dd0-bc3b-d45e5fab17b1" (UID: "50ca9494-74db-4dd0-bc3b-d45e5fab17b1"). InnerVolumeSpecName "kube-api-access-d2vtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:49:48.350668 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.350442 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "50ca9494-74db-4dd0-bc3b-d45e5fab17b1" (UID: "50ca9494-74db-4dd0-bc3b-d45e5fab17b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 15:49:48.350668 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.350528 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-istio-token" (OuterVolumeSpecName: "istio-token") pod "50ca9494-74db-4dd0-bc3b-d45e5fab17b1" (UID: "50ca9494-74db-4dd0-bc3b-d45e5fab17b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:49:48.449537 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.449481 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-credential-socket\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:49:48.449537 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.449532 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-istio-data\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:49:48.449537 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.449547 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-istio-token\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:49:48.449781 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.449561 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-istio-podinfo\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:49:48.449781 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.449572 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-istio-envoy\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:49:48.449781 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.449584 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2vtv\" (UniqueName: \"kubernetes.io/projected/50ca9494-74db-4dd0-bc3b-d45e5fab17b1-kube-api-access-d2vtv\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:49:48.699887 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.699788 2570 generic.go:356] "Generic (PLEG): container finished" podID="50ca9494-74db-4dd0-bc3b-d45e5fab17b1" containerID="0c1c5b57963dd9794b8b2923e3db9ce4a0132a427d37288ed2655e2d325c7db9" exitCode=0 Jul 06 15:49:48.699887 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.699842 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" Jul 06 15:49:48.699887 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.699871 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" event={"ID":"50ca9494-74db-4dd0-bc3b-d45e5fab17b1","Type":"ContainerDied","Data":"0c1c5b57963dd9794b8b2923e3db9ce4a0132a427d37288ed2655e2d325c7db9"} Jul 06 15:49:48.700153 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.699925 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf" event={"ID":"50ca9494-74db-4dd0-bc3b-d45e5fab17b1","Type":"ContainerDied","Data":"693158d96c1b638bbf9679a84873b556b14e6a0b3f07a78488698bd279de8c54"} Jul 06 15:49:48.700153 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.699947 2570 scope.go:117] "RemoveContainer" containerID="0c1c5b57963dd9794b8b2923e3db9ce4a0132a427d37288ed2655e2d325c7db9" Jul 06 15:49:48.712340 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.712323 2570 scope.go:117] "RemoveContainer" containerID="ce58da4053da0b52e5da520d047dffc2bec3615149e995c09bf5154fed290f73" Jul 06 15:49:48.720546 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.720525 2570 scope.go:117] "RemoveContainer" containerID="0c1c5b57963dd9794b8b2923e3db9ce4a0132a427d37288ed2655e2d325c7db9" Jul 06 15:49:48.720796 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:49:48.720775 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c1c5b57963dd9794b8b2923e3db9ce4a0132a427d37288ed2655e2d325c7db9\": container with ID starting with 0c1c5b57963dd9794b8b2923e3db9ce4a0132a427d37288ed2655e2d325c7db9 not found: ID does not exist" containerID="0c1c5b57963dd9794b8b2923e3db9ce4a0132a427d37288ed2655e2d325c7db9" Jul 06 15:49:48.720855 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.720806 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c1c5b57963dd9794b8b2923e3db9ce4a0132a427d37288ed2655e2d325c7db9"} err="failed to get container status \"0c1c5b57963dd9794b8b2923e3db9ce4a0132a427d37288ed2655e2d325c7db9\": rpc error: code = NotFound desc = could not find container \"0c1c5b57963dd9794b8b2923e3db9ce4a0132a427d37288ed2655e2d325c7db9\": container with ID starting with 0c1c5b57963dd9794b8b2923e3db9ce4a0132a427d37288ed2655e2d325c7db9 not found: ID does not exist" Jul 06 15:49:48.720855 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.720824 2570 scope.go:117] "RemoveContainer" containerID="ce58da4053da0b52e5da520d047dffc2bec3615149e995c09bf5154fed290f73" Jul 06 15:49:48.721023 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:49:48.721007 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce58da4053da0b52e5da520d047dffc2bec3615149e995c09bf5154fed290f73\": container with ID starting with ce58da4053da0b52e5da520d047dffc2bec3615149e995c09bf5154fed290f73 not found: ID does not exist" containerID="ce58da4053da0b52e5da520d047dffc2bec3615149e995c09bf5154fed290f73" Jul 06 15:49:48.721063 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.721027 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce58da4053da0b52e5da520d047dffc2bec3615149e995c09bf5154fed290f73"} err="failed to get container status \"ce58da4053da0b52e5da520d047dffc2bec3615149e995c09bf5154fed290f73\": rpc error: code = NotFound desc = could not find container \"ce58da4053da0b52e5da520d047dffc2bec3615149e995c09bf5154fed290f73\": container with ID starting with ce58da4053da0b52e5da520d047dffc2bec3615149e995c09bf5154fed290f73 not found: ID does not exist" Jul 06 15:49:48.731749 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.731722 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf"] Jul 06 15:49:48.757230 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.757187 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6bb1764c-openshift-default-75b7d6f46c-lfmpf"] Jul 06 15:49:48.799045 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:48.799017 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50ca9494-74db-4dd0-bc3b-d45e5fab17b1" path="/var/lib/kubelet/pods/50ca9494-74db-4dd0-bc3b-d45e5fab17b1/volumes" Jul 06 15:49:53.414453 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:53.414421 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:49:53.491531 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:53.491470 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:49:54.093425 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:49:54.093387 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:00.194538 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:00.194488 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:00.309225 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:00.309192 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:00.426665 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:00.426633 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:10.933841 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:10.933799 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:11.362476 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:11.362448 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:27.144584 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:27.144462 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:32.336447 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:32.336416 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:39.846925 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:39.846891 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:40.612043 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.612008 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv"] Jul 06 15:50:40.612414 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.612402 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50ca9494-74db-4dd0-bc3b-d45e5fab17b1" containerName="istio-proxy" Jul 06 15:50:40.612461 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.612415 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="50ca9494-74db-4dd0-bc3b-d45e5fab17b1" containerName="istio-proxy" Jul 06 15:50:40.612461 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.612433 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b52eeda2-cce7-45d0-9ebe-40f030e061db" containerName="maas-api" Jul 06 15:50:40.612461 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.612439 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b52eeda2-cce7-45d0-9ebe-40f030e061db" containerName="maas-api" Jul 06 15:50:40.612582 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.612515 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="50ca9494-74db-4dd0-bc3b-d45e5fab17b1" containerName="istio-proxy" Jul 06 15:50:40.612582 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.612524 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b52eeda2-cce7-45d0-9ebe-40f030e061db" containerName="maas-api" Jul 06 15:50:40.614651 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.614635 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:50:40.617959 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.617937 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7a30a8-openshift-default-dockercfg-5dbrg\"" Jul 06 15:50:40.649829 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.649788 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv"] Jul 06 15:50:40.734701 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.734661 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21ae110e-062d-4af2-9938-329252177d7f-workload-certs\") pod \"e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:50:40.734895 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.734715 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21ae110e-062d-4af2-9938-329252177d7f-istio-data\") pod \"e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:50:40.734895 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.734784 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21ae110e-062d-4af2-9938-329252177d7f-istio-token\") pod \"e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:50:40.734895 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.734832 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ae110e-062d-4af2-9938-329252177d7f-credential-socket\") pod \"e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:50:40.734895 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.734856 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9skc9\" (UniqueName: \"kubernetes.io/projected/21ae110e-062d-4af2-9938-329252177d7f-kube-api-access-9skc9\") pod \"e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:50:40.735110 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.734932 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21ae110e-062d-4af2-9938-329252177d7f-istiod-ca-cert\") pod \"e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:50:40.735110 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.734963 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ae110e-062d-4af2-9938-329252177d7f-workload-socket\") pod \"e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:50:40.735110 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.734994 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21ae110e-062d-4af2-9938-329252177d7f-istio-envoy\") pod \"e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:50:40.735110 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.735058 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21ae110e-062d-4af2-9938-329252177d7f-istio-podinfo\") pod \"e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:50:40.836510 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.836461 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21ae110e-062d-4af2-9938-329252177d7f-workload-certs\") pod \"e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:50:40.836510 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.836517 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21ae110e-062d-4af2-9938-329252177d7f-istio-data\") pod \"e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:50:40.836767 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.836543 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21ae110e-062d-4af2-9938-329252177d7f-istio-token\") pod \"e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:50:40.836767 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.836564 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ae110e-062d-4af2-9938-329252177d7f-credential-socket\") pod \"e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:50:40.836767 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.836580 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9skc9\" (UniqueName: \"kubernetes.io/projected/21ae110e-062d-4af2-9938-329252177d7f-kube-api-access-9skc9\") pod \"e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:50:40.836767 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.836622 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21ae110e-062d-4af2-9938-329252177d7f-istiod-ca-cert\") pod \"e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:50:40.836767 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.836652 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ae110e-062d-4af2-9938-329252177d7f-workload-socket\") pod \"e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:50:40.836767 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.836680 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21ae110e-062d-4af2-9938-329252177d7f-istio-envoy\") pod \"e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:50:40.836767 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.836737 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21ae110e-062d-4af2-9938-329252177d7f-istio-podinfo\") pod \"e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:50:40.837128 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.836936 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21ae110e-062d-4af2-9938-329252177d7f-workload-certs\") pod \"e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:50:40.837128 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.837062 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21ae110e-062d-4af2-9938-329252177d7f-istio-data\") pod \"e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:50:40.837128 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.837097 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ae110e-062d-4af2-9938-329252177d7f-credential-socket\") pod \"e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:50:40.837294 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.837154 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ae110e-062d-4af2-9938-329252177d7f-workload-socket\") pod \"e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:50:40.837603 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.837574 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21ae110e-062d-4af2-9938-329252177d7f-istiod-ca-cert\") pod \"e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:50:40.839006 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.838983 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21ae110e-062d-4af2-9938-329252177d7f-istio-envoy\") pod \"e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:50:40.839615 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.839594 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21ae110e-062d-4af2-9938-329252177d7f-istio-podinfo\") pod \"e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:50:40.849240 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.849220 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21ae110e-062d-4af2-9938-329252177d7f-istio-token\") pod \"e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:50:40.854924 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.854898 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9skc9\" (UniqueName: \"kubernetes.io/projected/21ae110e-062d-4af2-9938-329252177d7f-kube-api-access-9skc9\") pod \"e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:50:40.924320 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:40.924228 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:50:41.084167 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:41.084143 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv"] Jul 06 15:50:41.086284 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:50:41.086259 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ae110e_062d_4af2_9938_329252177d7f.slice/crio-59548f38bf7f89d65454df843cddc9718196a7dfc90240e279b16171bd9b2709 WatchSource:0}: Error finding container 59548f38bf7f89d65454df843cddc9718196a7dfc90240e279b16171bd9b2709: Status 404 returned error can't find the container with id 59548f38bf7f89d65454df843cddc9718196a7dfc90240e279b16171bd9b2709 Jul 06 15:50:41.088417 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:41.088383 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:50:41.088484 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:41.088445 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:50:41.088484 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:41.088472 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:50:41.975246 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:41.975213 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" event={"ID":"21ae110e-062d-4af2-9938-329252177d7f","Type":"ContainerStarted","Data":"418c65a0e464389365982b4ca2dbc4ac77cf185b263349c7169c5d54def7913c"} Jul 06 15:50:41.975644 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:41.975259 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" event={"ID":"21ae110e-062d-4af2-9938-329252177d7f","Type":"ContainerStarted","Data":"59548f38bf7f89d65454df843cddc9718196a7dfc90240e279b16171bd9b2709"} Jul 06 15:50:42.005283 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:42.005207 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" podStartSLOduration=2.005191828 podStartE2EDuration="2.005191828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:50:42.00424536 +0000 UTC m=+1988.617993897" watchObservedRunningTime="2026-07-06 15:50:42.005191828 +0000 UTC m=+1988.618940364" Jul 06 15:50:42.714416 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:42.714377 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:42.925002 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:42.924961 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:50:42.930372 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:42.930345 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:50:42.979857 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:42.979779 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:50:42.980678 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:42.980660 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:50:44.137923 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:44.137888 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:44.215308 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:44.215277 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx"] Jul 06 15:50:44.215772 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:44.215754 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50ca9494-74db-4dd0-bc3b-d45e5fab17b1" containerName="istio-proxy" Jul 06 15:50:44.215873 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:44.215773 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="50ca9494-74db-4dd0-bc3b-d45e5fab17b1" containerName="istio-proxy" Jul 06 15:50:44.215873 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:44.215862 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="50ca9494-74db-4dd0-bc3b-d45e5fab17b1" containerName="istio-proxy" Jul 06 15:50:44.217980 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:44.217959 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx" Jul 06 15:50:44.222738 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:44.222718 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-7a30a8-serving-cert\"" Jul 06 15:50:44.238387 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:44.238361 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx"] Jul 06 15:50:44.271250 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:44.271206 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xpmr\" (UniqueName: \"kubernetes.io/projected/d4872501-9b29-47b6-8d7c-7692848c43be-kube-api-access-5xpmr\") pod \"maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx\" (UID: \"d4872501-9b29-47b6-8d7c-7692848c43be\") " pod="opendatahub/maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx" Jul 06 15:50:44.271403 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:44.271332 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4872501-9b29-47b6-8d7c-7692848c43be-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx\" (UID: \"d4872501-9b29-47b6-8d7c-7692848c43be\") " pod="opendatahub/maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx" Jul 06 15:50:44.271403 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:44.271377 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4872501-9b29-47b6-8d7c-7692848c43be-maas-api-tls\") pod \"maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx\" (UID: \"d4872501-9b29-47b6-8d7c-7692848c43be\") " pod="opendatahub/maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx" Jul 06 15:50:44.278236 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:44.278213 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:44.372313 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:44.372277 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xpmr\" (UniqueName: \"kubernetes.io/projected/d4872501-9b29-47b6-8d7c-7692848c43be-kube-api-access-5xpmr\") pod \"maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx\" (UID: \"d4872501-9b29-47b6-8d7c-7692848c43be\") " pod="opendatahub/maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx" Jul 06 15:50:44.372557 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:44.372425 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4872501-9b29-47b6-8d7c-7692848c43be-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx\" (UID: \"d4872501-9b29-47b6-8d7c-7692848c43be\") " pod="opendatahub/maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx" Jul 06 15:50:44.372557 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:44.372453 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4872501-9b29-47b6-8d7c-7692848c43be-maas-api-tls\") pod \"maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx\" (UID: \"d4872501-9b29-47b6-8d7c-7692848c43be\") " pod="opendatahub/maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx" Jul 06 15:50:44.372694 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:50:44.372596 2570 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-a-7a30a8-serving-cert: secret "maas-api-e2e-inf-a-7a30a8-serving-cert" not found Jul 06 15:50:44.372694 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:50:44.372654 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4872501-9b29-47b6-8d7c-7692848c43be-maas-api-tls podName:d4872501-9b29-47b6-8d7c-7692848c43be nodeName:}" failed. No retries permitted until 2026-07-06 15:50:44.87263893 +0000 UTC m=+1991.486387448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d4872501-9b29-47b6-8d7c-7692848c43be-maas-api-tls") pod "maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx" (UID: "d4872501-9b29-47b6-8d7c-7692848c43be") : secret "maas-api-e2e-inf-a-7a30a8-serving-cert" not found Jul 06 15:50:44.373067 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:44.373048 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4872501-9b29-47b6-8d7c-7692848c43be-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx\" (UID: \"d4872501-9b29-47b6-8d7c-7692848c43be\") " pod="opendatahub/maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx" Jul 06 15:50:44.389440 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:44.389375 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xpmr\" (UniqueName: \"kubernetes.io/projected/d4872501-9b29-47b6-8d7c-7692848c43be-kube-api-access-5xpmr\") pod \"maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx\" (UID: \"d4872501-9b29-47b6-8d7c-7692848c43be\") " pod="opendatahub/maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx" Jul 06 15:50:44.492021 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:44.491991 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:44.595423 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:44.595390 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:44.879540 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:44.879484 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4872501-9b29-47b6-8d7c-7692848c43be-maas-api-tls\") pod \"maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx\" (UID: \"d4872501-9b29-47b6-8d7c-7692848c43be\") " pod="opendatahub/maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx" Jul 06 15:50:44.882442 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:44.882419 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4872501-9b29-47b6-8d7c-7692848c43be-maas-api-tls\") pod \"maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx\" (UID: \"d4872501-9b29-47b6-8d7c-7692848c43be\") " pod="opendatahub/maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx" Jul 06 15:50:44.994626 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:44.994584 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:45.128579 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:45.128546 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx" Jul 06 15:50:45.175877 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:45.175848 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:45.385609 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:45.385578 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx"] Jul 06 15:50:45.387696 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:50:45.387672 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4872501_9b29_47b6_8d7c_7692848c43be.slice/crio-e0b92af8e982a651502a8fe5c12e2e61fa4ee32c2413c834abeea5ac768cf301 WatchSource:0}: Error finding container e0b92af8e982a651502a8fe5c12e2e61fa4ee32c2413c834abeea5ac768cf301: Status 404 returned error can't find the container with id e0b92af8e982a651502a8fe5c12e2e61fa4ee32c2413c834abeea5ac768cf301 Jul 06 15:50:45.389286 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:45.389264 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 15:50:45.995891 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:45.995854 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx" event={"ID":"d4872501-9b29-47b6-8d7c-7692848c43be","Type":"ContainerStarted","Data":"2665bcfe84b44dde91a727c890335da0983c759f8c0e4e1248975c3a749e0bc8"} Jul 06 15:50:45.996092 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:45.995912 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx" Jul 06 15:50:45.996092 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:45.995933 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx" event={"ID":"d4872501-9b29-47b6-8d7c-7692848c43be","Type":"ContainerStarted","Data":"e0b92af8e982a651502a8fe5c12e2e61fa4ee32c2413c834abeea5ac768cf301"} Jul 06 15:50:46.046977 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:46.046924 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx" podStartSLOduration=1.5954802780000001 podStartE2EDuration="2.046910706s" totalImagesPullingTime="451.430428ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:50:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:50:46.046724252 +0000 UTC m=+1992.660472787" watchObservedRunningTime="2026-07-06 15:50:46.046910706 +0000 UTC m=+1992.660659240" Jul 06 15:50:48.804303 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:48.804274 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:49.124062 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:49.124032 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:49.562603 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:49.562479 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:49.805794 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:49.805759 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:52.006346 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:52.006320 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx" Jul 06 15:50:57.482840 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:57.482802 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:57.578775 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:57.578747 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-edc987-68f496c949-txrfz"] Jul 06 15:50:57.581484 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:57.581460 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-edc987-68f496c949-txrfz" Jul 06 15:50:57.585495 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:57.585477 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-edc987-serving-cert\"" Jul 06 15:50:57.606331 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:57.606304 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-edc987-68f496c949-txrfz"] Jul 06 15:50:57.651435 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:57.651403 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:57.704957 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:57.704925 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c-maas-api-tls\") pod \"maas-api-e2e-inf-b-edc987-68f496c949-txrfz\" (UID: \"8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c\") " pod="opendatahub/maas-api-e2e-inf-b-edc987-68f496c949-txrfz" Jul 06 15:50:57.705151 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:57.704965 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6g7h\" (UniqueName: \"kubernetes.io/projected/8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c-kube-api-access-g6g7h\") pod \"maas-api-e2e-inf-b-edc987-68f496c949-txrfz\" (UID: \"8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c\") " pod="opendatahub/maas-api-e2e-inf-b-edc987-68f496c949-txrfz" Jul 06 15:50:57.705151 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:57.705022 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-edc987-68f496c949-txrfz\" (UID: \"8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c\") " pod="opendatahub/maas-api-e2e-inf-b-edc987-68f496c949-txrfz" Jul 06 15:50:57.765131 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:57.765050 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:57.806477 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:57.806444 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c-maas-api-tls\") pod \"maas-api-e2e-inf-b-edc987-68f496c949-txrfz\" (UID: \"8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c\") " pod="opendatahub/maas-api-e2e-inf-b-edc987-68f496c949-txrfz" Jul 06 15:50:57.806477 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:57.806478 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6g7h\" (UniqueName: \"kubernetes.io/projected/8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c-kube-api-access-g6g7h\") pod \"maas-api-e2e-inf-b-edc987-68f496c949-txrfz\" (UID: \"8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c\") " pod="opendatahub/maas-api-e2e-inf-b-edc987-68f496c949-txrfz" Jul 06 15:50:57.806752 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:50:57.806623 2570 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-b-edc987-serving-cert: secret "maas-api-e2e-inf-b-edc987-serving-cert" not found Jul 06 15:50:57.806752 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:57.806654 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-edc987-68f496c949-txrfz\" (UID: \"8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c\") " pod="opendatahub/maas-api-e2e-inf-b-edc987-68f496c949-txrfz" Jul 06 15:50:57.806752 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:50:57.806691 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c-maas-api-tls podName:8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c nodeName:}" failed. No retries permitted until 2026-07-06 15:50:58.306669716 +0000 UTC m=+2004.920418242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c-maas-api-tls") pod "maas-api-e2e-inf-b-edc987-68f496c949-txrfz" (UID: "8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c") : secret "maas-api-e2e-inf-b-edc987-serving-cert" not found Jul 06 15:50:57.807285 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:57.807266 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-edc987-68f496c949-txrfz\" (UID: \"8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c\") " pod="opendatahub/maas-api-e2e-inf-b-edc987-68f496c949-txrfz" Jul 06 15:50:57.832586 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:57.832557 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6g7h\" (UniqueName: \"kubernetes.io/projected/8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c-kube-api-access-g6g7h\") pod \"maas-api-e2e-inf-b-edc987-68f496c949-txrfz\" (UID: \"8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c\") " pod="opendatahub/maas-api-e2e-inf-b-edc987-68f496c949-txrfz" Jul 06 15:50:58.173365 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:58.173327 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:58.311804 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:58.311764 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c-maas-api-tls\") pod \"maas-api-e2e-inf-b-edc987-68f496c949-txrfz\" (UID: \"8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c\") " pod="opendatahub/maas-api-e2e-inf-b-edc987-68f496c949-txrfz" Jul 06 15:50:58.314233 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:58.314204 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c-maas-api-tls\") pod \"maas-api-e2e-inf-b-edc987-68f496c949-txrfz\" (UID: \"8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c\") " pod="opendatahub/maas-api-e2e-inf-b-edc987-68f496c949-txrfz" Jul 06 15:50:58.493184 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:58.493097 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-edc987-68f496c949-txrfz" Jul 06 15:50:58.726231 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:58.726208 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-edc987-68f496c949-txrfz"] Jul 06 15:50:58.728840 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:50:58.728805 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd7bca6_3b61_42ea_b8ab_ae14f68adb3c.slice/crio-709bc2b3d9a306af091c8b161df2cb1a41226f817f50685ca4a66951ab906032 WatchSource:0}: Error finding container 709bc2b3d9a306af091c8b161df2cb1a41226f817f50685ca4a66951ab906032: Status 404 returned error can't find the container with id 709bc2b3d9a306af091c8b161df2cb1a41226f817f50685ca4a66951ab906032 Jul 06 15:50:58.996484 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:58.996446 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:59.070528 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:59.070422 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-edc987-68f496c949-txrfz" event={"ID":"8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c","Type":"ContainerStarted","Data":"709bc2b3d9a306af091c8b161df2cb1a41226f817f50685ca4a66951ab906032"} Jul 06 15:50:59.139760 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:50:59.139728 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:51:00.077760 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:00.077721 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-edc987-68f496c949-txrfz" event={"ID":"8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c","Type":"ContainerStarted","Data":"c550ff08b5e5972138e6c312c02b9fe7871780e040048ddcaab6f19b9b3d9595"} Jul 06 15:51:00.078227 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:00.077871 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-edc987-68f496c949-txrfz" Jul 06 15:51:00.102084 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:00.102010 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-edc987-68f496c949-txrfz" podStartSLOduration=2.742222489 podStartE2EDuration="3.101988499s" totalImagesPullingTime="359.76601ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:50:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:51:00.098758117 +0000 UTC m=+2006.712506653" watchObservedRunningTime="2026-07-06 15:51:00.101988499 +0000 UTC m=+2006.715737036" Jul 06 15:51:01.965484 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:01.965392 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:51:02.088915 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:02.088884 2570 generic.go:356] "Generic (PLEG): container finished" podID="21ae110e-062d-4af2-9938-329252177d7f" containerID="418c65a0e464389365982b4ca2dbc4ac77cf185b263349c7169c5d54def7913c" exitCode=0 Jul 06 15:51:02.089089 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:02.088931 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" event={"ID":"21ae110e-062d-4af2-9938-329252177d7f","Type":"ContainerDied","Data":"418c65a0e464389365982b4ca2dbc4ac77cf185b263349c7169c5d54def7913c"} Jul 06 15:51:02.089379 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:02.089366 2570 scope.go:117] "RemoveContainer" containerID="418c65a0e464389365982b4ca2dbc4ac77cf185b263349c7169c5d54def7913c" Jul 06 15:51:02.090387 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:02.090359 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:51:02.090438 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:02.090428 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:51:02.090480 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:02.090457 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 15:51:03.096088 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:03.096055 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" event={"ID":"21ae110e-062d-4af2-9938-329252177d7f","Type":"ContainerStarted","Data":"b2768cb0e84963517e0d6a6ed9df358ecdd9aa7cf81eebc540bb1654afa104c1"} Jul 06 15:51:03.925163 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:03.925126 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:51:03.925365 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:03.925343 2570 patch_prober.go:28] interesting pod/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 06 15:51:03.925419 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:03.925389 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" podUID="21ae110e-062d-4af2-9938-329252177d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 06 15:51:04.924824 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:04.924790 2570 patch_prober.go:28] interesting pod/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 06 15:51:04.925197 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:04.924846 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" podUID="21ae110e-062d-4af2-9938-329252177d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 06 15:51:05.925601 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:05.925569 2570 patch_prober.go:28] interesting pod/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 06 15:51:05.925979 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:05.925621 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" podUID="21ae110e-062d-4af2-9938-329252177d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 06 15:51:06.088049 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:06.088021 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-edc987-68f496c949-txrfz" Jul 06 15:51:06.224871 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:06.224787 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:51:06.409842 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:06.409812 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:51:06.925161 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:06.925128 2570 patch_prober.go:28] interesting pod/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 06 15:51:06.925343 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:06.925180 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" podUID="21ae110e-062d-4af2-9938-329252177d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 06 15:51:07.928695 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:07.928666 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:51:07.929074 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:07.929027 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:51:07.929795 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:07.929774 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:51:12.720911 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:12.720882 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:51:21.406796 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:21.406764 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:51:21.507242 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:21.507207 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:51:30.465300 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:30.465254 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:51:34.869747 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:34.869716 2570 scope.go:117] "RemoveContainer" containerID="33018b6c5567f837ad3b8871c0816f147fa7bcd46a0a0ab23fc4f0cd1fe01d72" Jul 06 15:51:50.372799 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:50.372767 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:51:51.370836 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:51.370802 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:51:51.622086 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:51.622005 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:51:51.973940 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:51.973854 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:51:52.149852 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:52.149821 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:51:52.260070 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:52.259978 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:51:52.507859 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:52.507821 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx"] Jul 06 15:51:52.508071 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:52.508039 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx" podUID="d4872501-9b29-47b6-8d7c-7692848c43be" containerName="maas-api" containerID="cri-o://2665bcfe84b44dde91a727c890335da0983c759f8c0e4e1248975c3a749e0bc8" gracePeriod=30 Jul 06 15:51:52.602557 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:52.602515 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:51:52.762289 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:52.762263 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx" Jul 06 15:51:52.849822 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:52.849789 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4872501-9b29-47b6-8d7c-7692848c43be-openshift-service-ca\") pod \"d4872501-9b29-47b6-8d7c-7692848c43be\" (UID: \"d4872501-9b29-47b6-8d7c-7692848c43be\") " Jul 06 15:51:52.850014 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:52.849834 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4872501-9b29-47b6-8d7c-7692848c43be-maas-api-tls\") pod \"d4872501-9b29-47b6-8d7c-7692848c43be\" (UID: \"d4872501-9b29-47b6-8d7c-7692848c43be\") " Jul 06 15:51:52.850014 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:52.849894 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xpmr\" (UniqueName: \"kubernetes.io/projected/d4872501-9b29-47b6-8d7c-7692848c43be-kube-api-access-5xpmr\") pod \"d4872501-9b29-47b6-8d7c-7692848c43be\" (UID: \"d4872501-9b29-47b6-8d7c-7692848c43be\") " Jul 06 15:51:52.850174 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:52.850146 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4872501-9b29-47b6-8d7c-7692848c43be-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d4872501-9b29-47b6-8d7c-7692848c43be" (UID: "d4872501-9b29-47b6-8d7c-7692848c43be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:51:52.851975 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:52.851947 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4872501-9b29-47b6-8d7c-7692848c43be-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d4872501-9b29-47b6-8d7c-7692848c43be" (UID: "d4872501-9b29-47b6-8d7c-7692848c43be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:51:52.852067 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:52.851995 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4872501-9b29-47b6-8d7c-7692848c43be-kube-api-access-5xpmr" (OuterVolumeSpecName: "kube-api-access-5xpmr") pod "d4872501-9b29-47b6-8d7c-7692848c43be" (UID: "d4872501-9b29-47b6-8d7c-7692848c43be"). InnerVolumeSpecName "kube-api-access-5xpmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:51:52.951051 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:52.951021 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4872501-9b29-47b6-8d7c-7692848c43be-openshift-service-ca\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:51:52.951051 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:52.951048 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4872501-9b29-47b6-8d7c-7692848c43be-maas-api-tls\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:51:52.951051 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:52.951057 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xpmr\" (UniqueName: \"kubernetes.io/projected/d4872501-9b29-47b6-8d7c-7692848c43be-kube-api-access-5xpmr\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:51:53.353284 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:53.353250 2570 generic.go:356] "Generic (PLEG): container finished" podID="d4872501-9b29-47b6-8d7c-7692848c43be" containerID="2665bcfe84b44dde91a727c890335da0983c759f8c0e4e1248975c3a749e0bc8" exitCode=0 Jul 06 15:51:53.353436 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:53.353303 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx" Jul 06 15:51:53.353436 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:53.353333 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx" event={"ID":"d4872501-9b29-47b6-8d7c-7692848c43be","Type":"ContainerDied","Data":"2665bcfe84b44dde91a727c890335da0983c759f8c0e4e1248975c3a749e0bc8"} Jul 06 15:51:53.353436 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:53.353384 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx" event={"ID":"d4872501-9b29-47b6-8d7c-7692848c43be","Type":"ContainerDied","Data":"e0b92af8e982a651502a8fe5c12e2e61fa4ee32c2413c834abeea5ac768cf301"} Jul 06 15:51:53.353436 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:53.353407 2570 scope.go:117] "RemoveContainer" containerID="2665bcfe84b44dde91a727c890335da0983c759f8c0e4e1248975c3a749e0bc8" Jul 06 15:51:53.362340 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:53.362321 2570 scope.go:117] "RemoveContainer" containerID="2665bcfe84b44dde91a727c890335da0983c759f8c0e4e1248975c3a749e0bc8" Jul 06 15:51:53.362652 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:51:53.362630 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2665bcfe84b44dde91a727c890335da0983c759f8c0e4e1248975c3a749e0bc8\": container with ID starting with 2665bcfe84b44dde91a727c890335da0983c759f8c0e4e1248975c3a749e0bc8 not found: ID does not exist" containerID="2665bcfe84b44dde91a727c890335da0983c759f8c0e4e1248975c3a749e0bc8" Jul 06 15:51:53.362735 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:53.362660 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2665bcfe84b44dde91a727c890335da0983c759f8c0e4e1248975c3a749e0bc8"} err="failed to get container status \"2665bcfe84b44dde91a727c890335da0983c759f8c0e4e1248975c3a749e0bc8\": rpc error: code = NotFound desc = could not find container \"2665bcfe84b44dde91a727c890335da0983c759f8c0e4e1248975c3a749e0bc8\": container with ID starting with 2665bcfe84b44dde91a727c890335da0983c759f8c0e4e1248975c3a749e0bc8 not found: ID does not exist" Jul 06 15:51:53.379816 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:53.379780 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx"] Jul 06 15:51:53.388135 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:53.388104 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-7a30a8-9b4886f78-qfbvx"] Jul 06 15:51:54.113611 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:54.113573 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:51:54.216442 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:54.216407 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:51:54.801142 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:54.801108 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4872501-9b29-47b6-8d7c-7692848c43be" path="/var/lib/kubelet/pods/d4872501-9b29-47b6-8d7c-7692848c43be/volumes" Jul 06 15:51:57.058690 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:51:57.058655 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:52:34.796552 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:34.796521 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-mlw9w_91e0c855-5b32-4ca5-bab6-8e3aae20c1e0/console-operator/2.log" Jul 06 15:52:34.801212 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:34.801188 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-mlw9w_91e0c855-5b32-4ca5-bab6-8e3aae20c1e0/console-operator/2.log" Jul 06 15:52:34.801376 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:34.801273 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hxpw_3eca6180-d678-4729-ad6f-979a025af841/ovn-acl-logging/0.log" Jul 06 15:52:34.804989 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:34.804967 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hxpw_3eca6180-d678-4729-ad6f-979a025af841/ovn-acl-logging/0.log" Jul 06 15:52:35.429112 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:35.429082 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:52:35.740806 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:35.740708 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv"] Jul 06 15:52:35.741015 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:35.740970 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" podUID="21ae110e-062d-4af2-9938-329252177d7f" containerName="istio-proxy" containerID="cri-o://b2768cb0e84963517e0d6a6ed9df358ecdd9aa7cf81eebc540bb1654afa104c1" gracePeriod=30 Jul 06 15:52:35.794459 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:35.794426 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:52:35.995883 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:35.995794 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:52:36.348367 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:36.348337 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:52:36.794071 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:36.793971 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:52:36.911875 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:36.911845 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:52:37.113549 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:37.113518 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:52:37.338197 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:37.338161 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-edc987-68f496c949-txrfz"] Jul 06 15:52:37.338431 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:37.338406 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-edc987-68f496c949-txrfz" podUID="8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c" containerName="maas-api" containerID="cri-o://c550ff08b5e5972138e6c312c02b9fe7871780e040048ddcaab6f19b9b3d9595" gracePeriod=30 Jul 06 15:52:37.505752 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:37.505710 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:52:37.573961 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:37.573929 2570 generic.go:356] "Generic (PLEG): container finished" podID="8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c" containerID="c550ff08b5e5972138e6c312c02b9fe7871780e040048ddcaab6f19b9b3d9595" exitCode=0 Jul 06 15:52:37.574141 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:37.574005 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-edc987-68f496c949-txrfz" event={"ID":"8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c","Type":"ContainerDied","Data":"c550ff08b5e5972138e6c312c02b9fe7871780e040048ddcaab6f19b9b3d9595"} Jul 06 15:52:37.574141 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:37.574054 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-edc987-68f496c949-txrfz" event={"ID":"8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c","Type":"ContainerDied","Data":"709bc2b3d9a306af091c8b161df2cb1a41226f817f50685ca4a66951ab906032"} Jul 06 15:52:37.574141 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:37.574075 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="709bc2b3d9a306af091c8b161df2cb1a41226f817f50685ca4a66951ab906032" Jul 06 15:52:37.580971 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:37.580952 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-edc987-68f496c949-txrfz" Jul 06 15:52:37.682524 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:37.682463 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c-openshift-service-ca\") pod \"8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c\" (UID: \"8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c\") " Jul 06 15:52:37.682711 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:37.682583 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6g7h\" (UniqueName: \"kubernetes.io/projected/8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c-kube-api-access-g6g7h\") pod \"8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c\" (UID: \"8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c\") " Jul 06 15:52:37.682711 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:37.682629 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c-maas-api-tls\") pod \"8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c\" (UID: \"8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c\") " Jul 06 15:52:37.682840 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:37.682816 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c" (UID: "8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:52:37.682919 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:37.682902 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c-openshift-service-ca\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:52:37.684784 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:37.684748 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c-kube-api-access-g6g7h" (OuterVolumeSpecName: "kube-api-access-g6g7h") pod "8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c" (UID: "8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c"). InnerVolumeSpecName "kube-api-access-g6g7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:52:37.684784 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:37.684774 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c" (UID: "8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:52:37.783647 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:37.783547 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6g7h\" (UniqueName: \"kubernetes.io/projected/8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c-kube-api-access-g6g7h\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:52:37.783647 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:37.783587 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c-maas-api-tls\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:52:37.931493 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:37.931454 2570 patch_prober.go:28] interesting pod/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 15:52:37.931671 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:37.931556 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" podUID="21ae110e-062d-4af2-9938-329252177d7f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 15:52:38.578946 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:38.578915 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-edc987-68f496c949-txrfz" Jul 06 15:52:38.605526 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:38.605488 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-edc987-68f496c949-txrfz"] Jul 06 15:52:38.611392 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:38.611372 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-edc987-68f496c949-txrfz"] Jul 06 15:52:38.799979 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:38.799945 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c" path="/var/lib/kubelet/pods/8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c/volumes" Jul 06 15:52:41.001049 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.000982 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:52:41.117879 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.117846 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21ae110e-062d-4af2-9938-329252177d7f-workload-certs\") pod \"21ae110e-062d-4af2-9938-329252177d7f\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " Jul 06 15:52:41.118055 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.117920 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21ae110e-062d-4af2-9938-329252177d7f-istio-podinfo\") pod \"21ae110e-062d-4af2-9938-329252177d7f\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " Jul 06 15:52:41.118055 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.117984 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21ae110e-062d-4af2-9938-329252177d7f-istio-data\") pod \"21ae110e-062d-4af2-9938-329252177d7f\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " Jul 06 15:52:41.118055 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.118050 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21ae110e-062d-4af2-9938-329252177d7f-istiod-ca-cert\") pod \"21ae110e-062d-4af2-9938-329252177d7f\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " Jul 06 15:52:41.118233 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.118091 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ae110e-062d-4af2-9938-329252177d7f-credential-socket\") pod \"21ae110e-062d-4af2-9938-329252177d7f\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " Jul 06 15:52:41.118233 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.118118 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21ae110e-062d-4af2-9938-329252177d7f-istio-token\") pod \"21ae110e-062d-4af2-9938-329252177d7f\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " Jul 06 15:52:41.118233 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.118147 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9skc9\" (UniqueName: \"kubernetes.io/projected/21ae110e-062d-4af2-9938-329252177d7f-kube-api-access-9skc9\") pod \"21ae110e-062d-4af2-9938-329252177d7f\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " Jul 06 15:52:41.118233 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.118177 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ae110e-062d-4af2-9938-329252177d7f-workload-socket\") pod \"21ae110e-062d-4af2-9938-329252177d7f\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " Jul 06 15:52:41.118233 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.118197 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ae110e-062d-4af2-9938-329252177d7f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "21ae110e-062d-4af2-9938-329252177d7f" (UID: "21ae110e-062d-4af2-9938-329252177d7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:52:41.118233 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.118219 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21ae110e-062d-4af2-9938-329252177d7f-istio-envoy\") pod \"21ae110e-062d-4af2-9938-329252177d7f\" (UID: \"21ae110e-062d-4af2-9938-329252177d7f\") " Jul 06 15:52:41.118582 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.118538 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ae110e-062d-4af2-9938-329252177d7f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "21ae110e-062d-4af2-9938-329252177d7f" (UID: "21ae110e-062d-4af2-9938-329252177d7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:52:41.118711 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.118666 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ae110e-062d-4af2-9938-329252177d7f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "21ae110e-062d-4af2-9938-329252177d7f" (UID: "21ae110e-062d-4af2-9938-329252177d7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:52:41.118711 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.118683 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21ae110e-062d-4af2-9938-329252177d7f-workload-certs\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:52:41.118815 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.118748 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21ae110e-062d-4af2-9938-329252177d7f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "21ae110e-062d-4af2-9938-329252177d7f" (UID: "21ae110e-062d-4af2-9938-329252177d7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:52:41.119154 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.119131 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ae110e-062d-4af2-9938-329252177d7f-istio-data" (OuterVolumeSpecName: "istio-data") pod "21ae110e-062d-4af2-9938-329252177d7f" (UID: "21ae110e-062d-4af2-9938-329252177d7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:52:41.120433 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.120410 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ae110e-062d-4af2-9938-329252177d7f-kube-api-access-9skc9" (OuterVolumeSpecName: "kube-api-access-9skc9") pod "21ae110e-062d-4af2-9938-329252177d7f" (UID: "21ae110e-062d-4af2-9938-329252177d7f"). InnerVolumeSpecName "kube-api-access-9skc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:52:41.120524 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.120437 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ae110e-062d-4af2-9938-329252177d7f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "21ae110e-062d-4af2-9938-329252177d7f" (UID: "21ae110e-062d-4af2-9938-329252177d7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:52:41.120597 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.120581 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ae110e-062d-4af2-9938-329252177d7f-istio-token" (OuterVolumeSpecName: "istio-token") pod "21ae110e-062d-4af2-9938-329252177d7f" (UID: "21ae110e-062d-4af2-9938-329252177d7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:52:41.120636 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.120593 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21ae110e-062d-4af2-9938-329252177d7f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "21ae110e-062d-4af2-9938-329252177d7f" (UID: "21ae110e-062d-4af2-9938-329252177d7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 15:52:41.219819 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.219781 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21ae110e-062d-4af2-9938-329252177d7f-istiod-ca-cert\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:52:41.219819 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.219812 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ae110e-062d-4af2-9938-329252177d7f-credential-socket\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:52:41.219819 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.219822 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21ae110e-062d-4af2-9938-329252177d7f-istio-token\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:52:41.219819 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.219831 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9skc9\" (UniqueName: \"kubernetes.io/projected/21ae110e-062d-4af2-9938-329252177d7f-kube-api-access-9skc9\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:52:41.220077 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.219839 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ae110e-062d-4af2-9938-329252177d7f-workload-socket\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:52:41.220077 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.219849 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21ae110e-062d-4af2-9938-329252177d7f-istio-envoy\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:52:41.220077 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.219858 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21ae110e-062d-4af2-9938-329252177d7f-istio-podinfo\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:52:41.220077 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.219865 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21ae110e-062d-4af2-9938-329252177d7f-istio-data\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:52:41.594548 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.594513 2570 generic.go:356] "Generic (PLEG): container finished" podID="21ae110e-062d-4af2-9938-329252177d7f" containerID="b2768cb0e84963517e0d6a6ed9df358ecdd9aa7cf81eebc540bb1654afa104c1" exitCode=0 Jul 06 15:52:41.594732 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.594565 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" event={"ID":"21ae110e-062d-4af2-9938-329252177d7f","Type":"ContainerDied","Data":"b2768cb0e84963517e0d6a6ed9df358ecdd9aa7cf81eebc540bb1654afa104c1"} Jul 06 15:52:41.594732 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.594573 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" Jul 06 15:52:41.594732 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.594614 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv" event={"ID":"21ae110e-062d-4af2-9938-329252177d7f","Type":"ContainerDied","Data":"59548f38bf7f89d65454df843cddc9718196a7dfc90240e279b16171bd9b2709"} Jul 06 15:52:41.594732 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.594636 2570 scope.go:117] "RemoveContainer" containerID="b2768cb0e84963517e0d6a6ed9df358ecdd9aa7cf81eebc540bb1654afa104c1" Jul 06 15:52:41.603804 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.603779 2570 scope.go:117] "RemoveContainer" containerID="418c65a0e464389365982b4ca2dbc4ac77cf185b263349c7169c5d54def7913c" Jul 06 15:52:41.611985 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.611962 2570 scope.go:117] "RemoveContainer" containerID="b2768cb0e84963517e0d6a6ed9df358ecdd9aa7cf81eebc540bb1654afa104c1" Jul 06 15:52:41.612272 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:52:41.612248 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2768cb0e84963517e0d6a6ed9df358ecdd9aa7cf81eebc540bb1654afa104c1\": container with ID starting with b2768cb0e84963517e0d6a6ed9df358ecdd9aa7cf81eebc540bb1654afa104c1 not found: ID does not exist" containerID="b2768cb0e84963517e0d6a6ed9df358ecdd9aa7cf81eebc540bb1654afa104c1" Jul 06 15:52:41.612348 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.612282 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2768cb0e84963517e0d6a6ed9df358ecdd9aa7cf81eebc540bb1654afa104c1"} err="failed to get container status \"b2768cb0e84963517e0d6a6ed9df358ecdd9aa7cf81eebc540bb1654afa104c1\": rpc error: code = NotFound desc = could not find container \"b2768cb0e84963517e0d6a6ed9df358ecdd9aa7cf81eebc540bb1654afa104c1\": container with ID starting with b2768cb0e84963517e0d6a6ed9df358ecdd9aa7cf81eebc540bb1654afa104c1 not found: ID does not exist" Jul 06 15:52:41.612348 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.612302 2570 scope.go:117] "RemoveContainer" containerID="418c65a0e464389365982b4ca2dbc4ac77cf185b263349c7169c5d54def7913c" Jul 06 15:52:41.612551 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:52:41.612535 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"418c65a0e464389365982b4ca2dbc4ac77cf185b263349c7169c5d54def7913c\": container with ID starting with 418c65a0e464389365982b4ca2dbc4ac77cf185b263349c7169c5d54def7913c not found: ID does not exist" containerID="418c65a0e464389365982b4ca2dbc4ac77cf185b263349c7169c5d54def7913c" Jul 06 15:52:41.612603 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.612556 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"418c65a0e464389365982b4ca2dbc4ac77cf185b263349c7169c5d54def7913c"} err="failed to get container status \"418c65a0e464389365982b4ca2dbc4ac77cf185b263349c7169c5d54def7913c\": rpc error: code = NotFound desc = could not find container \"418c65a0e464389365982b4ca2dbc4ac77cf185b263349c7169c5d54def7913c\": container with ID starting with 418c65a0e464389365982b4ca2dbc4ac77cf185b263349c7169c5d54def7913c not found: ID does not exist" Jul 06 15:52:41.628355 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.628303 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv"] Jul 06 15:52:41.628554 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:41.628372 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7a30a8-openshift-default-84cbd8d8cc-jcsfv"] Jul 06 15:52:42.800034 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:42.799999 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21ae110e-062d-4af2-9938-329252177d7f" path="/var/lib/kubelet/pods/21ae110e-062d-4af2-9938-329252177d7f/volumes" Jul 06 15:52:43.605182 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:43.605114 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:52:43.718727 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:43.718697 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:52:43.831828 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:43.831769 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:52:44.331330 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:44.331297 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:52:44.498170 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:44.498137 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:52:44.694197 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:44.694121 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:52:44.989608 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:44.989521 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:52:48.129141 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:48.129112 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:52:48.253342 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:48.253307 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v"] Jul 06 15:52:48.253785 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:48.253766 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4872501-9b29-47b6-8d7c-7692848c43be" containerName="maas-api" Jul 06 15:52:48.253889 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:48.253787 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4872501-9b29-47b6-8d7c-7692848c43be" containerName="maas-api" Jul 06 15:52:48.253889 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:48.253815 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c" containerName="maas-api" Jul 06 15:52:48.253889 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:48.253823 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c" containerName="maas-api" Jul 06 15:52:48.253889 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:48.253837 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21ae110e-062d-4af2-9938-329252177d7f" containerName="istio-proxy" Jul 06 15:52:48.253889 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:48.253846 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="21ae110e-062d-4af2-9938-329252177d7f" containerName="istio-proxy" Jul 06 15:52:48.253889 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:48.253861 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21ae110e-062d-4af2-9938-329252177d7f" containerName="istio-proxy" Jul 06 15:52:48.253889 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:48.253868 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="21ae110e-062d-4af2-9938-329252177d7f" containerName="istio-proxy" Jul 06 15:52:48.254234 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:48.253958 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="21ae110e-062d-4af2-9938-329252177d7f" containerName="istio-proxy" Jul 06 15:52:48.254234 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:48.253970 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4872501-9b29-47b6-8d7c-7692848c43be" containerName="maas-api" Jul 06 15:52:48.254234 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:48.253978 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bd7bca6-3b61-42ea-b8ab-ae14f68adb3c" containerName="maas-api" Jul 06 15:52:48.254234 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:48.253986 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="21ae110e-062d-4af2-9938-329252177d7f" containerName="istio-proxy" Jul 06 15:52:48.258156 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:48.258137 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v" Jul 06 15:52:48.265309 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:48.265290 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-7f81c0-serving-cert\"" Jul 06 15:52:48.290155 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:48.290131 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v"] Jul 06 15:52:48.333468 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:48.333435 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:52:48.388344 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:48.388258 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3ec3589-fb4e-444c-b3df-0a1111dfaa07-openshift-service-ca\") pod \"maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v\" (UID: \"f3ec3589-fb4e-444c-b3df-0a1111dfaa07\") " pod="opendatahub/maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v" Jul 06 15:52:48.388344 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:48.388325 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm6sh\" (UniqueName: \"kubernetes.io/projected/f3ec3589-fb4e-444c-b3df-0a1111dfaa07-kube-api-access-bm6sh\") pod \"maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v\" (UID: \"f3ec3589-fb4e-444c-b3df-0a1111dfaa07\") " pod="opendatahub/maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v" Jul 06 15:52:48.388657 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:48.388406 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3ec3589-fb4e-444c-b3df-0a1111dfaa07-maas-api-tls\") pod \"maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v\" (UID: \"f3ec3589-fb4e-444c-b3df-0a1111dfaa07\") " pod="opendatahub/maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v" Jul 06 15:52:48.442329 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:48.442299 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:52:48.489891 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:48.489852 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm6sh\" (UniqueName: \"kubernetes.io/projected/f3ec3589-fb4e-444c-b3df-0a1111dfaa07-kube-api-access-bm6sh\") pod \"maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v\" (UID: \"f3ec3589-fb4e-444c-b3df-0a1111dfaa07\") " pod="opendatahub/maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v" Jul 06 15:52:48.490066 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:48.489935 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3ec3589-fb4e-444c-b3df-0a1111dfaa07-maas-api-tls\") pod \"maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v\" (UID: \"f3ec3589-fb4e-444c-b3df-0a1111dfaa07\") " pod="opendatahub/maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v" Jul 06 15:52:48.490066 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:48.490006 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3ec3589-fb4e-444c-b3df-0a1111dfaa07-openshift-service-ca\") pod \"maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v\" (UID: \"f3ec3589-fb4e-444c-b3df-0a1111dfaa07\") " pod="opendatahub/maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v" Jul 06 15:52:48.490738 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:48.490720 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3ec3589-fb4e-444c-b3df-0a1111dfaa07-openshift-service-ca\") pod \"maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v\" (UID: \"f3ec3589-fb4e-444c-b3df-0a1111dfaa07\") " pod="opendatahub/maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v" Jul 06 15:52:48.492343 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:48.492328 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3ec3589-fb4e-444c-b3df-0a1111dfaa07-maas-api-tls\") pod \"maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v\" (UID: \"f3ec3589-fb4e-444c-b3df-0a1111dfaa07\") " pod="opendatahub/maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v" Jul 06 15:52:48.506777 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:48.506746 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm6sh\" (UniqueName: \"kubernetes.io/projected/f3ec3589-fb4e-444c-b3df-0a1111dfaa07-kube-api-access-bm6sh\") pod \"maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v\" (UID: \"f3ec3589-fb4e-444c-b3df-0a1111dfaa07\") " pod="opendatahub/maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v" Jul 06 15:52:48.567979 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:48.567943 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v" Jul 06 15:52:48.575442 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:48.575408 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:52:48.709988 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:48.709962 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v"] Jul 06 15:52:48.712163 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:52:48.712133 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3ec3589_fb4e_444c_b3df_0a1111dfaa07.slice/crio-2edca9cddfa8ea27672fa5e5fc2e689f4f637118c53508d7336d1a5527ed2501 WatchSource:0}: Error finding container 2edca9cddfa8ea27672fa5e5fc2e689f4f637118c53508d7336d1a5527ed2501: Status 404 returned error can't find the container with id 2edca9cddfa8ea27672fa5e5fc2e689f4f637118c53508d7336d1a5527ed2501 Jul 06 15:52:48.995891 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:48.995804 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:52:49.154514 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:49.154478 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:52:49.640005 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:49.639972 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v" event={"ID":"f3ec3589-fb4e-444c-b3df-0a1111dfaa07","Type":"ContainerStarted","Data":"1e2cbe237712ec7e45d697a398a95c4e5169c8f350c5f3aef1dbe6a6a1114bbc"} Jul 06 15:52:49.640203 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:49.640032 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v" event={"ID":"f3ec3589-fb4e-444c-b3df-0a1111dfaa07","Type":"ContainerStarted","Data":"2edca9cddfa8ea27672fa5e5fc2e689f4f637118c53508d7336d1a5527ed2501"} Jul 06 15:52:49.640203 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:49.640114 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v" Jul 06 15:52:49.661519 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:49.661452 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v" podStartSLOduration=0.823487943 podStartE2EDuration="1.661436441s" totalImagesPullingTime="837.948498ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:52:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:52:49.660108813 +0000 UTC m=+2116.273857353" watchObservedRunningTime="2026-07-06 15:52:49.661436441 +0000 UTC m=+2116.275184976" Jul 06 15:52:52.472519 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:52.472478 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:52:52.936996 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:52.936963 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:52:53.252933 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:53.252841 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:52:53.800685 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:53.800652 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:52:55.650834 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:55.650806 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v" Jul 06 15:52:55.973822 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:55.973706 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:52:56.031877 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:56.031846 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7"] Jul 06 15:52:56.035845 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:56.035823 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7" Jul 06 15:52:56.038828 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:56.038803 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-c967ab-serving-cert\"" Jul 06 15:52:56.052709 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:56.052684 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7"] Jul 06 15:52:56.101983 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:56.101934 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:52:56.167123 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:56.167086 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f810d2b-7722-4c51-aa58-1f802ed2e082-openshift-service-ca\") pod \"maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7\" (UID: \"9f810d2b-7722-4c51-aa58-1f802ed2e082\") " pod="opendatahub/maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7" Jul 06 15:52:56.167123 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:56.167122 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pv2v\" (UniqueName: \"kubernetes.io/projected/9f810d2b-7722-4c51-aa58-1f802ed2e082-kube-api-access-6pv2v\") pod \"maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7\" (UID: \"9f810d2b-7722-4c51-aa58-1f802ed2e082\") " pod="opendatahub/maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7" Jul 06 15:52:56.167338 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:56.167152 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f810d2b-7722-4c51-aa58-1f802ed2e082-maas-api-tls\") pod \"maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7\" (UID: \"9f810d2b-7722-4c51-aa58-1f802ed2e082\") " pod="opendatahub/maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7" Jul 06 15:52:56.194818 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:56.194789 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:52:56.267820 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:56.267737 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f810d2b-7722-4c51-aa58-1f802ed2e082-openshift-service-ca\") pod \"maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7\" (UID: \"9f810d2b-7722-4c51-aa58-1f802ed2e082\") " pod="opendatahub/maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7" Jul 06 15:52:56.267820 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:56.267773 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pv2v\" (UniqueName: \"kubernetes.io/projected/9f810d2b-7722-4c51-aa58-1f802ed2e082-kube-api-access-6pv2v\") pod \"maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7\" (UID: \"9f810d2b-7722-4c51-aa58-1f802ed2e082\") " pod="opendatahub/maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7" Jul 06 15:52:56.267820 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:56.267791 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f810d2b-7722-4c51-aa58-1f802ed2e082-maas-api-tls\") pod \"maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7\" (UID: \"9f810d2b-7722-4c51-aa58-1f802ed2e082\") " pod="opendatahub/maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7" Jul 06 15:52:56.268036 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:52:56.267980 2570 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-b-c967ab-serving-cert: secret "maas-api-e2e-api-b-c967ab-serving-cert" not found Jul 06 15:52:56.268078 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:52:56.268056 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f810d2b-7722-4c51-aa58-1f802ed2e082-maas-api-tls podName:9f810d2b-7722-4c51-aa58-1f802ed2e082 nodeName:}" failed. No retries permitted until 2026-07-06 15:52:56.768034954 +0000 UTC m=+2123.381783485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9f810d2b-7722-4c51-aa58-1f802ed2e082-maas-api-tls") pod "maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7" (UID: "9f810d2b-7722-4c51-aa58-1f802ed2e082") : secret "maas-api-e2e-api-b-c967ab-serving-cert" not found Jul 06 15:52:56.268468 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:56.268451 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f810d2b-7722-4c51-aa58-1f802ed2e082-openshift-service-ca\") pod \"maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7\" (UID: \"9f810d2b-7722-4c51-aa58-1f802ed2e082\") " pod="opendatahub/maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7" Jul 06 15:52:56.284644 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:56.284621 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pv2v\" (UniqueName: \"kubernetes.io/projected/9f810d2b-7722-4c51-aa58-1f802ed2e082-kube-api-access-6pv2v\") pod \"maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7\" (UID: \"9f810d2b-7722-4c51-aa58-1f802ed2e082\") " pod="opendatahub/maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7" Jul 06 15:52:56.638205 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:56.638173 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:52:56.773364 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:56.773330 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f810d2b-7722-4c51-aa58-1f802ed2e082-maas-api-tls\") pod \"maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7\" (UID: \"9f810d2b-7722-4c51-aa58-1f802ed2e082\") " pod="opendatahub/maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7" Jul 06 15:52:56.775841 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:56.775819 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f810d2b-7722-4c51-aa58-1f802ed2e082-maas-api-tls\") pod \"maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7\" (UID: \"9f810d2b-7722-4c51-aa58-1f802ed2e082\") " pod="opendatahub/maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7" Jul 06 15:52:56.869906 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:56.869866 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:52:56.947078 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:56.946993 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7" Jul 06 15:52:56.975595 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:56.975545 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:52:57.096439 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:57.096392 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7"] Jul 06 15:52:57.098713 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:52:57.098684 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f810d2b_7722_4c51_aa58_1f802ed2e082.slice/crio-14b451015e1f4c33eafe9f66c9e25605146ca611bd8ee057cfd4b6aaf79eb946 WatchSource:0}: Error finding container 14b451015e1f4c33eafe9f66c9e25605146ca611bd8ee057cfd4b6aaf79eb946: Status 404 returned error can't find the container with id 14b451015e1f4c33eafe9f66c9e25605146ca611bd8ee057cfd4b6aaf79eb946 Jul 06 15:52:57.680694 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:57.680657 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7" event={"ID":"9f810d2b-7722-4c51-aa58-1f802ed2e082","Type":"ContainerStarted","Data":"9994090f6c5368802c4595f992291c6e0a66791235e231addc4d990fb47dae82"} Jul 06 15:52:57.680852 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:57.680712 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7" event={"ID":"9f810d2b-7722-4c51-aa58-1f802ed2e082","Type":"ContainerStarted","Data":"14b451015e1f4c33eafe9f66c9e25605146ca611bd8ee057cfd4b6aaf79eb946"} Jul 06 15:52:57.680852 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:57.680744 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7" Jul 06 15:52:57.724454 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:52:57.724355 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7" podStartSLOduration=1.390718071 podStartE2EDuration="1.724339312s" totalImagesPullingTime="333.621241ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:52:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:52:57.723688011 +0000 UTC m=+2124.337436547" watchObservedRunningTime="2026-07-06 15:52:57.724339312 +0000 UTC m=+2124.338087848" Jul 06 15:53:00.479567 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:00.479529 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:03.691221 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:03.691191 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7" Jul 06 15:53:04.425952 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:04.425923 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:07.876226 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:07.876180 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:08.251091 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:08.251007 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:08.921414 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:08.921385 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:09.364061 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:09.364031 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:09.647246 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:09.647167 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:10.029392 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:10.029317 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:10.159315 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:10.159288 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v"] Jul 06 15:53:10.159523 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:10.159486 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v" podUID="f3ec3589-fb4e-444c-b3df-0a1111dfaa07" containerName="maas-api" containerID="cri-o://1e2cbe237712ec7e45d697a398a95c4e5169c8f350c5f3aef1dbe6a6a1114bbc" gracePeriod=30 Jul 06 15:53:10.202746 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:10.202716 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:10.407822 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:10.407793 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v" Jul 06 15:53:10.502334 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:10.502305 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bm6sh\" (UniqueName: \"kubernetes.io/projected/f3ec3589-fb4e-444c-b3df-0a1111dfaa07-kube-api-access-bm6sh\") pod \"f3ec3589-fb4e-444c-b3df-0a1111dfaa07\" (UID: \"f3ec3589-fb4e-444c-b3df-0a1111dfaa07\") " Jul 06 15:53:10.502560 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:10.502410 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3ec3589-fb4e-444c-b3df-0a1111dfaa07-openshift-service-ca\") pod \"f3ec3589-fb4e-444c-b3df-0a1111dfaa07\" (UID: \"f3ec3589-fb4e-444c-b3df-0a1111dfaa07\") " Jul 06 15:53:10.502560 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:10.502456 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3ec3589-fb4e-444c-b3df-0a1111dfaa07-maas-api-tls\") pod \"f3ec3589-fb4e-444c-b3df-0a1111dfaa07\" (UID: \"f3ec3589-fb4e-444c-b3df-0a1111dfaa07\") " Jul 06 15:53:10.502816 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:10.502791 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3ec3589-fb4e-444c-b3df-0a1111dfaa07-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f3ec3589-fb4e-444c-b3df-0a1111dfaa07" (UID: "f3ec3589-fb4e-444c-b3df-0a1111dfaa07"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:53:10.504539 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:10.504496 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3ec3589-fb4e-444c-b3df-0a1111dfaa07-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f3ec3589-fb4e-444c-b3df-0a1111dfaa07" (UID: "f3ec3589-fb4e-444c-b3df-0a1111dfaa07"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:53:10.504612 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:10.504585 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3ec3589-fb4e-444c-b3df-0a1111dfaa07-kube-api-access-bm6sh" (OuterVolumeSpecName: "kube-api-access-bm6sh") pod "f3ec3589-fb4e-444c-b3df-0a1111dfaa07" (UID: "f3ec3589-fb4e-444c-b3df-0a1111dfaa07"). InnerVolumeSpecName "kube-api-access-bm6sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:53:10.603738 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:10.603656 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm6sh\" (UniqueName: \"kubernetes.io/projected/f3ec3589-fb4e-444c-b3df-0a1111dfaa07-kube-api-access-bm6sh\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:53:10.603738 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:10.603682 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3ec3589-fb4e-444c-b3df-0a1111dfaa07-openshift-service-ca\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:53:10.603738 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:10.603693 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3ec3589-fb4e-444c-b3df-0a1111dfaa07-maas-api-tls\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:53:10.748170 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:10.748135 2570 generic.go:356] "Generic (PLEG): container finished" podID="f3ec3589-fb4e-444c-b3df-0a1111dfaa07" containerID="1e2cbe237712ec7e45d697a398a95c4e5169c8f350c5f3aef1dbe6a6a1114bbc" exitCode=0 Jul 06 15:53:10.748347 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:10.748183 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v" Jul 06 15:53:10.748347 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:10.748198 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v" event={"ID":"f3ec3589-fb4e-444c-b3df-0a1111dfaa07","Type":"ContainerDied","Data":"1e2cbe237712ec7e45d697a398a95c4e5169c8f350c5f3aef1dbe6a6a1114bbc"} Jul 06 15:53:10.748347 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:10.748240 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v" event={"ID":"f3ec3589-fb4e-444c-b3df-0a1111dfaa07","Type":"ContainerDied","Data":"2edca9cddfa8ea27672fa5e5fc2e689f4f637118c53508d7336d1a5527ed2501"} Jul 06 15:53:10.748347 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:10.748262 2570 scope.go:117] "RemoveContainer" containerID="1e2cbe237712ec7e45d697a398a95c4e5169c8f350c5f3aef1dbe6a6a1114bbc" Jul 06 15:53:10.765422 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:10.765401 2570 scope.go:117] "RemoveContainer" containerID="1e2cbe237712ec7e45d697a398a95c4e5169c8f350c5f3aef1dbe6a6a1114bbc" Jul 06 15:53:10.765731 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:53:10.765707 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e2cbe237712ec7e45d697a398a95c4e5169c8f350c5f3aef1dbe6a6a1114bbc\": container with ID starting with 1e2cbe237712ec7e45d697a398a95c4e5169c8f350c5f3aef1dbe6a6a1114bbc not found: ID does not exist" containerID="1e2cbe237712ec7e45d697a398a95c4e5169c8f350c5f3aef1dbe6a6a1114bbc" Jul 06 15:53:10.765787 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:10.765731 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e2cbe237712ec7e45d697a398a95c4e5169c8f350c5f3aef1dbe6a6a1114bbc"} err="failed to get container status \"1e2cbe237712ec7e45d697a398a95c4e5169c8f350c5f3aef1dbe6a6a1114bbc\": rpc error: code = NotFound desc = could not find container \"1e2cbe237712ec7e45d697a398a95c4e5169c8f350c5f3aef1dbe6a6a1114bbc\": container with ID starting with 1e2cbe237712ec7e45d697a398a95c4e5169c8f350c5f3aef1dbe6a6a1114bbc not found: ID does not exist" Jul 06 15:53:10.781341 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:10.781320 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v"] Jul 06 15:53:10.783120 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:10.783101 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-7f81c0-5c58d64cb-q7w6v"] Jul 06 15:53:10.798975 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:10.798949 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3ec3589-fb4e-444c-b3df-0a1111dfaa07" path="/var/lib/kubelet/pods/f3ec3589-fb4e-444c-b3df-0a1111dfaa07/volumes" Jul 06 15:53:16.580791 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:16.580758 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:16.714793 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:16.714761 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:16.884434 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:16.884333 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:17.189307 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:17.189227 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:17.641183 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:17.641150 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:18.054639 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:18.054556 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:18.330372 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:18.330341 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:18.451637 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:18.451600 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:18.663718 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:18.663628 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7"] Jul 06 15:53:18.664147 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:18.663881 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7" podUID="9f810d2b-7722-4c51-aa58-1f802ed2e082" containerName="maas-api" containerID="cri-o://9994090f6c5368802c4595f992291c6e0a66791235e231addc4d990fb47dae82" gracePeriod=30 Jul 06 15:53:18.686103 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:18.686076 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7" podUID="9f810d2b-7722-4c51-aa58-1f802ed2e082" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.51:8443/health\": dial tcp 10.134.0.51:8443: connect: connection refused" Jul 06 15:53:18.789301 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:18.789269 2570 generic.go:356] "Generic (PLEG): container finished" podID="9f810d2b-7722-4c51-aa58-1f802ed2e082" containerID="9994090f6c5368802c4595f992291c6e0a66791235e231addc4d990fb47dae82" exitCode=0 Jul 06 15:53:18.789490 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:18.789338 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7" event={"ID":"9f810d2b-7722-4c51-aa58-1f802ed2e082","Type":"ContainerDied","Data":"9994090f6c5368802c4595f992291c6e0a66791235e231addc4d990fb47dae82"} Jul 06 15:53:18.820644 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:18.820612 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:18.919513 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:18.919438 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7" Jul 06 15:53:18.983240 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:18.983208 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pv2v\" (UniqueName: \"kubernetes.io/projected/9f810d2b-7722-4c51-aa58-1f802ed2e082-kube-api-access-6pv2v\") pod \"9f810d2b-7722-4c51-aa58-1f802ed2e082\" (UID: \"9f810d2b-7722-4c51-aa58-1f802ed2e082\") " Jul 06 15:53:18.983409 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:18.983327 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f810d2b-7722-4c51-aa58-1f802ed2e082-openshift-service-ca\") pod \"9f810d2b-7722-4c51-aa58-1f802ed2e082\" (UID: \"9f810d2b-7722-4c51-aa58-1f802ed2e082\") " Jul 06 15:53:18.983409 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:18.983364 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f810d2b-7722-4c51-aa58-1f802ed2e082-maas-api-tls\") pod \"9f810d2b-7722-4c51-aa58-1f802ed2e082\" (UID: \"9f810d2b-7722-4c51-aa58-1f802ed2e082\") " Jul 06 15:53:18.983804 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:18.983770 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f810d2b-7722-4c51-aa58-1f802ed2e082-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9f810d2b-7722-4c51-aa58-1f802ed2e082" (UID: "9f810d2b-7722-4c51-aa58-1f802ed2e082"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:53:18.985398 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:18.985372 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f810d2b-7722-4c51-aa58-1f802ed2e082-kube-api-access-6pv2v" (OuterVolumeSpecName: "kube-api-access-6pv2v") pod "9f810d2b-7722-4c51-aa58-1f802ed2e082" (UID: "9f810d2b-7722-4c51-aa58-1f802ed2e082"). InnerVolumeSpecName "kube-api-access-6pv2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:53:18.985517 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:18.985417 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f810d2b-7722-4c51-aa58-1f802ed2e082-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9f810d2b-7722-4c51-aa58-1f802ed2e082" (UID: "9f810d2b-7722-4c51-aa58-1f802ed2e082"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:53:19.084919 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:19.084884 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f810d2b-7722-4c51-aa58-1f802ed2e082-openshift-service-ca\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:53:19.084919 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:19.084914 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f810d2b-7722-4c51-aa58-1f802ed2e082-maas-api-tls\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:53:19.084919 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:19.084925 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pv2v\" (UniqueName: \"kubernetes.io/projected/9f810d2b-7722-4c51-aa58-1f802ed2e082-kube-api-access-6pv2v\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:53:19.795813 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:19.795782 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7" Jul 06 15:53:19.796290 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:19.795834 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7" event={"ID":"9f810d2b-7722-4c51-aa58-1f802ed2e082","Type":"ContainerDied","Data":"14b451015e1f4c33eafe9f66c9e25605146ca611bd8ee057cfd4b6aaf79eb946"} Jul 06 15:53:19.796290 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:19.795895 2570 scope.go:117] "RemoveContainer" containerID="9994090f6c5368802c4595f992291c6e0a66791235e231addc4d990fb47dae82" Jul 06 15:53:19.840098 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:19.840042 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7"] Jul 06 15:53:19.844349 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:19.844324 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-c967ab-68b7bf5c58-l2tr7"] Jul 06 15:53:20.800073 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:20.800039 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f810d2b-7722-4c51-aa58-1f802ed2e082" path="/var/lib/kubelet/pods/9f810d2b-7722-4c51-aa58-1f802ed2e082/volumes" Jul 06 15:53:25.158791 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:25.158715 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:25.294269 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:25.294237 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:25.487476 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:25.487384 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:25.713705 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:25.713673 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:26.093849 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:26.093817 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:26.508721 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:26.508636 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:26.667016 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:26.666978 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:29.426323 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:29.426287 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:29.497244 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:29.497209 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-379916-7c4967dc55-tq99b"] Jul 06 15:53:29.497692 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:29.497674 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3ec3589-fb4e-444c-b3df-0a1111dfaa07" containerName="maas-api" Jul 06 15:53:29.497791 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:29.497693 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3ec3589-fb4e-444c-b3df-0a1111dfaa07" containerName="maas-api" Jul 06 15:53:29.497791 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:29.497718 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f810d2b-7722-4c51-aa58-1f802ed2e082" containerName="maas-api" Jul 06 15:53:29.497791 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:29.497727 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f810d2b-7722-4c51-aa58-1f802ed2e082" containerName="maas-api" Jul 06 15:53:29.497943 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:29.497811 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3ec3589-fb4e-444c-b3df-0a1111dfaa07" containerName="maas-api" Jul 06 15:53:29.497943 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:29.497823 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f810d2b-7722-4c51-aa58-1f802ed2e082" containerName="maas-api" Jul 06 15:53:29.500865 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:29.500843 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-379916-7c4967dc55-tq99b" Jul 06 15:53:29.504971 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:29.504947 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-379916-serving-cert\"" Jul 06 15:53:29.520946 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:29.520908 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-379916-7c4967dc55-tq99b"] Jul 06 15:53:29.587032 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:29.586995 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6743211a-0b82-48d7-bc29-75d14913fc5a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-379916-7c4967dc55-tq99b\" (UID: \"6743211a-0b82-48d7-bc29-75d14913fc5a\") " pod="opendatahub/maas-api-e2e-shared-a-379916-7c4967dc55-tq99b" Jul 06 15:53:29.587282 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:29.587039 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5drp\" (UniqueName: \"kubernetes.io/projected/6743211a-0b82-48d7-bc29-75d14913fc5a-kube-api-access-l5drp\") pod \"maas-api-e2e-shared-a-379916-7c4967dc55-tq99b\" (UID: \"6743211a-0b82-48d7-bc29-75d14913fc5a\") " pod="opendatahub/maas-api-e2e-shared-a-379916-7c4967dc55-tq99b" Jul 06 15:53:29.587282 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:29.587134 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6743211a-0b82-48d7-bc29-75d14913fc5a-maas-api-tls\") pod \"maas-api-e2e-shared-a-379916-7c4967dc55-tq99b\" (UID: \"6743211a-0b82-48d7-bc29-75d14913fc5a\") " pod="opendatahub/maas-api-e2e-shared-a-379916-7c4967dc55-tq99b" Jul 06 15:53:29.687991 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:29.687897 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6743211a-0b82-48d7-bc29-75d14913fc5a-maas-api-tls\") pod \"maas-api-e2e-shared-a-379916-7c4967dc55-tq99b\" (UID: \"6743211a-0b82-48d7-bc29-75d14913fc5a\") " pod="opendatahub/maas-api-e2e-shared-a-379916-7c4967dc55-tq99b" Jul 06 15:53:29.687991 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:29.687972 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6743211a-0b82-48d7-bc29-75d14913fc5a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-379916-7c4967dc55-tq99b\" (UID: \"6743211a-0b82-48d7-bc29-75d14913fc5a\") " pod="opendatahub/maas-api-e2e-shared-a-379916-7c4967dc55-tq99b" Jul 06 15:53:29.687991 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:29.687990 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5drp\" (UniqueName: \"kubernetes.io/projected/6743211a-0b82-48d7-bc29-75d14913fc5a-kube-api-access-l5drp\") pod \"maas-api-e2e-shared-a-379916-7c4967dc55-tq99b\" (UID: \"6743211a-0b82-48d7-bc29-75d14913fc5a\") " pod="opendatahub/maas-api-e2e-shared-a-379916-7c4967dc55-tq99b" Jul 06 15:53:29.688281 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:53:29.688073 2570 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-a-379916-serving-cert: secret "maas-api-e2e-shared-a-379916-serving-cert" not found Jul 06 15:53:29.688281 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:53:29.688181 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6743211a-0b82-48d7-bc29-75d14913fc5a-maas-api-tls podName:6743211a-0b82-48d7-bc29-75d14913fc5a nodeName:}" failed. No retries permitted until 2026-07-06 15:53:30.188159893 +0000 UTC m=+2156.801908421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6743211a-0b82-48d7-bc29-75d14913fc5a-maas-api-tls") pod "maas-api-e2e-shared-a-379916-7c4967dc55-tq99b" (UID: "6743211a-0b82-48d7-bc29-75d14913fc5a") : secret "maas-api-e2e-shared-a-379916-serving-cert" not found Jul 06 15:53:29.688681 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:29.688663 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6743211a-0b82-48d7-bc29-75d14913fc5a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-379916-7c4967dc55-tq99b\" (UID: \"6743211a-0b82-48d7-bc29-75d14913fc5a\") " pod="opendatahub/maas-api-e2e-shared-a-379916-7c4967dc55-tq99b" Jul 06 15:53:29.713157 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:29.713123 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5drp\" (UniqueName: \"kubernetes.io/projected/6743211a-0b82-48d7-bc29-75d14913fc5a-kube-api-access-l5drp\") pod \"maas-api-e2e-shared-a-379916-7c4967dc55-tq99b\" (UID: \"6743211a-0b82-48d7-bc29-75d14913fc5a\") " pod="opendatahub/maas-api-e2e-shared-a-379916-7c4967dc55-tq99b" Jul 06 15:53:30.151212 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:30.151182 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:30.193760 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:30.193727 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6743211a-0b82-48d7-bc29-75d14913fc5a-maas-api-tls\") pod \"maas-api-e2e-shared-a-379916-7c4967dc55-tq99b\" (UID: \"6743211a-0b82-48d7-bc29-75d14913fc5a\") " pod="opendatahub/maas-api-e2e-shared-a-379916-7c4967dc55-tq99b" Jul 06 15:53:30.196106 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:30.196078 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6743211a-0b82-48d7-bc29-75d14913fc5a-maas-api-tls\") pod \"maas-api-e2e-shared-a-379916-7c4967dc55-tq99b\" (UID: \"6743211a-0b82-48d7-bc29-75d14913fc5a\") " pod="opendatahub/maas-api-e2e-shared-a-379916-7c4967dc55-tq99b" Jul 06 15:53:30.271907 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:30.271875 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:30.411889 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:30.411796 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-379916-7c4967dc55-tq99b" Jul 06 15:53:30.563990 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:30.563952 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:30.574670 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:30.574644 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-379916-7c4967dc55-tq99b"] Jul 06 15:53:30.578530 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:53:30.578492 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6743211a_0b82_48d7_bc29_75d14913fc5a.slice/crio-04449f14239e6dbfd937ce424e7aa4d334f79656dcafd798b4f1ade162c1bd3b WatchSource:0}: Error finding container 04449f14239e6dbfd937ce424e7aa4d334f79656dcafd798b4f1ade162c1bd3b: Status 404 returned error can't find the container with id 04449f14239e6dbfd937ce424e7aa4d334f79656dcafd798b4f1ade162c1bd3b Jul 06 15:53:30.714099 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:30.713999 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:30.851324 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:30.851286 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-379916-7c4967dc55-tq99b" event={"ID":"6743211a-0b82-48d7-bc29-75d14913fc5a","Type":"ContainerStarted","Data":"04449f14239e6dbfd937ce424e7aa4d334f79656dcafd798b4f1ade162c1bd3b"} Jul 06 15:53:31.857440 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:31.857405 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-379916-7c4967dc55-tq99b" event={"ID":"6743211a-0b82-48d7-bc29-75d14913fc5a","Type":"ContainerStarted","Data":"df17983a95120a217c171b79ecaa16a0d18f95995cf82ce363ef2c20283a1209"} Jul 06 15:53:31.857916 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:31.857463 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-379916-7c4967dc55-tq99b" Jul 06 15:53:31.885517 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:31.885455 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-379916-7c4967dc55-tq99b" podStartSLOduration=1.76628356 podStartE2EDuration="2.885439s" totalImagesPullingTime="1.11915544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:53:31.881791331 +0000 UTC m=+2158.495539867" watchObservedRunningTime="2026-07-06 15:53:31.885439 +0000 UTC m=+2158.499187536" Jul 06 15:53:33.886581 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:33.886547 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:34.259055 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:34.258966 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:34.634098 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:34.634065 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:34.902597 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:34.902488 2570 scope.go:117] "RemoveContainer" containerID="e863ebd9c1b671950d68e5e026d8a556af96961c88a48087788eb233d8f8a4d3" Jul 06 15:53:35.059034 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:35.059004 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:35.150781 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:35.150748 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:35.538637 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:35.538600 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:37.406704 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:37.406658 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:37.659328 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:37.659242 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:37.676746 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:37.676719 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv"] Jul 06 15:53:37.681692 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:37.681676 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv" Jul 06 15:53:37.685658 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:37.685637 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-650de1-serving-cert\"" Jul 06 15:53:37.771274 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:37.771239 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b017a0-a17b-4f3a-b5a7-d587666b4a8a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv\" (UID: \"21b017a0-a17b-4f3a-b5a7-d587666b4a8a\") " pod="opendatahub/maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv" Jul 06 15:53:37.771462 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:37.771302 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzvpb\" (UniqueName: \"kubernetes.io/projected/21b017a0-a17b-4f3a-b5a7-d587666b4a8a-kube-api-access-wzvpb\") pod \"maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv\" (UID: \"21b017a0-a17b-4f3a-b5a7-d587666b4a8a\") " pod="opendatahub/maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv" Jul 06 15:53:37.771462 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:37.771335 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21b017a0-a17b-4f3a-b5a7-d587666b4a8a-maas-api-tls\") pod \"maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv\" (UID: \"21b017a0-a17b-4f3a-b5a7-d587666b4a8a\") " pod="opendatahub/maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv" Jul 06 15:53:37.831779 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:37.831743 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv"] Jul 06 15:53:37.867090 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:37.867063 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-379916-7c4967dc55-tq99b" Jul 06 15:53:37.872814 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:37.872787 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b017a0-a17b-4f3a-b5a7-d587666b4a8a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv\" (UID: \"21b017a0-a17b-4f3a-b5a7-d587666b4a8a\") " pod="opendatahub/maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv" Jul 06 15:53:37.872960 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:37.872843 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzvpb\" (UniqueName: \"kubernetes.io/projected/21b017a0-a17b-4f3a-b5a7-d587666b4a8a-kube-api-access-wzvpb\") pod \"maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv\" (UID: \"21b017a0-a17b-4f3a-b5a7-d587666b4a8a\") " pod="opendatahub/maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv" Jul 06 15:53:37.872960 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:37.872875 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21b017a0-a17b-4f3a-b5a7-d587666b4a8a-maas-api-tls\") pod \"maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv\" (UID: \"21b017a0-a17b-4f3a-b5a7-d587666b4a8a\") " pod="opendatahub/maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv" Jul 06 15:53:37.873071 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:53:37.873055 2570 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-b-650de1-serving-cert: secret "maas-api-e2e-shared-b-650de1-serving-cert" not found Jul 06 15:53:37.873128 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:53:37.873119 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21b017a0-a17b-4f3a-b5a7-d587666b4a8a-maas-api-tls podName:21b017a0-a17b-4f3a-b5a7-d587666b4a8a nodeName:}" failed. No retries permitted until 2026-07-06 15:53:38.37310114 +0000 UTC m=+2164.986849667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/21b017a0-a17b-4f3a-b5a7-d587666b4a8a-maas-api-tls") pod "maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv" (UID: "21b017a0-a17b-4f3a-b5a7-d587666b4a8a") : secret "maas-api-e2e-shared-b-650de1-serving-cert" not found Jul 06 15:53:37.873375 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:37.873356 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b017a0-a17b-4f3a-b5a7-d587666b4a8a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv\" (UID: \"21b017a0-a17b-4f3a-b5a7-d587666b4a8a\") " pod="opendatahub/maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv" Jul 06 15:53:37.895813 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:37.895788 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzvpb\" (UniqueName: \"kubernetes.io/projected/21b017a0-a17b-4f3a-b5a7-d587666b4a8a-kube-api-access-wzvpb\") pod \"maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv\" (UID: \"21b017a0-a17b-4f3a-b5a7-d587666b4a8a\") " pod="opendatahub/maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv" Jul 06 15:53:37.923396 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:37.923311 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:38.280934 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:38.280841 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:38.377829 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:38.377796 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21b017a0-a17b-4f3a-b5a7-d587666b4a8a-maas-api-tls\") pod \"maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv\" (UID: \"21b017a0-a17b-4f3a-b5a7-d587666b4a8a\") " pod="opendatahub/maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv" Jul 06 15:53:38.380218 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:38.380190 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21b017a0-a17b-4f3a-b5a7-d587666b4a8a-maas-api-tls\") pod \"maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv\" (UID: \"21b017a0-a17b-4f3a-b5a7-d587666b4a8a\") " pod="opendatahub/maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv" Jul 06 15:53:38.522187 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:38.522149 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:38.594635 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:38.594605 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv" Jul 06 15:53:38.634390 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:38.634354 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:38.741387 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:38.741363 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv"] Jul 06 15:53:38.743410 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:53:38.743381 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b017a0_a17b_4f3a_b5a7_d587666b4a8a.slice/crio-c8fd4ff93986109cc8b44f136f82267773efdc4e71a9cdaabeb49d683b3fa33e WatchSource:0}: Error finding container c8fd4ff93986109cc8b44f136f82267773efdc4e71a9cdaabeb49d683b3fa33e: Status 404 returned error can't find the container with id c8fd4ff93986109cc8b44f136f82267773efdc4e71a9cdaabeb49d683b3fa33e Jul 06 15:53:38.889748 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:38.889663 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv" event={"ID":"21b017a0-a17b-4f3a-b5a7-d587666b4a8a","Type":"ContainerStarted","Data":"c8fd4ff93986109cc8b44f136f82267773efdc4e71a9cdaabeb49d683b3fa33e"} Jul 06 15:53:39.896442 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:39.896406 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv" event={"ID":"21b017a0-a17b-4f3a-b5a7-d587666b4a8a","Type":"ContainerStarted","Data":"7c8163ceaaf776e9e243ed0972db0732f295d52562dfdd7c7998e5af2f0d2450"} Jul 06 15:53:39.896838 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:39.896468 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv" Jul 06 15:53:39.925070 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:39.925004 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv" podStartSLOduration=1.870373527 podStartE2EDuration="2.92498211s" totalImagesPullingTime="1.054608583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:53:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:53:39.921091078 +0000 UTC m=+2166.534839626" watchObservedRunningTime="2026-07-06 15:53:39.92498211 +0000 UTC m=+2166.538730648" Jul 06 15:53:41.761490 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:41.761450 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:45.908729 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:45.908698 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv" Jul 06 15:53:51.634645 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:51.634595 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:51.791733 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:51.791700 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:58.786871 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:58.786833 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:59.585821 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:59.585791 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:53:59.689137 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:53:59.689101 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:54:16.491262 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:16.491230 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:54:28.055615 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:54:28.055566 2570 controller.go:195] "Failed to update lease" err="Put \"https://a1726f74684e14eb892bb0f401116777-7ae0b4b8ed5baa14.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-108.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Jul 06 15:54:38.055956 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:54:38.055895 2570 controller.go:195] "Failed to update lease" err="Put \"https://a1726f74684e14eb892bb0f401116777-7ae0b4b8ed5baa14.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-108.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Jul 06 15:54:48.056643 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:54:48.056594 2570 controller.go:195] "Failed to update lease" err="Put \"https://a1726f74684e14eb892bb0f401116777-7ae0b4b8ed5baa14.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-108.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Jul 06 15:54:56.305657 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.305555 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-dnp9j\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.305657 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.305556 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.305657 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.305591 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.305657 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.305556 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.305657 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.305565 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.305657 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.305623 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.305657 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.305632 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.306325 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.305668 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.306325 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.305667 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.306325 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.305688 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-379916-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.306325 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.305694 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.306325 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.305723 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.306325 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.305751 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.306325 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.305757 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-rd652\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.306325 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.305724 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.306325 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.305795 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.306325 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.305805 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.306325 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.305816 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-6kchn\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.306325 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.305821 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.306325 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.305845 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-pscfl\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.306325 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.305853 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.306325 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.305867 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9vhkj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.306325 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.305890 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"original-pull-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.306325 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.305914 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.306325 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.305932 2570 reflector.go:571] "Warning: watch ended with error" reflector="pkg/kubelet/config/apiserver.go:66" type="*v1.Pod" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307239 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.305944 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307239 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.305951 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-pdbjc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307239 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.305971 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307239 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.305974 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"default-dockercfg-8k4vd\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307239 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.305995 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307239 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306010 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307239 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306033 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307239 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306047 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307239 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306050 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307239 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306070 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307239 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306082 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307239 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306090 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307239 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306098 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307239 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306116 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307239 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306140 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-bhhp6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307239 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306188 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8n8jq\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307239 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306210 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-pmxsn\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307239 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306242 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307861 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306260 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307861 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306280 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-650de1-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307861 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306334 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307861 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306344 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307861 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:54:56.306345 2570 controller.go:195] "Failed to update lease" err="Put \"https://a1726f74684e14eb892bb0f401116777-7ae0b4b8ed5baa14.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-108.ec2.internal?timeout=10s\": http2: client connection lost" Jul 06 15:54:56.307861 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306352 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307861 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306368 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307861 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306387 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307861 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306385 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307861 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306411 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307861 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306414 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-rvqm6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307861 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306436 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307861 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306441 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307861 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306457 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rqfh6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307861 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306476 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-v7t7r\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307861 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306488 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307861 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306490 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.307861 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306537 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308407 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306543 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308407 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306552 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308407 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306556 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308407 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306588 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308407 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306540 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308407 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306597 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308407 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306583 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308407 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306611 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308407 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306618 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308407 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306563 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308407 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306590 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-f644w\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308407 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306635 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308407 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306641 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308407 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306645 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7t2f5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308407 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306604 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308407 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306675 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308407 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306676 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308407 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306678 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308966 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306694 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308966 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306728 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308966 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306716 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308966 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306741 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308966 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306749 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308966 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306751 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308966 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306740 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308966 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306729 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308966 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306769 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308966 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306776 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308966 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306786 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308966 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306784 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308966 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306808 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308966 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306818 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-x9pws\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308966 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306829 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-x62fk\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308966 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306836 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-gjmr6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308966 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306851 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.308966 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306860 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"default-dockercfg-rd9dd\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.309518 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306865 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-swqh6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.309518 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306853 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-9ln4amltt5nqj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.309518 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306888 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.309518 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306877 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-s2zzx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.309518 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306902 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vpwgw\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.309518 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306889 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.309518 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306909 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.309518 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306908 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8l7xg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.309518 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306922 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.309518 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306923 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.309518 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306930 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.309518 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306944 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-87dt7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.309518 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306949 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.309518 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306953 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.309518 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306968 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.309518 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306976 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.309518 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306984 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c8of7rodm035b\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.309518 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306997 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310083 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307002 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310083 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307011 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-dockercfg-x6ccw\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310083 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307000 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310083 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307020 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310083 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306970 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310083 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307033 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310083 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307034 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310083 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307046 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310083 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.306981 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310083 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307027 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310083 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307059 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gh99x\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310083 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307071 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7fmgx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310083 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307084 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310083 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307096 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310083 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307095 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310083 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307099 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310083 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307119 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"operator-dockercfg-fvb8l\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310083 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307126 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4jl77\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310649 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307138 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310649 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307150 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310649 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307153 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310649 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307164 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hswq6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310649 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307109 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310649 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307174 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-fpfwb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310649 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307177 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310649 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307104 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310649 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307155 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310649 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307196 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310649 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307138 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310649 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307198 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310649 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307214 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310649 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307173 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fs6r9\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310649 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307228 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-x2rt6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310649 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307222 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310649 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307238 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kc4hf\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.310649 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307226 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.311210 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307249 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ts4hg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.311210 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307197 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.311210 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307250 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.311210 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307281 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.311210 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307214 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.311210 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307263 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.311210 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307154 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:56.311210 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:56.307283 2570 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-8d4tk\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 15:54:57.122689 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.122652 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-pmxsn\"" Jul 06 15:54:57.122689 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.122675 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 15:54:57.123701 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.123682 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8l7xg\"" Jul 06 15:54:57.123754 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.123700 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 15:54:57.133609 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.133579 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 15:54:57.133753 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.133670 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8k4vd\"" Jul 06 15:54:57.133894 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.133881 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 06 15:54:57.141772 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.141751 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 15:54:57.144127 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.144114 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 06 15:54:57.152819 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.152802 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 15:54:57.153486 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.153469 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 06 15:54:57.158858 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.158842 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 15:54:57.159377 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.159361 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 15:54:57.167001 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.166984 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 15:54:57.167518 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.167483 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 06 15:54:57.174021 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.173994 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 15:54:57.174112 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.174073 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7t2f5\"" Jul 06 15:54:57.176004 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.175989 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-swqh6\"" Jul 06 15:54:57.187956 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.187936 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 15:54:57.190961 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.190947 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 15:54:57.191722 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.191686 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 15:54:57.203628 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.203608 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 15:54:57.203774 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.203612 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 06 15:54:57.204666 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.204652 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vpwgw\"" Jul 06 15:54:57.205253 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.205236 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9ln4amltt5nqj\"" Jul 06 15:54:57.228145 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.228121 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 06 15:54:57.229837 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.229820 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-x2rt6\"" Jul 06 15:54:57.237100 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.237082 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-87dt7\"" Jul 06 15:54:57.237155 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.237116 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 06 15:54:57.242888 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.242871 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 15:54:57.243998 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.243986 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-fpfwb\"" Jul 06 15:54:57.249051 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.249036 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 06 15:54:57.250110 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.250096 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fs6r9\"" Jul 06 15:54:57.252295 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.252279 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 15:54:57.255020 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.255005 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 06 15:54:57.257255 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.257243 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 15:54:57.270072 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.270051 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 06 15:54:57.293515 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.293476 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 15:54:57.296644 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.296620 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-x62fk\"" Jul 06 15:54:57.303822 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.303797 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 15:54:57.307627 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.307609 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-650de1-serving-cert\"" Jul 06 15:54:57.308057 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.307851 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 06 15:54:57.313478 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.313459 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 06 15:54:57.332674 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.332647 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 15:54:57.333657 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.333639 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 06 15:54:57.343188 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.343169 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-v7t7r\"" Jul 06 15:54:57.349161 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.349137 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 06 15:54:57.349283 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.349228 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 06 15:54:57.360589 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.360564 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rd9dd\"" Jul 06 15:54:57.360706 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.360564 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 15:54:57.365628 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.365612 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 15:54:57.370107 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.370080 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-6kchn\"" Jul 06 15:54:57.370235 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.370203 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 15:54:57.374011 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.373956 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-bhhp6\"" Jul 06 15:54:57.374275 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.374259 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-pdbjc\"" Jul 06 15:54:57.400895 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.400862 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 06 15:54:57.415345 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.415319 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 15:54:57.428944 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.428911 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 15:54:57.435137 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.435116 2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 15:54:57.437028 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.437012 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 15:54:57.437239 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.437224 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-s2zzx\"" Jul 06 15:54:57.437319 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.437228 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rqfh6\"" Jul 06 15:54:57.444965 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.444946 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 15:54:57.457618 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.457591 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 15:54:57.457725 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.457596 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 15:54:57.469907 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.469874 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 06 15:54:57.470067 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.470049 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 06 15:54:57.471588 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.471571 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 06 15:54:57.485890 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.485863 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 15:54:57.492750 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.492721 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 06 15:54:57.493580 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.493567 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kc4hf\"" Jul 06 15:54:57.498095 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.498077 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 15:54:57.505020 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.504996 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 15:54:57.510119 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.510100 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 06 15:54:57.522770 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.522741 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 15:54:57.532439 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.532419 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 15:54:57.533258 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.533243 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gh99x\"" Jul 06 15:54:57.541071 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.541054 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-rd652\"" Jul 06 15:54:57.542287 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.542266 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9vhkj\"" Jul 06 15:54:57.547773 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.547757 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 15:54:57.547921 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.547906 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 06 15:54:57.548178 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.548164 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 15:54:57.548423 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.548409 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 15:54:57.563520 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.563479 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-dnp9j\"" Jul 06 15:54:57.565778 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.565763 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-rvqm6\"" Jul 06 15:54:57.565871 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.565780 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-f644w\"" Jul 06 15:54:57.566792 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.566778 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 15:54:57.578191 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.578168 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 15:54:57.586269 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.586251 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 15:54:57.586337 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.586251 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-x6ccw\"" Jul 06 15:54:57.594971 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.594952 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 15:54:57.602830 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.602811 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 15:54:57.610551 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.610531 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 15:54:57.611954 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.611937 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 15:54:57.624228 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.624167 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 15:54:57.624402 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.624388 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 15:54:57.624447 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.624400 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 06 15:54:57.625866 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.625850 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 15:54:57.626383 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.626370 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-x9pws\"" Jul 06 15:54:57.634087 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.634064 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 15:54:57.634663 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.634647 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 15:54:57.634890 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.634876 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 15:54:57.636339 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.636325 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 15:54:57.636855 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.636840 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hswq6\"" Jul 06 15:54:57.641265 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.641246 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 15:54:57.645418 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.645403 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 15:54:57.657401 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.657379 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 15:54:57.667578 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.667558 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-fvb8l\"" Jul 06 15:54:57.671365 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.671348 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 06 15:54:57.678131 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.678116 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 15:54:57.678664 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.678647 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 15:54:57.678758 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.678749 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c8of7rodm035b\"" Jul 06 15:54:57.686869 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.686850 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 15:54:57.686928 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.686852 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 06 15:54:57.687643 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.687630 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 15:54:57.700370 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.700348 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 15:54:57.701847 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.701831 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 06 15:54:57.706263 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.706246 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ts4hg\"" Jul 06 15:54:57.708386 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.708371 2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 15:54:57.712626 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.712611 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 06 15:54:57.713698 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.713684 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-8d4tk\"" Jul 06 15:54:57.717769 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.717754 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 06 15:54:57.717844 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.717830 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 15:54:57.718006 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.717993 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 15:54:57.736142 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.736122 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 06 15:54:57.744772 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.744756 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 06 15:54:57.754813 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.754789 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 15:54:57.755956 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.755938 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 15:54:57.761278 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.761261 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 15:54:57.763619 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.763604 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 15:54:57.772457 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.772438 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 06 15:54:57.780129 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.780110 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4jl77\"" Jul 06 15:54:57.783228 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.783209 2570 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 15:54:57.791847 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.791828 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 15:54:57.819358 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.819331 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 15:54:57.832291 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.832261 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 06 15:54:57.853041 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.853018 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 15:54:57.878717 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.878645 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 06 15:54:57.892371 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.892343 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 15:54:57.913797 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.913769 2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 15:54:57.931878 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.931854 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 15:54:57.952101 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.952074 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 15:54:57.971822 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.971798 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 15:54:57.992207 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:57.992184 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8n8jq\"" Jul 06 15:54:58.013232 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:58.013205 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 06 15:54:58.033029 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:58.033003 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 15:54:58.053021 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:58.052998 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 06 15:54:58.075044 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:58.075021 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-379916-serving-cert\"" Jul 06 15:54:58.092418 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:58.092395 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 15:54:58.113163 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:58.113141 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7fmgx\"" Jul 06 15:54:58.132602 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:58.132533 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-pscfl\"" Jul 06 15:54:58.152737 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:58.152711 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 06 15:54:58.173356 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:58.173330 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 15:54:58.194496 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:58.194472 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 06 15:54:58.212454 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:58.212430 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 15:54:58.233180 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:58.233159 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 15:54:58.254944 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:58.254918 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 15:54:58.274368 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:54:58.274345 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-gjmr6\"" Jul 06 15:55:51.558420 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:55:51.558385 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:56:27.259290 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:27.259209 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:56:28.243284 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:28.243250 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:56:34.800623 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:34.800593 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r"] Jul 06 15:56:34.803875 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:34.803859 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r" Jul 06 15:56:34.808829 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:34.808804 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-379916\"/\"sub-test-model-379916-kserve-self-signed-certs\"" Jul 06 15:56:34.809914 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:34.809882 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-379916\"/\"openshift-service-ca.crt\"" Jul 06 15:56:34.812234 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:34.812217 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-379916\"/\"default-dockercfg-bvmvm\"" Jul 06 15:56:34.812806 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:34.812789 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-379916\"/\"kube-root-ca.crt\"" Jul 06 15:56:34.825714 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:34.825682 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r"] Jul 06 15:56:34.910111 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:34.910080 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86cc6da1-89e6-4faf-8c67-a07436af5ec7-model-cache\") pod \"sub-test-model-379916-kserve-55c867d997-nnb8r\" (UID: \"86cc6da1-89e6-4faf-8c67-a07436af5ec7\") " pod="ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r" Jul 06 15:56:34.910274 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:34.910132 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctccr\" (UniqueName: \"kubernetes.io/projected/86cc6da1-89e6-4faf-8c67-a07436af5ec7-kube-api-access-ctccr\") pod \"sub-test-model-379916-kserve-55c867d997-nnb8r\" (UID: \"86cc6da1-89e6-4faf-8c67-a07436af5ec7\") " pod="ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r" Jul 06 15:56:34.910274 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:34.910221 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86cc6da1-89e6-4faf-8c67-a07436af5ec7-home\") pod \"sub-test-model-379916-kserve-55c867d997-nnb8r\" (UID: \"86cc6da1-89e6-4faf-8c67-a07436af5ec7\") " pod="ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r" Jul 06 15:56:34.910348 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:34.910283 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86cc6da1-89e6-4faf-8c67-a07436af5ec7-dshm\") pod \"sub-test-model-379916-kserve-55c867d997-nnb8r\" (UID: \"86cc6da1-89e6-4faf-8c67-a07436af5ec7\") " pod="ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r" Jul 06 15:56:34.910408 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:34.910394 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86cc6da1-89e6-4faf-8c67-a07436af5ec7-tls-certs\") pod \"sub-test-model-379916-kserve-55c867d997-nnb8r\" (UID: \"86cc6da1-89e6-4faf-8c67-a07436af5ec7\") " pod="ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r" Jul 06 15:56:34.910469 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:34.910458 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86cc6da1-89e6-4faf-8c67-a07436af5ec7-tmp-dir\") pod \"sub-test-model-379916-kserve-55c867d997-nnb8r\" (UID: \"86cc6da1-89e6-4faf-8c67-a07436af5ec7\") " pod="ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r" Jul 06 15:56:34.926255 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:34.926228 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:56:35.011139 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:35.011101 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86cc6da1-89e6-4faf-8c67-a07436af5ec7-tmp-dir\") pod \"sub-test-model-379916-kserve-55c867d997-nnb8r\" (UID: \"86cc6da1-89e6-4faf-8c67-a07436af5ec7\") " pod="ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r" Jul 06 15:56:35.011332 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:35.011155 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86cc6da1-89e6-4faf-8c67-a07436af5ec7-model-cache\") pod \"sub-test-model-379916-kserve-55c867d997-nnb8r\" (UID: \"86cc6da1-89e6-4faf-8c67-a07436af5ec7\") " pod="ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r" Jul 06 15:56:35.011332 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:35.011192 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctccr\" (UniqueName: \"kubernetes.io/projected/86cc6da1-89e6-4faf-8c67-a07436af5ec7-kube-api-access-ctccr\") pod \"sub-test-model-379916-kserve-55c867d997-nnb8r\" (UID: \"86cc6da1-89e6-4faf-8c67-a07436af5ec7\") " pod="ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r" Jul 06 15:56:35.011332 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:35.011228 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86cc6da1-89e6-4faf-8c67-a07436af5ec7-home\") pod \"sub-test-model-379916-kserve-55c867d997-nnb8r\" (UID: \"86cc6da1-89e6-4faf-8c67-a07436af5ec7\") " pod="ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r" Jul 06 15:56:35.011332 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:35.011260 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86cc6da1-89e6-4faf-8c67-a07436af5ec7-dshm\") pod \"sub-test-model-379916-kserve-55c867d997-nnb8r\" (UID: \"86cc6da1-89e6-4faf-8c67-a07436af5ec7\") " pod="ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r" Jul 06 15:56:35.011332 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:35.011286 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86cc6da1-89e6-4faf-8c67-a07436af5ec7-tls-certs\") pod \"sub-test-model-379916-kserve-55c867d997-nnb8r\" (UID: \"86cc6da1-89e6-4faf-8c67-a07436af5ec7\") " pod="ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r" Jul 06 15:56:35.011632 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:35.011595 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86cc6da1-89e6-4faf-8c67-a07436af5ec7-tmp-dir\") pod \"sub-test-model-379916-kserve-55c867d997-nnb8r\" (UID: \"86cc6da1-89e6-4faf-8c67-a07436af5ec7\") " pod="ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r" Jul 06 15:56:35.011693 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:35.011669 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86cc6da1-89e6-4faf-8c67-a07436af5ec7-model-cache\") pod \"sub-test-model-379916-kserve-55c867d997-nnb8r\" (UID: \"86cc6da1-89e6-4faf-8c67-a07436af5ec7\") " pod="ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r" Jul 06 15:56:35.011729 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:35.011691 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86cc6da1-89e6-4faf-8c67-a07436af5ec7-home\") pod \"sub-test-model-379916-kserve-55c867d997-nnb8r\" (UID: \"86cc6da1-89e6-4faf-8c67-a07436af5ec7\") " pod="ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r" Jul 06 15:56:35.013541 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:35.013518 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86cc6da1-89e6-4faf-8c67-a07436af5ec7-dshm\") pod \"sub-test-model-379916-kserve-55c867d997-nnb8r\" (UID: \"86cc6da1-89e6-4faf-8c67-a07436af5ec7\") " pod="ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r" Jul 06 15:56:35.013766 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:35.013748 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86cc6da1-89e6-4faf-8c67-a07436af5ec7-tls-certs\") pod \"sub-test-model-379916-kserve-55c867d997-nnb8r\" (UID: \"86cc6da1-89e6-4faf-8c67-a07436af5ec7\") " pod="ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r" Jul 06 15:56:35.023219 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:35.023199 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctccr\" (UniqueName: \"kubernetes.io/projected/86cc6da1-89e6-4faf-8c67-a07436af5ec7-kube-api-access-ctccr\") pod \"sub-test-model-379916-kserve-55c867d997-nnb8r\" (UID: \"86cc6da1-89e6-4faf-8c67-a07436af5ec7\") " pod="ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r" Jul 06 15:56:35.121820 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:35.121788 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r" Jul 06 15:56:35.278426 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:35.278387 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r"] Jul 06 15:56:35.283627 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:56:35.283601 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86cc6da1_89e6_4faf_8c67_a07436af5ec7.slice/crio-0137c4c1c9b531642548b9a88aecc82b055219ddf1e6d15c60588f717877ec90 WatchSource:0}: Error finding container 0137c4c1c9b531642548b9a88aecc82b055219ddf1e6d15c60588f717877ec90: Status 404 returned error can't find the container with id 0137c4c1c9b531642548b9a88aecc82b055219ddf1e6d15c60588f717877ec90 Jul 06 15:56:35.720266 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:35.720230 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:56:35.757200 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:35.757167 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r" event={"ID":"86cc6da1-89e6-4faf-8c67-a07436af5ec7","Type":"ContainerStarted","Data":"14a8591214043b235ecdcad14c1d3889508ea49b00ed058209a7b115491f90fb"} Jul 06 15:56:35.757371 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:35.757210 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r" event={"ID":"86cc6da1-89e6-4faf-8c67-a07436af5ec7","Type":"ContainerStarted","Data":"0137c4c1c9b531642548b9a88aecc82b055219ddf1e6d15c60588f717877ec90"} Jul 06 15:56:35.786211 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:35.786146 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r" podStartSLOduration=1.7861272540000002 podStartE2EDuration="1.786127254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:56:35.78321769 +0000 UTC m=+2342.396966228" watchObservedRunningTime="2026-07-06 15:56:35.786127254 +0000 UTC m=+2342.399875792" Jul 06 15:56:45.122140 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:45.122107 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r" Jul 06 15:56:45.122682 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:45.122159 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r" Jul 06 15:56:45.134626 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:45.134604 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r" Jul 06 15:56:45.823854 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:56:45.823824 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r" Jul 06 15:57:05.929562 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:57:05.929510 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:26.027124 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:57:26.027086 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:34.833861 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:57:34.833830 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-mlw9w_91e0c855-5b32-4ca5-bab6-8e3aae20c1e0/console-operator/2.log" Jul 06 15:57:34.837843 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:57:34.837814 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-mlw9w_91e0c855-5b32-4ca5-bab6-8e3aae20c1e0/console-operator/2.log" Jul 06 15:57:34.838061 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:57:34.838043 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hxpw_3eca6180-d678-4729-ad6f-979a025af841/ovn-acl-logging/0.log" Jul 06 15:57:34.841906 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:57:34.841885 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hxpw_3eca6180-d678-4729-ad6f-979a025af841/ovn-acl-logging/0.log" Jul 06 15:57:34.962679 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:57:34.962652 2570 scope.go:117] "RemoveContainer" containerID="c550ff08b5e5972138e6c312c02b9fe7871780e040048ddcaab6f19b9b3d9595" Jul 06 15:57:49.519003 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:57:49.518953 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r"] Jul 06 15:57:49.534616 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:57:49.534570 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:50.471201 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:57:50.471169 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:58:00.476917 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:00.476833 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:58:04.539247 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:04.539209 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r" podUID="86cc6da1-89e6-4faf-8c67-a07436af5ec7" containerName="main" containerID="cri-o://14a8591214043b235ecdcad14c1d3889508ea49b00ed058209a7b115491f90fb" gracePeriod=45 Jul 06 15:58:04.797275 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:04.797209 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r" Jul 06 15:58:04.890905 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:04.890867 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86cc6da1-89e6-4faf-8c67-a07436af5ec7-tmp-dir\") pod \"86cc6da1-89e6-4faf-8c67-a07436af5ec7\" (UID: \"86cc6da1-89e6-4faf-8c67-a07436af5ec7\") " Jul 06 15:58:04.891100 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:04.890922 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctccr\" (UniqueName: \"kubernetes.io/projected/86cc6da1-89e6-4faf-8c67-a07436af5ec7-kube-api-access-ctccr\") pod \"86cc6da1-89e6-4faf-8c67-a07436af5ec7\" (UID: \"86cc6da1-89e6-4faf-8c67-a07436af5ec7\") " Jul 06 15:58:04.891100 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:04.890953 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86cc6da1-89e6-4faf-8c67-a07436af5ec7-tls-certs\") pod \"86cc6da1-89e6-4faf-8c67-a07436af5ec7\" (UID: \"86cc6da1-89e6-4faf-8c67-a07436af5ec7\") " Jul 06 15:58:04.891100 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:04.890994 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86cc6da1-89e6-4faf-8c67-a07436af5ec7-home\") pod \"86cc6da1-89e6-4faf-8c67-a07436af5ec7\" (UID: \"86cc6da1-89e6-4faf-8c67-a07436af5ec7\") " Jul 06 15:58:04.891100 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:04.891020 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86cc6da1-89e6-4faf-8c67-a07436af5ec7-dshm\") pod \"86cc6da1-89e6-4faf-8c67-a07436af5ec7\" (UID: \"86cc6da1-89e6-4faf-8c67-a07436af5ec7\") " Jul 06 15:58:04.891100 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:04.891099 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86cc6da1-89e6-4faf-8c67-a07436af5ec7-model-cache\") pod \"86cc6da1-89e6-4faf-8c67-a07436af5ec7\" (UID: \"86cc6da1-89e6-4faf-8c67-a07436af5ec7\") " Jul 06 15:58:04.891373 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:04.891158 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86cc6da1-89e6-4faf-8c67-a07436af5ec7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "86cc6da1-89e6-4faf-8c67-a07436af5ec7" (UID: "86cc6da1-89e6-4faf-8c67-a07436af5ec7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:58:04.891373 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:04.891299 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86cc6da1-89e6-4faf-8c67-a07436af5ec7-home" (OuterVolumeSpecName: "home") pod "86cc6da1-89e6-4faf-8c67-a07436af5ec7" (UID: "86cc6da1-89e6-4faf-8c67-a07436af5ec7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:58:04.891479 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:04.891393 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86cc6da1-89e6-4faf-8c67-a07436af5ec7-model-cache" (OuterVolumeSpecName: "model-cache") pod "86cc6da1-89e6-4faf-8c67-a07436af5ec7" (UID: "86cc6da1-89e6-4faf-8c67-a07436af5ec7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:58:04.891479 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:04.891435 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86cc6da1-89e6-4faf-8c67-a07436af5ec7-home\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:58:04.891479 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:04.891453 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86cc6da1-89e6-4faf-8c67-a07436af5ec7-tmp-dir\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:58:04.893315 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:04.893289 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86cc6da1-89e6-4faf-8c67-a07436af5ec7-kube-api-access-ctccr" (OuterVolumeSpecName: "kube-api-access-ctccr") pod "86cc6da1-89e6-4faf-8c67-a07436af5ec7" (UID: "86cc6da1-89e6-4faf-8c67-a07436af5ec7"). InnerVolumeSpecName "kube-api-access-ctccr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:58:04.893426 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:04.893340 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86cc6da1-89e6-4faf-8c67-a07436af5ec7-dshm" (OuterVolumeSpecName: "dshm") pod "86cc6da1-89e6-4faf-8c67-a07436af5ec7" (UID: "86cc6da1-89e6-4faf-8c67-a07436af5ec7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:58:04.893426 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:04.893349 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86cc6da1-89e6-4faf-8c67-a07436af5ec7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "86cc6da1-89e6-4faf-8c67-a07436af5ec7" (UID: "86cc6da1-89e6-4faf-8c67-a07436af5ec7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:58:04.992181 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:04.992146 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86cc6da1-89e6-4faf-8c67-a07436af5ec7-dshm\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:58:04.992181 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:04.992175 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86cc6da1-89e6-4faf-8c67-a07436af5ec7-model-cache\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:58:04.992181 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:04.992186 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctccr\" (UniqueName: \"kubernetes.io/projected/86cc6da1-89e6-4faf-8c67-a07436af5ec7-kube-api-access-ctccr\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:58:04.992424 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:04.992194 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86cc6da1-89e6-4faf-8c67-a07436af5ec7-tls-certs\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:58:05.206811 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:05.206778 2570 generic.go:356] "Generic (PLEG): container finished" podID="86cc6da1-89e6-4faf-8c67-a07436af5ec7" containerID="14a8591214043b235ecdcad14c1d3889508ea49b00ed058209a7b115491f90fb" exitCode=0 Jul 06 15:58:05.206995 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:05.206820 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r" Jul 06 15:58:05.206995 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:05.206828 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r" event={"ID":"86cc6da1-89e6-4faf-8c67-a07436af5ec7","Type":"ContainerDied","Data":"14a8591214043b235ecdcad14c1d3889508ea49b00ed058209a7b115491f90fb"} Jul 06 15:58:05.206995 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:05.206859 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r" event={"ID":"86cc6da1-89e6-4faf-8c67-a07436af5ec7","Type":"ContainerDied","Data":"0137c4c1c9b531642548b9a88aecc82b055219ddf1e6d15c60588f717877ec90"} Jul 06 15:58:05.206995 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:05.206877 2570 scope.go:117] "RemoveContainer" containerID="14a8591214043b235ecdcad14c1d3889508ea49b00ed058209a7b115491f90fb" Jul 06 15:58:05.223778 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:05.223757 2570 scope.go:117] "RemoveContainer" containerID="14a8591214043b235ecdcad14c1d3889508ea49b00ed058209a7b115491f90fb" Jul 06 15:58:05.224089 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:58:05.224069 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14a8591214043b235ecdcad14c1d3889508ea49b00ed058209a7b115491f90fb\": container with ID starting with 14a8591214043b235ecdcad14c1d3889508ea49b00ed058209a7b115491f90fb not found: ID does not exist" containerID="14a8591214043b235ecdcad14c1d3889508ea49b00ed058209a7b115491f90fb" Jul 06 15:58:05.224145 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:05.224099 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a8591214043b235ecdcad14c1d3889508ea49b00ed058209a7b115491f90fb"} err="failed to get container status \"14a8591214043b235ecdcad14c1d3889508ea49b00ed058209a7b115491f90fb\": rpc error: code = NotFound desc = could not find container \"14a8591214043b235ecdcad14c1d3889508ea49b00ed058209a7b115491f90fb\": container with ID starting with 14a8591214043b235ecdcad14c1d3889508ea49b00ed058209a7b115491f90fb not found: ID does not exist" Jul 06 15:58:05.248126 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:05.248091 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r"] Jul 06 15:58:05.258045 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:05.257964 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-379916/sub-test-model-379916-kserve-55c867d997-nnb8r"] Jul 06 15:58:06.799633 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:06.799596 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86cc6da1-89e6-4faf-8c67-a07436af5ec7" path="/var/lib/kubelet/pods/86cc6da1-89e6-4faf-8c67-a07436af5ec7/volumes" Jul 06 15:58:08.782262 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:08.782229 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:58:08.961850 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:08.961816 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:58:34.370461 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.370431 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx"] Jul 06 15:58:34.370964 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.370822 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86cc6da1-89e6-4faf-8c67-a07436af5ec7" containerName="main" Jul 06 15:58:34.370964 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.370833 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="86cc6da1-89e6-4faf-8c67-a07436af5ec7" containerName="main" Jul 06 15:58:34.370964 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.370909 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="86cc6da1-89e6-4faf-8c67-a07436af5ec7" containerName="main" Jul 06 15:58:34.374132 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.374116 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx" Jul 06 15:58:34.379058 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.379033 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-650de1\"/\"rate-test-model-650de1-kserve-self-signed-certs\"" Jul 06 15:58:34.379058 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.379052 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-650de1\"/\"openshift-service-ca.crt\"" Jul 06 15:58:34.379242 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.379075 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-650de1\"/\"default-dockercfg-zpztf\"" Jul 06 15:58:34.379242 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.379052 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-650de1\"/\"kube-root-ca.crt\"" Jul 06 15:58:34.385838 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.385820 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx"] Jul 06 15:58:34.415405 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.415375 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:58:34.494769 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.494728 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a32277e9-af14-4e3b-9295-09255dd33aa4-tls-certs\") pod \"rate-test-model-650de1-kserve-589767864c-4f8wx\" (UID: \"a32277e9-af14-4e3b-9295-09255dd33aa4\") " pod="ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx" Jul 06 15:58:34.494977 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.494777 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr47s\" (UniqueName: \"kubernetes.io/projected/a32277e9-af14-4e3b-9295-09255dd33aa4-kube-api-access-mr47s\") pod \"rate-test-model-650de1-kserve-589767864c-4f8wx\" (UID: \"a32277e9-af14-4e3b-9295-09255dd33aa4\") " pod="ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx" Jul 06 15:58:34.494977 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.494838 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a32277e9-af14-4e3b-9295-09255dd33aa4-model-cache\") pod \"rate-test-model-650de1-kserve-589767864c-4f8wx\" (UID: \"a32277e9-af14-4e3b-9295-09255dd33aa4\") " pod="ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx" Jul 06 15:58:34.494977 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.494873 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a32277e9-af14-4e3b-9295-09255dd33aa4-home\") pod \"rate-test-model-650de1-kserve-589767864c-4f8wx\" (UID: \"a32277e9-af14-4e3b-9295-09255dd33aa4\") " pod="ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx" Jul 06 15:58:34.494977 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.494909 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a32277e9-af14-4e3b-9295-09255dd33aa4-dshm\") pod \"rate-test-model-650de1-kserve-589767864c-4f8wx\" (UID: \"a32277e9-af14-4e3b-9295-09255dd33aa4\") " pod="ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx" Jul 06 15:58:34.494977 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.494930 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a32277e9-af14-4e3b-9295-09255dd33aa4-tmp-dir\") pod \"rate-test-model-650de1-kserve-589767864c-4f8wx\" (UID: \"a32277e9-af14-4e3b-9295-09255dd33aa4\") " pod="ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx" Jul 06 15:58:34.596515 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.596468 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a32277e9-af14-4e3b-9295-09255dd33aa4-tls-certs\") pod \"rate-test-model-650de1-kserve-589767864c-4f8wx\" (UID: \"a32277e9-af14-4e3b-9295-09255dd33aa4\") " pod="ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx" Jul 06 15:58:34.596696 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.596533 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr47s\" (UniqueName: \"kubernetes.io/projected/a32277e9-af14-4e3b-9295-09255dd33aa4-kube-api-access-mr47s\") pod \"rate-test-model-650de1-kserve-589767864c-4f8wx\" (UID: \"a32277e9-af14-4e3b-9295-09255dd33aa4\") " pod="ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx" Jul 06 15:58:34.596696 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.596569 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a32277e9-af14-4e3b-9295-09255dd33aa4-model-cache\") pod \"rate-test-model-650de1-kserve-589767864c-4f8wx\" (UID: \"a32277e9-af14-4e3b-9295-09255dd33aa4\") " pod="ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx" Jul 06 15:58:34.596696 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.596593 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a32277e9-af14-4e3b-9295-09255dd33aa4-home\") pod \"rate-test-model-650de1-kserve-589767864c-4f8wx\" (UID: \"a32277e9-af14-4e3b-9295-09255dd33aa4\") " pod="ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx" Jul 06 15:58:34.596696 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.596634 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a32277e9-af14-4e3b-9295-09255dd33aa4-dshm\") pod \"rate-test-model-650de1-kserve-589767864c-4f8wx\" (UID: \"a32277e9-af14-4e3b-9295-09255dd33aa4\") " pod="ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx" Jul 06 15:58:34.596696 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.596655 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a32277e9-af14-4e3b-9295-09255dd33aa4-tmp-dir\") pod \"rate-test-model-650de1-kserve-589767864c-4f8wx\" (UID: \"a32277e9-af14-4e3b-9295-09255dd33aa4\") " pod="ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx" Jul 06 15:58:34.597049 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.597026 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a32277e9-af14-4e3b-9295-09255dd33aa4-model-cache\") pod \"rate-test-model-650de1-kserve-589767864c-4f8wx\" (UID: \"a32277e9-af14-4e3b-9295-09255dd33aa4\") " pod="ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx" Jul 06 15:58:34.597086 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.597039 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a32277e9-af14-4e3b-9295-09255dd33aa4-home\") pod \"rate-test-model-650de1-kserve-589767864c-4f8wx\" (UID: \"a32277e9-af14-4e3b-9295-09255dd33aa4\") " pod="ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx" Jul 06 15:58:34.597086 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.597074 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a32277e9-af14-4e3b-9295-09255dd33aa4-tmp-dir\") pod \"rate-test-model-650de1-kserve-589767864c-4f8wx\" (UID: \"a32277e9-af14-4e3b-9295-09255dd33aa4\") " pod="ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx" Jul 06 15:58:34.598867 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.598839 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a32277e9-af14-4e3b-9295-09255dd33aa4-dshm\") pod \"rate-test-model-650de1-kserve-589767864c-4f8wx\" (UID: \"a32277e9-af14-4e3b-9295-09255dd33aa4\") " pod="ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx" Jul 06 15:58:34.599412 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.599396 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-650de1\"/\"rate-test-model-650de1-kserve-self-signed-certs\"" Jul 06 15:58:34.608485 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.608461 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-650de1\"/\"kube-root-ca.crt\"" Jul 06 15:58:34.609212 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.609188 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a32277e9-af14-4e3b-9295-09255dd33aa4-tls-certs\") pod \"rate-test-model-650de1-kserve-589767864c-4f8wx\" (UID: \"a32277e9-af14-4e3b-9295-09255dd33aa4\") " pod="ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx" Jul 06 15:58:34.621297 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.621237 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-650de1\"/\"openshift-service-ca.crt\"" Jul 06 15:58:34.629065 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.629036 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr47s\" (UniqueName: \"kubernetes.io/projected/a32277e9-af14-4e3b-9295-09255dd33aa4-kube-api-access-mr47s\") pod \"rate-test-model-650de1-kserve-589767864c-4f8wx\" (UID: \"a32277e9-af14-4e3b-9295-09255dd33aa4\") " pod="ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx" Jul 06 15:58:34.707591 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.707555 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-650de1\"/\"default-dockercfg-zpztf\"" Jul 06 15:58:34.714280 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.714253 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx" Jul 06 15:58:34.856431 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:34.856406 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx"] Jul 06 15:58:34.858591 ip-10-0-130-108 kubenswrapper[2570]: W0706 15:58:34.858556 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda32277e9_af14_4e3b_9295_09255dd33aa4.slice/crio-c14cf2d27a734271414d823d0b181b672d2375fab83321f452780d47c4389d77 WatchSource:0}: Error finding container c14cf2d27a734271414d823d0b181b672d2375fab83321f452780d47c4389d77: Status 404 returned error can't find the container with id c14cf2d27a734271414d823d0b181b672d2375fab83321f452780d47c4389d77 Jul 06 15:58:35.363195 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:35.363157 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx" event={"ID":"a32277e9-af14-4e3b-9295-09255dd33aa4","Type":"ContainerStarted","Data":"89abc9ed65008c2f7dd33536e2307bf51242fa21b2d43986a3bace96e772ee8e"} Jul 06 15:58:35.363397 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:35.363202 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx" event={"ID":"a32277e9-af14-4e3b-9295-09255dd33aa4","Type":"ContainerStarted","Data":"c14cf2d27a734271414d823d0b181b672d2375fab83321f452780d47c4389d77"} Jul 06 15:58:35.398444 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:35.398392 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx" podStartSLOduration=1.398375052 podStartE2EDuration="1.398375052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:58:35.396322072 +0000 UTC m=+2462.010070610" watchObservedRunningTime="2026-07-06 15:58:35.398375052 +0000 UTC m=+2462.012123588" Jul 06 15:58:44.715427 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:44.715387 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx" Jul 06 15:58:44.715856 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:44.715448 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx" Jul 06 15:58:44.728301 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:44.728272 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx" Jul 06 15:58:45.425861 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:45.425828 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx" Jul 06 15:58:53.005820 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:53.005781 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:58:53.123944 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:58:53.123901 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:59:00.497850 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:00.497820 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:59:00.597595 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:00.597562 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:59:17.026213 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:17.026176 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:59:17.914786 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:17.914753 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx"] Jul 06 15:59:18.062481 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:18.062442 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:59:19.241572 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:19.241538 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:59:19.657835 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:19.657802 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:59:19.803900 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:19.803867 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:59:19.900896 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:19.900863 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:59:20.067187 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:20.067101 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-379916-7c4967dc55-tq99b"] Jul 06 15:59:20.067373 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:20.067344 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-379916-7c4967dc55-tq99b" podUID="6743211a-0b82-48d7-bc29-75d14913fc5a" containerName="maas-api" containerID="cri-o://df17983a95120a217c171b79ecaa16a0d18f95995cf82ce363ef2c20283a1209" gracePeriod=30 Jul 06 15:59:20.113768 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:20.113741 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:59:20.326436 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:20.326417 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-379916-7c4967dc55-tq99b" Jul 06 15:59:20.448232 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:20.448179 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5drp\" (UniqueName: \"kubernetes.io/projected/6743211a-0b82-48d7-bc29-75d14913fc5a-kube-api-access-l5drp\") pod \"6743211a-0b82-48d7-bc29-75d14913fc5a\" (UID: \"6743211a-0b82-48d7-bc29-75d14913fc5a\") " Jul 06 15:59:20.448232 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:20.448238 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6743211a-0b82-48d7-bc29-75d14913fc5a-maas-api-tls\") pod \"6743211a-0b82-48d7-bc29-75d14913fc5a\" (UID: \"6743211a-0b82-48d7-bc29-75d14913fc5a\") " Jul 06 15:59:20.448551 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:20.448301 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6743211a-0b82-48d7-bc29-75d14913fc5a-openshift-service-ca\") pod \"6743211a-0b82-48d7-bc29-75d14913fc5a\" (UID: \"6743211a-0b82-48d7-bc29-75d14913fc5a\") " Jul 06 15:59:20.448771 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:20.448746 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6743211a-0b82-48d7-bc29-75d14913fc5a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6743211a-0b82-48d7-bc29-75d14913fc5a" (UID: "6743211a-0b82-48d7-bc29-75d14913fc5a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:59:20.450357 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:20.450334 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6743211a-0b82-48d7-bc29-75d14913fc5a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6743211a-0b82-48d7-bc29-75d14913fc5a" (UID: "6743211a-0b82-48d7-bc29-75d14913fc5a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:59:20.450357 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:20.450347 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6743211a-0b82-48d7-bc29-75d14913fc5a-kube-api-access-l5drp" (OuterVolumeSpecName: "kube-api-access-l5drp") pod "6743211a-0b82-48d7-bc29-75d14913fc5a" (UID: "6743211a-0b82-48d7-bc29-75d14913fc5a"). InnerVolumeSpecName "kube-api-access-l5drp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:59:20.549248 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:20.549194 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5drp\" (UniqueName: \"kubernetes.io/projected/6743211a-0b82-48d7-bc29-75d14913fc5a-kube-api-access-l5drp\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:59:20.549248 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:20.549242 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6743211a-0b82-48d7-bc29-75d14913fc5a-maas-api-tls\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:59:20.549248 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:20.549251 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6743211a-0b82-48d7-bc29-75d14913fc5a-openshift-service-ca\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:59:20.586869 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:20.586795 2570 generic.go:356] "Generic (PLEG): container finished" podID="6743211a-0b82-48d7-bc29-75d14913fc5a" containerID="df17983a95120a217c171b79ecaa16a0d18f95995cf82ce363ef2c20283a1209" exitCode=0 Jul 06 15:59:20.586869 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:20.586836 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-379916-7c4967dc55-tq99b" Jul 06 15:59:20.587101 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:20.586878 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-379916-7c4967dc55-tq99b" event={"ID":"6743211a-0b82-48d7-bc29-75d14913fc5a","Type":"ContainerDied","Data":"df17983a95120a217c171b79ecaa16a0d18f95995cf82ce363ef2c20283a1209"} Jul 06 15:59:20.587101 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:20.586939 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-379916-7c4967dc55-tq99b" event={"ID":"6743211a-0b82-48d7-bc29-75d14913fc5a","Type":"ContainerDied","Data":"04449f14239e6dbfd937ce424e7aa4d334f79656dcafd798b4f1ade162c1bd3b"} Jul 06 15:59:20.587101 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:20.586970 2570 scope.go:117] "RemoveContainer" containerID="df17983a95120a217c171b79ecaa16a0d18f95995cf82ce363ef2c20283a1209" Jul 06 15:59:20.596095 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:20.596079 2570 scope.go:117] "RemoveContainer" containerID="df17983a95120a217c171b79ecaa16a0d18f95995cf82ce363ef2c20283a1209" Jul 06 15:59:20.596332 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:59:20.596312 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df17983a95120a217c171b79ecaa16a0d18f95995cf82ce363ef2c20283a1209\": container with ID starting with df17983a95120a217c171b79ecaa16a0d18f95995cf82ce363ef2c20283a1209 not found: ID does not exist" containerID="df17983a95120a217c171b79ecaa16a0d18f95995cf82ce363ef2c20283a1209" Jul 06 15:59:20.596376 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:20.596341 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df17983a95120a217c171b79ecaa16a0d18f95995cf82ce363ef2c20283a1209"} err="failed to get container status \"df17983a95120a217c171b79ecaa16a0d18f95995cf82ce363ef2c20283a1209\": rpc error: code = NotFound desc = could not find container \"df17983a95120a217c171b79ecaa16a0d18f95995cf82ce363ef2c20283a1209\": container with ID starting with df17983a95120a217c171b79ecaa16a0d18f95995cf82ce363ef2c20283a1209 not found: ID does not exist" Jul 06 15:59:20.616726 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:20.616674 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-379916-7c4967dc55-tq99b"] Jul 06 15:59:20.618824 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:20.618800 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-379916-7c4967dc55-tq99b"] Jul 06 15:59:20.799511 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:20.799462 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6743211a-0b82-48d7-bc29-75d14913fc5a" path="/var/lib/kubelet/pods/6743211a-0b82-48d7-bc29-75d14913fc5a/volumes" Jul 06 15:59:24.706458 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:24.706425 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:59:32.934680 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:32.934637 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx" podUID="a32277e9-af14-4e3b-9295-09255dd33aa4" containerName="main" containerID="cri-o://89abc9ed65008c2f7dd33536e2307bf51242fa21b2d43986a3bace96e772ee8e" gracePeriod=45 Jul 06 15:59:33.187217 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:33.187155 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx" Jul 06 15:59:33.374651 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:33.374614 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a32277e9-af14-4e3b-9295-09255dd33aa4-dshm\") pod \"a32277e9-af14-4e3b-9295-09255dd33aa4\" (UID: \"a32277e9-af14-4e3b-9295-09255dd33aa4\") " Jul 06 15:59:33.374856 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:33.374681 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a32277e9-af14-4e3b-9295-09255dd33aa4-tls-certs\") pod \"a32277e9-af14-4e3b-9295-09255dd33aa4\" (UID: \"a32277e9-af14-4e3b-9295-09255dd33aa4\") " Jul 06 15:59:33.374856 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:33.374767 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a32277e9-af14-4e3b-9295-09255dd33aa4-home\") pod \"a32277e9-af14-4e3b-9295-09255dd33aa4\" (UID: \"a32277e9-af14-4e3b-9295-09255dd33aa4\") " Jul 06 15:59:33.374856 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:33.374806 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a32277e9-af14-4e3b-9295-09255dd33aa4-tmp-dir\") pod \"a32277e9-af14-4e3b-9295-09255dd33aa4\" (UID: \"a32277e9-af14-4e3b-9295-09255dd33aa4\") " Jul 06 15:59:33.374856 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:33.374829 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mr47s\" (UniqueName: \"kubernetes.io/projected/a32277e9-af14-4e3b-9295-09255dd33aa4-kube-api-access-mr47s\") pod \"a32277e9-af14-4e3b-9295-09255dd33aa4\" (UID: \"a32277e9-af14-4e3b-9295-09255dd33aa4\") " Jul 06 15:59:33.374856 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:33.374854 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a32277e9-af14-4e3b-9295-09255dd33aa4-model-cache\") pod \"a32277e9-af14-4e3b-9295-09255dd33aa4\" (UID: \"a32277e9-af14-4e3b-9295-09255dd33aa4\") " Jul 06 15:59:33.375145 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:33.375061 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a32277e9-af14-4e3b-9295-09255dd33aa4-home" (OuterVolumeSpecName: "home") pod "a32277e9-af14-4e3b-9295-09255dd33aa4" (UID: "a32277e9-af14-4e3b-9295-09255dd33aa4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:59:33.375145 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:33.375081 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a32277e9-af14-4e3b-9295-09255dd33aa4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a32277e9-af14-4e3b-9295-09255dd33aa4" (UID: "a32277e9-af14-4e3b-9295-09255dd33aa4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:59:33.375244 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:33.375162 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a32277e9-af14-4e3b-9295-09255dd33aa4-model-cache" (OuterVolumeSpecName: "model-cache") pod "a32277e9-af14-4e3b-9295-09255dd33aa4" (UID: "a32277e9-af14-4e3b-9295-09255dd33aa4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:59:33.375244 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:33.375179 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a32277e9-af14-4e3b-9295-09255dd33aa4-home\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:59:33.376992 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:33.376958 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a32277e9-af14-4e3b-9295-09255dd33aa4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a32277e9-af14-4e3b-9295-09255dd33aa4" (UID: "a32277e9-af14-4e3b-9295-09255dd33aa4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:59:33.376992 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:33.376972 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a32277e9-af14-4e3b-9295-09255dd33aa4-dshm" (OuterVolumeSpecName: "dshm") pod "a32277e9-af14-4e3b-9295-09255dd33aa4" (UID: "a32277e9-af14-4e3b-9295-09255dd33aa4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:59:33.377147 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:33.377009 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a32277e9-af14-4e3b-9295-09255dd33aa4-kube-api-access-mr47s" (OuterVolumeSpecName: "kube-api-access-mr47s") pod "a32277e9-af14-4e3b-9295-09255dd33aa4" (UID: "a32277e9-af14-4e3b-9295-09255dd33aa4"). InnerVolumeSpecName "kube-api-access-mr47s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:59:33.476042 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:33.475947 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a32277e9-af14-4e3b-9295-09255dd33aa4-tls-certs\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:59:33.476042 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:33.475981 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a32277e9-af14-4e3b-9295-09255dd33aa4-tmp-dir\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:59:33.476042 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:33.475991 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr47s\" (UniqueName: \"kubernetes.io/projected/a32277e9-af14-4e3b-9295-09255dd33aa4-kube-api-access-mr47s\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:59:33.476042 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:33.476001 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a32277e9-af14-4e3b-9295-09255dd33aa4-model-cache\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:59:33.476042 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:33.476009 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a32277e9-af14-4e3b-9295-09255dd33aa4-dshm\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:59:33.651819 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:33.651779 2570 generic.go:356] "Generic (PLEG): container finished" podID="a32277e9-af14-4e3b-9295-09255dd33aa4" containerID="89abc9ed65008c2f7dd33536e2307bf51242fa21b2d43986a3bace96e772ee8e" exitCode=0 Jul 06 15:59:33.651819 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:33.651817 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx" Jul 06 15:59:33.652016 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:33.651831 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx" event={"ID":"a32277e9-af14-4e3b-9295-09255dd33aa4","Type":"ContainerDied","Data":"89abc9ed65008c2f7dd33536e2307bf51242fa21b2d43986a3bace96e772ee8e"} Jul 06 15:59:33.652016 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:33.651870 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx" event={"ID":"a32277e9-af14-4e3b-9295-09255dd33aa4","Type":"ContainerDied","Data":"c14cf2d27a734271414d823d0b181b672d2375fab83321f452780d47c4389d77"} Jul 06 15:59:33.652016 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:33.651903 2570 scope.go:117] "RemoveContainer" containerID="89abc9ed65008c2f7dd33536e2307bf51242fa21b2d43986a3bace96e772ee8e" Jul 06 15:59:33.665109 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:33.665087 2570 scope.go:117] "RemoveContainer" containerID="89abc9ed65008c2f7dd33536e2307bf51242fa21b2d43986a3bace96e772ee8e" Jul 06 15:59:33.665419 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:59:33.665400 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89abc9ed65008c2f7dd33536e2307bf51242fa21b2d43986a3bace96e772ee8e\": container with ID starting with 89abc9ed65008c2f7dd33536e2307bf51242fa21b2d43986a3bace96e772ee8e not found: ID does not exist" containerID="89abc9ed65008c2f7dd33536e2307bf51242fa21b2d43986a3bace96e772ee8e" Jul 06 15:59:33.665532 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:33.665426 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89abc9ed65008c2f7dd33536e2307bf51242fa21b2d43986a3bace96e772ee8e"} err="failed to get container status \"89abc9ed65008c2f7dd33536e2307bf51242fa21b2d43986a3bace96e772ee8e\": rpc error: code = NotFound desc = could not find container \"89abc9ed65008c2f7dd33536e2307bf51242fa21b2d43986a3bace96e772ee8e\": container with ID starting with 89abc9ed65008c2f7dd33536e2307bf51242fa21b2d43986a3bace96e772ee8e not found: ID does not exist" Jul 06 15:59:33.682640 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:33.682612 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx"] Jul 06 15:59:33.686531 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:33.686490 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-650de1/rate-test-model-650de1-kserve-589767864c-4f8wx"] Jul 06 15:59:34.799694 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:34.799660 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a32277e9-af14-4e3b-9295-09255dd33aa4" path="/var/lib/kubelet/pods/a32277e9-af14-4e3b-9295-09255dd33aa4/volumes" Jul 06 15:59:42.767480 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:42.767438 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:59:43.055036 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:43.054951 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:59:43.341154 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:43.341125 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:59:43.595007 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:43.594927 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:59:43.999058 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:43.998960 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:59:44.144919 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:44.144887 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:59:44.442514 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:44.442466 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:59:44.662636 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:44.662598 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv"] Jul 06 15:59:44.662833 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:44.662812 2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv" podUID="21b017a0-a17b-4f3a-b5a7-d587666b4a8a" containerName="maas-api" containerID="cri-o://7c8163ceaaf776e9e243ed0972db0732f295d52562dfdd7c7998e5af2f0d2450" gracePeriod=30 Jul 06 15:59:44.708193 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:44.708106 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:59:44.915736 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:44.915711 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv" Jul 06 15:59:44.984623 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:44.984531 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzvpb\" (UniqueName: \"kubernetes.io/projected/21b017a0-a17b-4f3a-b5a7-d587666b4a8a-kube-api-access-wzvpb\") pod \"21b017a0-a17b-4f3a-b5a7-d587666b4a8a\" (UID: \"21b017a0-a17b-4f3a-b5a7-d587666b4a8a\") " Jul 06 15:59:44.984802 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:44.984656 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b017a0-a17b-4f3a-b5a7-d587666b4a8a-openshift-service-ca\") pod \"21b017a0-a17b-4f3a-b5a7-d587666b4a8a\" (UID: \"21b017a0-a17b-4f3a-b5a7-d587666b4a8a\") " Jul 06 15:59:44.984802 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:44.984711 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21b017a0-a17b-4f3a-b5a7-d587666b4a8a-maas-api-tls\") pod \"21b017a0-a17b-4f3a-b5a7-d587666b4a8a\" (UID: \"21b017a0-a17b-4f3a-b5a7-d587666b4a8a\") " Jul 06 15:59:44.985043 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:44.985019 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21b017a0-a17b-4f3a-b5a7-d587666b4a8a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "21b017a0-a17b-4f3a-b5a7-d587666b4a8a" (UID: "21b017a0-a17b-4f3a-b5a7-d587666b4a8a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:59:44.986653 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:44.986627 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21b017a0-a17b-4f3a-b5a7-d587666b4a8a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "21b017a0-a17b-4f3a-b5a7-d587666b4a8a" (UID: "21b017a0-a17b-4f3a-b5a7-d587666b4a8a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:59:44.986765 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:44.986751 2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b017a0-a17b-4f3a-b5a7-d587666b4a8a-kube-api-access-wzvpb" (OuterVolumeSpecName: "kube-api-access-wzvpb") pod "21b017a0-a17b-4f3a-b5a7-d587666b4a8a" (UID: "21b017a0-a17b-4f3a-b5a7-d587666b4a8a"). InnerVolumeSpecName "kube-api-access-wzvpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:59:45.085801 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:45.085764 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b017a0-a17b-4f3a-b5a7-d587666b4a8a-openshift-service-ca\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:59:45.085801 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:45.085798 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21b017a0-a17b-4f3a-b5a7-d587666b4a8a-maas-api-tls\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:59:45.085801 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:45.085807 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzvpb\" (UniqueName: \"kubernetes.io/projected/21b017a0-a17b-4f3a-b5a7-d587666b4a8a-kube-api-access-wzvpb\") on node \"ip-10-0-130-108.ec2.internal\" DevicePath \"\"" Jul 06 15:59:45.716203 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:45.716167 2570 generic.go:356] "Generic (PLEG): container finished" podID="21b017a0-a17b-4f3a-b5a7-d587666b4a8a" containerID="7c8163ceaaf776e9e243ed0972db0732f295d52562dfdd7c7998e5af2f0d2450" exitCode=0 Jul 06 15:59:45.716408 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:45.716202 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv" event={"ID":"21b017a0-a17b-4f3a-b5a7-d587666b4a8a","Type":"ContainerDied","Data":"7c8163ceaaf776e9e243ed0972db0732f295d52562dfdd7c7998e5af2f0d2450"} Jul 06 15:59:45.716408 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:45.716261 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv" event={"ID":"21b017a0-a17b-4f3a-b5a7-d587666b4a8a","Type":"ContainerDied","Data":"c8fd4ff93986109cc8b44f136f82267773efdc4e71a9cdaabeb49d683b3fa33e"} Jul 06 15:59:45.716408 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:45.716214 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv" Jul 06 15:59:45.716408 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:45.716291 2570 scope.go:117] "RemoveContainer" containerID="7c8163ceaaf776e9e243ed0972db0732f295d52562dfdd7c7998e5af2f0d2450" Jul 06 15:59:45.725611 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:45.725595 2570 scope.go:117] "RemoveContainer" containerID="7c8163ceaaf776e9e243ed0972db0732f295d52562dfdd7c7998e5af2f0d2450" Jul 06 15:59:45.725875 ip-10-0-130-108 kubenswrapper[2570]: E0706 15:59:45.725854 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c8163ceaaf776e9e243ed0972db0732f295d52562dfdd7c7998e5af2f0d2450\": container with ID starting with 7c8163ceaaf776e9e243ed0972db0732f295d52562dfdd7c7998e5af2f0d2450 not found: ID does not exist" containerID="7c8163ceaaf776e9e243ed0972db0732f295d52562dfdd7c7998e5af2f0d2450" Jul 06 15:59:45.725919 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:45.725884 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c8163ceaaf776e9e243ed0972db0732f295d52562dfdd7c7998e5af2f0d2450"} err="failed to get container status \"7c8163ceaaf776e9e243ed0972db0732f295d52562dfdd7c7998e5af2f0d2450\": rpc error: code = NotFound desc = could not find container \"7c8163ceaaf776e9e243ed0972db0732f295d52562dfdd7c7998e5af2f0d2450\": container with ID starting with 7c8163ceaaf776e9e243ed0972db0732f295d52562dfdd7c7998e5af2f0d2450 not found: ID does not exist" Jul 06 15:59:45.740684 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:45.740658 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv"] Jul 06 15:59:45.745643 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:45.745621 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-650de1-7ddf99644b-5jdqv"] Jul 06 15:59:46.799889 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:46.799855 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21b017a0-a17b-4f3a-b5a7-d587666b4a8a" path="/var/lib/kubelet/pods/21b017a0-a17b-4f3a-b5a7-d587666b4a8a/volumes" Jul 06 15:59:50.697400 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:50.697360 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:59:50.793131 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:50.793041 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:59:51.155669 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:51.155634 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:59:51.350991 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:51.350960 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 15:59:56.753661 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:56.753626 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-v26k2_88aec541-064a-45e1-957e-8e8ef732b044/manager/0.log" Jul 06 15:59:56.995681 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:56.995654 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-78ddc7f974-xfcw7_12f2c889-1197-4510-ae8b-42ba30b4da87/maas-api/0.log" Jul 06 15:59:57.354638 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:57.354588 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78d5649946-8r2fm_89e49b52-85a2-4a39-bf50-13c848cde6f2/manager/0.log" Jul 06 15:59:57.732348 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:57.732259 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9957545c6-6drld_65571a24-12fe-4e9c-a18b-2fa6eab3aa6e/copy-manifests/0.log" Jul 06 15:59:57.740492 ip-10-0-130-108 kubenswrapper[2570]: I0706 15:59:57.740471 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9957545c6-6drld_65571a24-12fe-4e9c-a18b-2fa6eab3aa6e/manager/0.log" Jul 06 16:00:00.437896 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:00.437861 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-vx2ww_765378c0-492a-45a3-9856-ce85a713feb8/manager/0.log" Jul 06 16:00:00.819560 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:00.819460 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-695cd9447dgsb7d_6aa0b326-347a-4d2e-be45-5ff1c965dcf7/istio-proxy/0.log" Jul 06 16:00:01.335240 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:01.335207 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df9b4f89-7h54z_e54ba255-5f61-4cbb-a1f0-df4bb0d58876/istio-proxy/1.log" Jul 06 16:00:01.458229 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:01.458200 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-kpczm_e67caf7d-9368-4d1e-b12b-074d1a9e6bfd/payload-pre-processing/0.log" Jul 06 16:00:01.696627 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:01.696553 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-55896fdd7f-l5sg2_d00f67b5-9c7a-41f4-bd9e-2160fcd8641a/router/0.log" Jul 06 16:00:02.049709 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:02.049626 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-swvlt_ef5f40da-0145-420a-8ddc-9144d6898cb7/main/0.log" Jul 06 16:00:02.300208 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:02.300128 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-d6zzt_22e4c371-1e9b-433e-b0d3-658cb2894eae/main/0.log" Jul 06 16:00:02.419549 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:02.419514 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2d9tw_a239c966-2d9d-4709-8f4e-43eb160cad14/main/0.log" Jul 06 16:00:02.533769 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:02.533740 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-rzzwm_c7338594-417e-4104-afd8-4e8e6ced455c/main/0.log" Jul 06 16:00:07.142463 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:07.142435 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 16:00:07.389355 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:07.389319 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 16:00:09.637971 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:09.637935 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" limit=1024 Jul 06 16:00:10.734101 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:10.734067 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-l77mc_9aaa45f1-1a99-4e94-96f3-a12ddbe62e83/global-pull-secret-syncer/0.log" Jul 06 16:00:10.891467 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:10.891434 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-66jdx_ab57a97c-0e04-4482-9444-836a8b6a2f37/konnectivity-agent/0.log" Jul 06 16:00:11.034276 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:11.034188 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-108.ec2.internal_6f497d407b6748f0adb639ce1a9ea113/haproxy/0.log" Jul 06 16:00:16.255175 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:16.255071 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-vx2ww_765378c0-492a-45a3-9856-ce85a713feb8/manager/0.log" Jul 06 16:00:18.538173 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:18.538142 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-785fdbc95c-q4q4j_8ffb012d-1795-42fb-b6c8-db9419cedd19/metrics-server/0.log" Jul 06 16:00:18.575463 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:18.575419 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-4x8sx_278d8a2b-9af9-43dd-aeaf-0db49545ee23/monitoring-plugin/0.log" Jul 06 16:00:18.608074 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:18.608003 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bwbvt_8351bb23-9cd0-412e-8bef-d36f1eeff370/node-exporter/0.log" Jul 06 16:00:18.639529 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:18.639474 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bwbvt_8351bb23-9cd0-412e-8bef-d36f1eeff370/kube-rbac-proxy/0.log" Jul 06 16:00:18.665187 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:18.665160 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bwbvt_8351bb23-9cd0-412e-8bef-d36f1eeff370/init-textfile/0.log" Jul 06 16:00:19.009980 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:19.009903 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_48c083a8-ef73-46dc-8a62-d4748a1d4b4c/prometheus/0.log" Jul 06 16:00:19.038071 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:19.038043 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_48c083a8-ef73-46dc-8a62-d4748a1d4b4c/config-reloader/0.log" Jul 06 16:00:19.069545 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:19.069520 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_48c083a8-ef73-46dc-8a62-d4748a1d4b4c/thanos-sidecar/0.log" Jul 06 16:00:19.107248 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:19.107219 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_48c083a8-ef73-46dc-8a62-d4748a1d4b4c/kube-rbac-proxy-web/0.log" Jul 06 16:00:19.168139 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:19.168109 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_48c083a8-ef73-46dc-8a62-d4748a1d4b4c/kube-rbac-proxy/0.log" Jul 06 16:00:19.199892 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:19.199868 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_48c083a8-ef73-46dc-8a62-d4748a1d4b4c/kube-rbac-proxy-thanos/0.log" Jul 06 16:00:19.227243 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:19.227213 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_48c083a8-ef73-46dc-8a62-d4748a1d4b4c/init-config-reloader/0.log" Jul 06 16:00:20.661588 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:20.661546 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pjbpc/perf-node-gather-daemonset-rg9vs"] Jul 06 16:00:20.662063 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:20.662003 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a32277e9-af14-4e3b-9295-09255dd33aa4" containerName="main" Jul 06 16:00:20.662063 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:20.662025 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a32277e9-af14-4e3b-9295-09255dd33aa4" containerName="main" Jul 06 16:00:20.662063 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:20.662040 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21b017a0-a17b-4f3a-b5a7-d587666b4a8a" containerName="maas-api" Jul 06 16:00:20.662063 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:20.662045 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="21b017a0-a17b-4f3a-b5a7-d587666b4a8a" containerName="maas-api" Jul 06 16:00:20.662063 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:20.662054 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6743211a-0b82-48d7-bc29-75d14913fc5a" containerName="maas-api" Jul 06 16:00:20.662063 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:20.662060 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6743211a-0b82-48d7-bc29-75d14913fc5a" containerName="maas-api" Jul 06 16:00:20.662283 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:20.662119 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a32277e9-af14-4e3b-9295-09255dd33aa4" containerName="main" Jul 06 16:00:20.662283 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:20.662127 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="6743211a-0b82-48d7-bc29-75d14913fc5a" containerName="maas-api" Jul 06 16:00:20.662283 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:20.662133 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="21b017a0-a17b-4f3a-b5a7-d587666b4a8a" containerName="maas-api" Jul 06 16:00:20.665292 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:20.665271 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-rg9vs" Jul 06 16:00:20.668338 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:20.668311 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pjbpc\"/\"kube-root-ca.crt\"" Jul 06 16:00:20.668448 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:20.668420 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pjbpc\"/\"openshift-service-ca.crt\"" Jul 06 16:00:20.669571 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:20.669552 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pjbpc\"/\"default-dockercfg-mpklm\"" Jul 06 16:00:20.678975 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:20.678953 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pjbpc/perf-node-gather-daemonset-rg9vs"] Jul 06 16:00:20.733157 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:20.733110 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/edfa1979-7340-4f39-8088-99d640c5d2d0-sys\") pod \"perf-node-gather-daemonset-rg9vs\" (UID: \"edfa1979-7340-4f39-8088-99d640c5d2d0\") " pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-rg9vs" Jul 06 16:00:20.733352 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:20.733215 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg5l5\" (UniqueName: \"kubernetes.io/projected/edfa1979-7340-4f39-8088-99d640c5d2d0-kube-api-access-lg5l5\") pod \"perf-node-gather-daemonset-rg9vs\" (UID: \"edfa1979-7340-4f39-8088-99d640c5d2d0\") " pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-rg9vs" Jul 06 16:00:20.733352 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:20.733263 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/edfa1979-7340-4f39-8088-99d640c5d2d0-podres\") pod \"perf-node-gather-daemonset-rg9vs\" (UID: \"edfa1979-7340-4f39-8088-99d640c5d2d0\") " pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-rg9vs" Jul 06 16:00:20.733352 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:20.733284 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/edfa1979-7340-4f39-8088-99d640c5d2d0-lib-modules\") pod \"perf-node-gather-daemonset-rg9vs\" (UID: \"edfa1979-7340-4f39-8088-99d640c5d2d0\") " pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-rg9vs" Jul 06 16:00:20.733352 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:20.733310 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/edfa1979-7340-4f39-8088-99d640c5d2d0-proc\") pod \"perf-node-gather-daemonset-rg9vs\" (UID: \"edfa1979-7340-4f39-8088-99d640c5d2d0\") " pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-rg9vs" Jul 06 16:00:20.834494 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:20.834455 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg5l5\" (UniqueName: \"kubernetes.io/projected/edfa1979-7340-4f39-8088-99d640c5d2d0-kube-api-access-lg5l5\") pod \"perf-node-gather-daemonset-rg9vs\" (UID: \"edfa1979-7340-4f39-8088-99d640c5d2d0\") " pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-rg9vs" Jul 06 16:00:20.834707 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:20.834543 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/edfa1979-7340-4f39-8088-99d640c5d2d0-podres\") pod \"perf-node-gather-daemonset-rg9vs\" (UID: \"edfa1979-7340-4f39-8088-99d640c5d2d0\") " pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-rg9vs" Jul 06 16:00:20.834707 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:20.834574 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/edfa1979-7340-4f39-8088-99d640c5d2d0-lib-modules\") pod \"perf-node-gather-daemonset-rg9vs\" (UID: \"edfa1979-7340-4f39-8088-99d640c5d2d0\") " pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-rg9vs" Jul 06 16:00:20.834707 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:20.834614 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/edfa1979-7340-4f39-8088-99d640c5d2d0-proc\") pod \"perf-node-gather-daemonset-rg9vs\" (UID: \"edfa1979-7340-4f39-8088-99d640c5d2d0\") " pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-rg9vs" Jul 06 16:00:20.834707 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:20.834661 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/edfa1979-7340-4f39-8088-99d640c5d2d0-sys\") pod \"perf-node-gather-daemonset-rg9vs\" (UID: \"edfa1979-7340-4f39-8088-99d640c5d2d0\") " pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-rg9vs" Jul 06 16:00:20.834893 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:20.834725 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/edfa1979-7340-4f39-8088-99d640c5d2d0-podres\") pod \"perf-node-gather-daemonset-rg9vs\" (UID: \"edfa1979-7340-4f39-8088-99d640c5d2d0\") " pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-rg9vs" Jul 06 16:00:20.834893 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:20.834725 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/edfa1979-7340-4f39-8088-99d640c5d2d0-lib-modules\") pod \"perf-node-gather-daemonset-rg9vs\" (UID: \"edfa1979-7340-4f39-8088-99d640c5d2d0\") " pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-rg9vs" Jul 06 16:00:20.834893 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:20.834761 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/edfa1979-7340-4f39-8088-99d640c5d2d0-sys\") pod \"perf-node-gather-daemonset-rg9vs\" (UID: \"edfa1979-7340-4f39-8088-99d640c5d2d0\") " pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-rg9vs" Jul 06 16:00:20.834893 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:20.834770 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/edfa1979-7340-4f39-8088-99d640c5d2d0-proc\") pod \"perf-node-gather-daemonset-rg9vs\" (UID: \"edfa1979-7340-4f39-8088-99d640c5d2d0\") " pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-rg9vs" Jul 06 16:00:20.846160 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:20.846133 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg5l5\" (UniqueName: \"kubernetes.io/projected/edfa1979-7340-4f39-8088-99d640c5d2d0-kube-api-access-lg5l5\") pod \"perf-node-gather-daemonset-rg9vs\" (UID: \"edfa1979-7340-4f39-8088-99d640c5d2d0\") " pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-rg9vs" Jul 06 16:00:20.975715 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:20.975620 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-rg9vs" Jul 06 16:00:21.119238 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:21.119207 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pjbpc/perf-node-gather-daemonset-rg9vs"] Jul 06 16:00:21.120770 ip-10-0-130-108 kubenswrapper[2570]: W0706 16:00:21.120743 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podedfa1979_7340_4f39_8088_99d640c5d2d0.slice/crio-e603e91b4db6d577b9d33160be6698ec36d5fa8d6169ef5a9bbdb4d3c8335618 WatchSource:0}: Error finding container e603e91b4db6d577b9d33160be6698ec36d5fa8d6169ef5a9bbdb4d3c8335618: Status 404 returned error can't find the container with id e603e91b4db6d577b9d33160be6698ec36d5fa8d6169ef5a9bbdb4d3c8335618 Jul 06 16:00:21.412865 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:21.412834 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-mlw9w_91e0c855-5b32-4ca5-bab6-8e3aae20c1e0/console-operator/2.log" Jul 06 16:00:21.416525 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:21.416489 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-mlw9w_91e0c855-5b32-4ca5-bab6-8e3aae20c1e0/console-operator/3.log" Jul 06 16:00:21.889944 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:21.889911 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-rg9vs" event={"ID":"edfa1979-7340-4f39-8088-99d640c5d2d0","Type":"ContainerStarted","Data":"c00b249abf3d612559d94bda08aef178fcb694c1a145e514d2c69604d1f78de3"} Jul 06 16:00:21.890518 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:21.889960 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-rg9vs" event={"ID":"edfa1979-7340-4f39-8088-99d640c5d2d0","Type":"ContainerStarted","Data":"e603e91b4db6d577b9d33160be6698ec36d5fa8d6169ef5a9bbdb4d3c8335618"} Jul 06 16:00:21.890518 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:21.890049 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-rg9vs" Jul 06 16:00:21.911438 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:21.911387 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-rg9vs" podStartSLOduration=1.9113689520000001 podStartE2EDuration="1.911368952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:00:21.909613264 +0000 UTC m=+2568.523361799" watchObservedRunningTime="2026-07-06 16:00:21.911368952 +0000 UTC m=+2568.525117608" Jul 06 16:00:23.693899 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:23.693872 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8lh8v_1ce904e6-fad6-4078-ab8b-1a45fab198a9/dns/0.log" Jul 06 16:00:23.725106 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:23.725080 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8lh8v_1ce904e6-fad6-4078-ab8b-1a45fab198a9/kube-rbac-proxy/0.log" Jul 06 16:00:23.973721 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:23.973643 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mgd8j_65d43ca0-dc12-4e4f-8a7d-a42c0f10a651/dns-node-resolver/0.log" Jul 06 16:00:24.585770 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:24.585738 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-695c8f4bdb-jm29j_f993611f-b1fa-4b2f-8173-2e78ca6dde08/registry/0.log" Jul 06 16:00:24.699196 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:24.699164 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fbvqs_9a7b5582-0829-4212-8ff0-6379e5eb5306/node-ca/0.log" Jul 06 16:00:25.734781 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:25.734730 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-695cd9447dgsb7d_6aa0b326-347a-4d2e-be45-5ff1c965dcf7/istio-proxy/0.log" Jul 06 16:00:26.036929 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:26.036837 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df9b4f89-7h54z_e54ba255-5f61-4cbb-a1f0-df4bb0d58876/istio-proxy/0.log" Jul 06 16:00:26.061106 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:26.061071 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df9b4f89-7h54z_e54ba255-5f61-4cbb-a1f0-df4bb0d58876/istio-proxy/1.log" Jul 06 16:00:26.129712 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:26.129686 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-kpczm_e67caf7d-9368-4d1e-b12b-074d1a9e6bfd/payload-pre-processing/0.log" Jul 06 16:00:26.230430 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:26.230406 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-55896fdd7f-l5sg2_d00f67b5-9c7a-41f4-bd9e-2160fcd8641a/router/0.log" Jul 06 16:00:27.004920 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:27.004894 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n5z5d_9da83319-4207-4990-ad20-a322c1f750ec/serve-healthcheck-canary/0.log" Jul 06 16:00:27.660804 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:27.660770 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-9ktzq_9f53ff9d-d4c6-4b93-8eed-39e727850e02/insights-operator/0.log" Jul 06 16:00:27.662179 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:27.662152 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-9ktzq_9f53ff9d-d4c6-4b93-8eed-39e727850e02/insights-operator/1.log" Jul 06 16:00:27.692653 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:27.692626 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9hm78_92299e93-9bb8-452f-ba3a-523beff0880a/kube-rbac-proxy/0.log" Jul 06 16:00:27.721670 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:27.721645 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9hm78_92299e93-9bb8-452f-ba3a-523beff0880a/exporter/0.log" Jul 06 16:00:27.753036 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:27.753013 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9hm78_92299e93-9bb8-452f-ba3a-523beff0880a/extractor/0.log" Jul 06 16:00:27.904555 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:27.904527 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-rg9vs" Jul 06 16:00:30.515518 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:30.515482 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-v26k2_88aec541-064a-45e1-957e-8e8ef732b044/manager/0.log" Jul 06 16:00:30.782545 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:30.782516 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-78ddc7f974-xfcw7_12f2c889-1197-4510-ae8b-42ba30b4da87/maas-api/0.log" Jul 06 16:00:31.054226 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:31.054139 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78d5649946-8r2fm_89e49b52-85a2-4a39-bf50-13c848cde6f2/manager/0.log" Jul 06 16:00:31.153280 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:31.153251 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-695cd9447dgsb7d_6aa0b326-347a-4d2e-be45-5ff1c965dcf7/istio-proxy/0.log" Jul 06 16:00:31.367639 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:31.367604 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9957545c6-6drld_65571a24-12fe-4e9c-a18b-2fa6eab3aa6e/manager/0.log" Jul 06 16:00:31.401396 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:31.401374 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9957545c6-6drld_65571a24-12fe-4e9c-a18b-2fa6eab3aa6e/copy-manifests/0.log" Jul 06 16:00:31.452777 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:31.452744 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df9b4f89-7h54z_e54ba255-5f61-4cbb-a1f0-df4bb0d58876/istio-proxy/0.log" Jul 06 16:00:31.477005 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:31.476971 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df9b4f89-7h54z_e54ba255-5f61-4cbb-a1f0-df4bb0d58876/istio-proxy/1.log" Jul 06 16:00:31.502291 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:31.502259 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-kpczm_e67caf7d-9368-4d1e-b12b-074d1a9e6bfd/payload-pre-processing/0.log" Jul 06 16:00:31.559781 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:31.559749 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-55896fdd7f-l5sg2_d00f67b5-9c7a-41f4-bd9e-2160fcd8641a/router/0.log" Jul 06 16:00:32.849548 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:32.849526 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-695cd9447dgsb7d_6aa0b326-347a-4d2e-be45-5ff1c965dcf7/istio-proxy/0.log" Jul 06 16:00:33.068471 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:33.068443 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-59fc8bcc45-67btx_4f192268-d9b0-426f-af32-e188ce5c25c1/manager/0.log" Jul 06 16:00:33.088633 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:33.088605 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df9b4f89-7h54z_e54ba255-5f61-4cbb-a1f0-df4bb0d58876/istio-proxy/0.log" Jul 06 16:00:33.116034 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:33.115953 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df9b4f89-7h54z_e54ba255-5f61-4cbb-a1f0-df4bb0d58876/istio-proxy/1.log" Jul 06 16:00:33.136690 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:33.136662 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-kpczm_e67caf7d-9368-4d1e-b12b-074d1a9e6bfd/payload-pre-processing/0.log" Jul 06 16:00:33.203350 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:33.203324 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-55896fdd7f-l5sg2_d00f67b5-9c7a-41f4-bd9e-2160fcd8641a/router/0.log" Jul 06 16:00:37.975829 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:37.975801 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-jg2tk_b8c9d558-02e2-4906-8480-62d7ac3bd357/migrator/0.log" Jul 06 16:00:38.024035 ip-10-0-130-108 kubenswrapper[2570]: I0706 16:00:38.024005 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-jg2tk_b8c9d558-02e2-4906-8480-62d7ac3bd357/graceful-termination/0.log"